File Formats & Data Serialization

Master reading, writing, and transforming data across formats

Why File Formats Matter

Modern applications constantly exchange data using structured file formats. Whether you're building APIs, processing analytics data, or integrating with third-party services, understanding how to work with JSON, CSV, XML, and other formats is essential.

Common Use Cases:

API Communication: JSON for REST APIs
Data Analysis: CSV for Excel, pandas, R
Configuration: JSON, YAML, TOML
Data Pipelines: ETL processes, transformations
Archival: Parquet, Avro for big data

Quick Format Comparison

Format	Best For	Human Readable	Pros	Cons
JSON	APIs, Config	✓	Lightweight, universal	No date/binary support
CSV	Tabular data	✓	Simple, Excel-compatible	No nested structures
XML	Legacy systems	✓	Schemas, namespaces	Verbose
YAML	Config files	✓	Clean syntax	Indentation-sensitive
Parquet	Big data	✗	Compressed, columnar	Binary format

Working with JSON

JSON (JavaScript Object Notation) is the most common format for APIs and configuration files. Python's json module makes it easy to serialize and deserialize Python objects.

Serialization

Converting Python objects to JSON strings/files

Deserialization

Converting JSON strings/files to Python objects

Basic JSON Operations

import json

# Python dict to JSON
data = {
    "name": "Alice",
    "age": 30,
    "skills": ["Python", "SQL", "APIs"],
    "active": True,
    "salary": None
}

# Write to file
with open("user.json", "w") as f:
    json.dump(data, f, indent=4)

# Convert to JSON string
json_string = json.dumps(data, indent=2)
print(json_string)

# Read from file
with open("user.json") as f:
    loaded = json.load(f)

# Parse JSON string
parsed = json.loads('{"name": "Bob", "age": 25}')

print(loaded["name"])  # Alice
print(parsed["age"])   # 25

Advanced JSON Techniques

import json
from datetime import datetime
from decimal import Decimal

# Custom JSON encoder for special types
class CustomEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        if isinstance(obj, Decimal):
            return float(obj)
        if isinstance(obj, set):
            return list(obj)
        return super().default(obj)

# Using custom encoder
data = {
    "timestamp": datetime.now(),
    "price": Decimal("19.99"),
    "tags": {"python", "json", "tutorial"}
}

json_str = json.dumps(data, cls=CustomEncoder, indent=2)
print(json_str)

# Output:
# {
#   "timestamp": "2026-01-03T10:30:00.123456",
#   "price": 19.99,
#   "tags": ["python", "json", "tutorial"]
# }

# Pretty printing with sorting
data = {"z": 1, "a": 2, "m": 3}
print(json.dumps(data, indent=2, sort_keys=True))

# Compact JSON (no whitespace)
compact = json.dumps(data, separators=(',', ':'))
print(compact)  # {"z":1,"a":2,"m":3}

# Handling nested structures
nested = {
    "users": [
        {"id": 1, "name": "Alice", "roles": ["admin", "user"]},
        {"id": 2, "name": "Bob", "roles": ["user"]}
    ],
    "metadata": {
        "version": "1.0",
        "created": "2026-01-03"
    }
}

with open("complex.json", "w") as f:
    json.dump(nested, f, indent=2)

Key functions:

json.dump(obj, file) - Write Python object to file
json.dumps(obj) - Convert Python object to JSON string
json.load(file) - Read JSON from file to Python object
json.loads(string) - Parse JSON string to Python object

Watch out: JSON doesn't support Python's datetime, Decimal, or set types natively. Use custom encoders or convert to supported types first.

Working with CSV Files

CSV (Comma-Separated Values) is the universal format for tabular data, widely used in Excel, databases, and data analysis tools.

Basic CSV Operations

import csv

# Writing CSV - Basic approach
rows = [
    ["name", "age", "city"],
    ["Alice", 30, "New York"],
    ["Bob", 25, "Boston"]
]

with open("users.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerows(rows)

# Reading CSV - Basic approach
with open("users.csv") as f:
    reader = csv.reader(f)
    header = next(reader)  # Skip header
    for row in reader:
        print(f"{row[0]} is {row[1]} years old")

# Writing CSV - Dict approach (recommended)
users = [
    {"name": "Alice", "age": 30, "city": "New York"},
    {"name": "Bob", "age": 25, "city": "Boston"}
]

with open("users.csv", "w", newline="") as f:
    fieldnames = ["name", "age", "city"]
    writer = csv.DictWriter(f, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(users)

# Reading CSV - Dict approach (recommended)
with open("users.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(f"{row['name']} lives in {row['city']}")

Advanced CSV Techniques

import csv

# Custom delimiters
with open("data.tsv", "w", newline="") as f:
    writer = csv.writer(f, delimiter='\t')  # Tab-separated
    writer.writerow(["name", "score"])
    writer.writerow(["Alice", 95])

# Quoting strategies
with open("text.csv", "w", newline="") as f:
    writer = csv.writer(f, quoting=csv.QUOTE_NONNUMERIC)
    writer.writerow(["name", "comment"])
    writer.writerow(["Alice", "Said, 'Hello world!'"])
    # Output: "Alice","Said, 'Hello world!'"

# Handling missing values
data = [
    {"name": "Alice", "age": 30, "email": "alice@example.com"},
    {"name": "Bob", "age": None, "email": ""},  # Missing data
]

with open("users.csv", "w", newline="") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "age", "email"],
                            restval="N/A")  # Default for missing
    writer.writeheader()
    writer.writerows(data)

# Reading with type conversion
with open("numbers.csv") as f:
    reader = csv.DictReader(f)
    for row in reader:
        # CSV reads everything as strings!
        age = int(row["age"]) if row["age"] else 0
        salary = float(row["salary"]) if row["salary"] else 0.0
        active = row["active"].lower() == "true"

# Filtering data while reading
with open("users.csv") as f:
    reader = csv.DictReader(f)
    active_users = [row for row in reader if row["active"] == "true"]

# Writing filtered data
with open("active_users.csv", "w", newline="") as f:
    if active_users:
        writer = csv.DictWriter(f, fieldnames=active_users[0].keys())
        writer.writeheader()
        writer.writerows(active_users)

Best practices:

Always use newline="" when opening CSV files for writing
Prefer DictReader and DictWriter for clarity
Remember CSV stores everything as strings - convert types as needed
Use with statements to ensure files are properly closed

Working with XML

XML is still common in enterprise systems, SOAP APIs, and configuration files. Python provides xml.etree.ElementTree for XML processing.

import xml.etree.ElementTree as ET

# Parsing XML
xml_string = """
<catalog>
    <book id="1">
        <title>Python Basics</title>
        <author>Alice Smith</author>
        <price>29.99</price>
    </book>
    <book id="2">
        <title>Advanced Python</title>
        <author>Bob Jones</author>
        <price>39.99</price>
    </book>
</catalog>
"""

root = ET.fromstring(xml_string)

# Navigate XML
for book in root.findall('book'):
    book_id = book.get('id')
    title = book.find('title').text
    author = book.find('author').text
    price = float(book.find('price').text)
    print(f"Book {book_id}: {title} by {author} - {price}")

# Creating XML
catalog = ET.Element('catalog')

book1 = ET.SubElement(catalog, 'book', id='1')
ET.SubElement(book1, 'title').text = 'Python Basics'
ET.SubElement(book1, 'author').text = 'Alice Smith'
ET.SubElement(book1, 'price').text = '29.99'

# Write to file with pretty printing
tree = ET.ElementTree(catalog)
ET.indent(tree, space="  ")  # Python 3.9+
tree.write('catalog.xml', encoding='utf-8', xml_declaration=True)

# Reading from file
tree = ET.parse('catalog.xml')
root = tree.getroot()

# Using XPath-like queries
expensive_books = root.findall(".//book[price>30]")  # Requires lxml

Note: For complex XML with namespaces and schemas, consider using thelxml library which provides more features and better performance.

Working with YAML

YAML is popular for configuration files (Docker, Kubernetes, CI/CD) due to its clean, human-readable syntax.

# Install: pip install pyyaml
import yaml

# Python dict to YAML
config = {
    "database": {
        "host": "localhost",
        "port": 5432,
        "credentials": {
            "username": "admin",
            "password": "secret"
        }
    },
    "features": ["auth", "api", "analytics"],
    "debug": True
}

# Write YAML
with open("config.yaml", "w") as f:
    yaml.dump(config, f, default_flow_style=False)

# Read YAML
with open("config.yaml") as f:
    loaded = yaml.safe_load(f)  # Use safe_load for security

print(loaded["database"]["host"])  # localhost

# YAML supports multiple documents in one file
yaml_multi = """
---
name: Document 1
type: config
---
name: Document 2
type: data
"""

documents = yaml.safe_load_all(yaml_multi)
for doc in documents:
    print(doc["name"])

# YAML example with anchors and aliases
yaml_anchors = """
defaults: &defaults
  timeout: 30
  retries: 3

production:
  <<: *defaults
  host: prod.example.com

development:
  <<: *defaults
  host: dev.example.com
  timeout: 60
"""

config = yaml.safe_load(yaml_anchors)
print(config["production"]["timeout"])  # 30 (inherited)

Security note: Always use yaml.safe_load() instead ofyaml.load() to prevent arbitrary code execution.

Plain Text Files

Plain text files are fundamental for logs, notes, and unstructured data.

# Writing text
with open("notes.txt", "w") as f:
    f.write("Hello, world!\n")
    f.write("This is line 2\n")

# Appending to file
with open("log.txt", "a") as f:
    f.write("New log entry\n")

# Reading entire file
with open("notes.txt") as f:
    content = f.read()
    print(content)

# Reading line by line (memory efficient)
with open("large_file.txt") as f:
    for line in f:
        print(line.strip())  # Remove \n

# Reading all lines into list
with open("notes.txt") as f:
    lines = f.readlines()  # Returns list of strings

# Reading specific number of characters
with open("notes.txt") as f:
    chunk = f.read(100)  # Read first 100 characters

# File encoding (important for non-ASCII text)
with open("international.txt", "w", encoding="utf-8") as f:
    f.write("Hello 世界 مرحبا\n")

with open("international.txt", encoding="utf-8") as f:
    content = f.read()

# Processing log files
with open("app.log") as f:
    errors = [line for line in f if "ERROR" in line]
    print(f"Found {len(errors)} errors")

Tip: For large files, always read line-by-line instead of loading the entire file into memory with read().

Transforming Data Between Formats

Real-world applications often need to convert data from one format to another. This is a core skill in ETL (Extract, Transform, Load) pipelines.

Sample JSON Data:

{
  "users": [
    {
      "id": 1,
      "name": "Alice Johnson",
      "email": "alice@example.com",
      "active": true
    },
    {
      "id": 2,
      "name": "Bob Smith",
      "email": "bob@example.com",
      "active": false
    }
  ]
}

Common Conversion Patterns

import json
import csv

# 1. CSV to JSON
def csv_to_json(csv_file, json_file):
    with open(csv_file) as f:
        reader = csv.DictReader(f)
        data = list(reader)

    with open(json_file, "w") as f:
        json.dump(data, f, indent=2)

# Usage
csv_to_json("users.csv", "users.json")

# 2. JSON to CSV
def json_to_csv(json_file, csv_file):
    with open(json_file) as f:
        data = json.load(f)

    # Handle nested JSON
    if isinstance(data, dict):
        data = [data]  # Wrap single object in list

    if data:
        with open(csv_file, "w", newline="") as f:
            writer = csv.DictWriter(f, fieldnames=data[0].keys())
            writer.writeheader()
            writer.writerows(data)

# Usage
json_to_csv("users.json", "users.csv")

# 3. JSON to JSON (transformation)
def transform_user_data(input_file, output_file):
    with open(input_file) as f:
        users = json.load(f)

    # Transform: filter active users and rename fields
    transformed = [
        {
            "user_id": user["id"],
            "full_name": user["name"],
            "contact": user["email"]
        }
        for user in users
        if user.get("active", False)
    ]

    with open(output_file, "w") as f:
        json.dump(transformed, f, indent=2)

# 4. CSV filtering and aggregation
def analyze_sales_csv(input_file, output_file):
    from collections import defaultdict

    sales_by_region = defaultdict(float)

    with open(input_file) as f:
        reader = csv.DictReader(f)
        for row in reader:
            region = row["region"]
            amount = float(row["amount"])
            sales_by_region[region] += amount

    # Write aggregated results
    with open(output_file, "w", newline="") as f:
        writer = csv.writer(f)
        writer.writerow(["region", "total_sales"])
        for region, total in sorted(sales_by_region.items()):
            writer.writerow([region, f"{total:.2f}"])

# 5. Multi-format pipeline
def data_pipeline(csv_input, json_output, filtered_csv):
    # Step 1: Read CSV
    with open(csv_input) as f:
        data = list(csv.DictReader(f))

    # Step 2: Transform and save as JSON
    with open(json_output, "w") as f:
        json.dump(data, f, indent=2)

    # Step 3: Filter and save back to CSV
    active_users = [row for row in data if row["active"] == "true"]

    with open(filtered_csv, "w", newline="") as f:
        if active_users:
            writer = csv.DictWriter(f, fieldnames=active_users[0].keys())
            writer.writeheader()
            writer.writerows(active_users)

    return len(data), len(active_users)

# Usage
total, active = data_pipeline("all_users.csv", "users.json", "active.csv")
print(f"Processed {total} users, {active} active")

Real-world ETL: Production pipelines use tools like pandas, Apache Airflow, or cloud services (AWS Glue, Azure Data Factory) for large-scale transformations, but these fundamental techniques form the building blocks.

Binary Formats & Serialization

Binary formats are more efficient for large datasets and machine-to-machine communication.

Pickle (Python-specific)

Serialize any Python object, but only for Python applications

import pickle

data = {"users": [1, 2, 3]}
with open("data.pkl", "wb") as f:
    pickle.dump(data, f)

with open("data.pkl", "rb") as f:
    loaded = pickle.load(f)

MessagePack

Like JSON but binary - faster and more compact

import msgpack

data = {"users": [1, 2, 3]}
packed = msgpack.packb(data)
unpacked = msgpack.unpackb(packed)

# Parquet - Columnar format for analytics (requires pyarrow or fastparquet)
import pandas as pd

# Create dataframe
df = pd.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [30, 25, 35],
    "salary": [70000, 60000, 80000]
})

# Write to Parquet
df.to_parquet("data.parquet", compression="snappy")

# Read from Parquet
df_loaded = pd.read_parquet("data.parquet")

# Parquet benefits:
# - Columnar storage (fast aggregations)
# - Built-in compression (smaller files)
# - Schema evolution support
# - Native type support (dates, decimals)

# Protocol Buffers (protobuf) - Google's format
# Requires .proto schema definition
# Strongly typed, versioned, very efficient
# Common in microservices and gRPC

Choosing a format:

Human readability needed? → JSON, CSV, YAML
Large datasets? → Parquet, Avro
Speed critical? → MessagePack, Protocol Buffers
Cross-language? → JSON, Protocol Buffers, MessagePack
Python only? → Pickle (simplest)

Best Practices & Common Pitfalls

✓ Do This

Always use with statements for file I/O
Specify encoding explicitly (encoding="utf-8")
Handle missing/malformed data gracefully
Use json.load/dump for files, loads/dumps for strings
Validate data before writing
Use DictReader/DictWriter for CSVs
Set newline="" for CSV on Windows
Consider streaming for large files

✗ Avoid This

Forgetting to close files (use with)
Loading huge files entirely into memory
Using yaml.load() (security risk)
Ignoring character encoding issues
Assuming CSV has consistent structure
Not handling JSON decode errors
Using Pickle for untrusted data (security risk)
Hardcoding file paths

# Error handling example
import json

def safe_load_json(filepath):
    """Safely load JSON with comprehensive error handling"""
    try:
        with open(filepath, encoding="utf-8") as f:
            return json.load(f)
    except FileNotFoundError:
        print(f"Error: File '{filepath}' not found")
        return None
    except json.JSONDecodeError as e:
        print(f"Error: Invalid JSON in '{filepath}': {e}")
        return None
    except UnicodeDecodeError:
        print(f"Error: Encoding issue in '{filepath}'")
        return None
    except Exception as e:
        print(f"Unexpected error: {e}")
        return None

# Streaming large files
def process_large_csv(filepath):
    """Process CSV without loading entire file into memory"""
    total = 0
    with open(filepath) as f:
        reader = csv.DictReader(f)
        for row in reader:
            # Process one row at a time
            total += float(row["amount"])
    return total

Key Takeaways

JSON is the universal language of APIs - lightweight and widely supported
CSV is perfect for tabular data - Excel-compatible and simple
Choose formats based on your needs - readability vs efficiency
Always handle errors gracefully - files can be corrupted or missing
Use context managers - with statements prevent resource leaks
Binary formats for performance - Parquet, MessagePack for big data
Data transformation is a core skill - essential for modern data pipelines

Practice Exercises

Exercise 1: JSON Filtering

Read a JSON file containing user data. Filter for active users over 25 years old and export them to a new CSV file with only name, email, and age columns.

Exercise 2: Log Analysis

Parse a plain text log file and extract all ERROR entries. Generate a JSON report containing: total errors, errors by type, and timestamp of first/last error.

Exercise 3: Data Pipeline

Build a complete ETL pipeline: Read sales data from CSV, calculate total revenue by region and product, then output results in both JSON (for API) and CSV (for Excel).

Exercise 4: Configuration Manager

Create a config manager that reads settings from YAML, validates required fields, provides defaults for missing values, and can export current config to JSON.

Additional Resources

Documentation: docs.python.org/3/library/json.html, docs.python.org/3/library/csv.html
Libraries: pandas (data analysis), lxml (advanced XML), pyarrow (Parquet)
Tools: jq (JSON processor), csvkit (CSV utilities)
Formats: json.org, yaml.org, parquet.apache.org

What's Next?

You can now handle various file formats! Let's learn how to write tests to ensure your code works correctly.

Unit Testing - Write automated tests with unittest and pytest
Test-Driven Development - Learn the TDD workflow for better code quality
Debugging Techniques - Master debugging tools and strategies

Back to "Decorators"

Python IntermediateLesson 6

Next: "Testing & Debugging"