Context Managers & Protocols

Mastering lifecycle guarantees and behavioral contracts

Why Context Managers Matter

Context managers are Python's answer to RAII (Resource Acquisition Is Initialization). They ensure that setup and teardown logic are inseparable, creating "safe zones" for operations that involve files, sockets, locks, or database transactions. They encode lifecycle guarantees, ensuring resources are acquired and released correctly, even in the presence of errors. At scale, they become a core building block for correctness, safety, and composability.

The Problem: Manual Resource Management

Without context managers, developers must remember to manually clean up resources, leading to common bugs like resource leaks, deadlocks, and inconsistent state.

# Bad: Resource leak if exception occurs
f = open('data.txt')
data = f.read()  # What if this raises an exception?
f.close()  # This never executes!

# Better but verbose
f = open('data.txt')
try:
    data = f.read()
finally:
    f.close()  # Guaranteed to run

# Best: Context manager handles everything
with open('data.txt') as f:
    data = f.read()  # Cleanup automatic, exception-safe

The Context Manager Protocol

A context manager is any object implementing two magic methods that define entry and exit behavior:

__enter__(self) -> Any
    # Called when entering 'with' block
    # Return value is bound to 'as' variable
    
__exit__(self, exc_type, exc_value, traceback) -> bool | None
    # Called when exiting 'with' block
    # Receives exception info if one occurred
    # Return True to suppress exception, False/None to propagate
Execution Flow: Python calls __enter__() at the start of the with block, then __exit__() at the end, even if exceptions occur.

Implementing a Context Manager Manually

class ManagedFile:
    def __init__(self, path, mode='r'):
        self.path = path
        self.mode = mode
        self.file = None

    def __enter__(self):
        print(f"Opening {self.path}")
        self.file = open(self.path, self.mode)
        return self.file  # This is what 'as f' receives

    def __exit__(self, exc_type, exc_value, traceback):
        print(f"Closing {self.path}")
        if self.file:
            self.file.close()
        
        # Log exception if one occurred
        if exc_type is not None:
            print(f"Exception occurred: {exc_type.__name__}: {exc_value}")
        
        return False  # Propagate exceptions (don't suppress)

# Usage
with ManagedFile('data.txt') as f:
    content = f.read()
    # File automatically closes, even if exception occurs
Critical Rule: __exit__() must be exception-safe and idempotent. It should handle cleanup even if partially initialized or called multiple times.

Exception Suppression & Error Handling

The __exit__ method receives exception details and can choose to suppress (swallow) or propagate them.

from contextlib import suppress

# Standard library: suppress specific exceptions
with suppress(FileNotFoundError, PermissionError):
    os.remove('might_not_exist.txt')
# Continues execution if file doesn't exist

# Custom suppression for retryable operations
class RetryableOperation:
    def __init__(self, max_attempts=3):
        self.max_attempts = max_attempts
        self.attempt = 0
    
    def __enter__(self):
        return self
    
    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type is None:
            return False  # Success, no exception
        
        self.attempt += 1
        if self.attempt < self.max_attempts:
            print(f"Attempt {self.attempt} failed, retrying...")
            return True  # Suppress exception, allow retry
        
        print(f"All {self.max_attempts} attempts failed")
        return False  # Propagate exception

# Selective suppression by exception type
class IgnoreSpecificErrors:
    def __init__(self, *exception_types):
        self.exception_types = exception_types
    
    def __enter__(self):
        return self
    
    def __exit__(self, exc_type, exc_value, traceback):
        # Suppress only specified exceptions
        return exc_type is not None and issubclass(exc_type, self.exception_types)
Warning: Returning True unconditionally from __exit__()suppresses ALL exceptions, including KeyboardInterrupt and SystemExit. Always check exc_type before suppressing.

Generator-Based Context Managers

The @contextmanager decorator from contextlib lets you create context managers using generator functions. Code before yieldruns on entry, code after runs on exit.

from contextlib import contextmanager
import time
import os

@contextmanager
def timer(name):
    """Time a code block and print duration"""
    start = time.time()
    print(f"Starting {name}...")
    try:
        yield  # Control returns to 'with' block here
    finally:
        duration = time.time() - start
        print(f"{name} took {duration:.2f} seconds")

# Usage
with timer("Data processing"):
    # Your code here
    time.sleep(1)

@contextmanager
def temporary_change(obj, attr, new_value):
    """Temporarily change an attribute, then restore it"""
    original = getattr(obj, attr)
    setattr(obj, attr, new_value)
    try:
        yield obj
    finally:
        setattr(obj, attr, original)

# Usage
from types import SimpleNamespace
config = SimpleNamespace(debug=False)
with temporary_change(config, 'debug', True):
    print(config.debug)  # True
print(config.debug)  # False (restored)

@contextmanager
def atomic_write(path):
    """Write to temp file, rename on success"""
    temp_path = f"{path}.tmp"
    f = open(temp_path, 'w')
    try:
        yield f
        f.close()
        os.rename(temp_path, path)  # Atomic on Unix
    except Exception:
        f.close()
        os.remove(temp_path)
        raise
Best Practice: Prefer @contextmanager for most cases. Use class-based context managers only when you need to maintain state or implement reusable context manager objects.

Managing Dynamic Contexts: ExitStack

ExitStack manages an arbitrary number of context managers dynamically, ensuring all are properly cleaned up even if errors occur during setup.

import json
import shutil
from contextlib import ExitStack, contextmanager

def process_files(file_paths):
    """Process multiple files with guaranteed cleanup"""
    with ExitStack() as stack:
        # Dynamically add context managers
        files = [stack.enter_context(open(p)) for p in file_paths]

        # All files guaranteed to close, even if loop fails partway
        for f in files:
            data = f.read()
            # ... process data ...

def merge_configs(*config_paths):
    """Merge multiple config files safely"""
    with ExitStack() as stack:
        configs = [
            json.load(stack.enter_context(open(p)))
            for p in config_paths
        ]
        return {k: v for config in configs for k, v in config.items()}

# Conditional resource management
@contextmanager
def conditional_lock(lock, should_lock):
    """Optionally acquire a lock"""
    with ExitStack() as stack:
        if should_lock:
            stack.enter_context(lock)
        yield

# Callback registration for custom cleanup
temp_dir = "/tmp/work"
with ExitStack() as stack:
    # Register arbitrary cleanup functions
    stack.callback(print, "Cleaning up...")
    stack.callback(shutil.rmtree, temp_dir, ignore_errors=True)

    # Do work...
Pro Tip: ExitStack is essential when the number of resources isn't known until runtime, such as processing user-provided file lists or managing database connection pools.

Real-World Context Manager Patterns

1. Database Transactions

@contextmanager
def transaction(db_connection):
    """ACID transaction with automatic rollback on error"""
    db_connection.begin()
    try:
        yield db_connection
        db_connection.commit()
    except Exception:
        db_connection.rollback()
        raise

# Usage
with transaction(db) as conn:
    conn.execute("INSERT INTO users ...")
    conn.execute("UPDATE accounts ...")
    # Automatically commits if successful, rolls back if error

2. Thread Safety & Locks

import threading

# Standard library locks are context managers
lock = threading.Lock()

with lock:
    # Critical section - guaranteed to release lock
    shared_resource.modify()

# Timeout-aware locking
@contextmanager
def timeout_lock(lock, timeout):
    acquired = lock.acquire(timeout=timeout)
    if not acquired:
        raise TimeoutError("Could not acquire lock")
    try:
        yield
    finally:
        lock.release()

3. Temporary State Changes

@contextmanager
def chdir(path):
    """Temporarily change working directory"""
    original = os.getcwd()
    try:
        os.chdir(path)
        yield
    finally:
        os.chdir(original)

@contextmanager
def env_var(key, value):
    """Temporarily set environment variable"""
    old_value = os.environ.get(key)
    os.environ[key] = value
    try:
        yield
    finally:
        if old_value is None:
            del os.environ[key]
        else:
            os.environ[key] = old_value

4. Logging & Monitoring

@contextmanager
def log_execution(operation_name, logger):
    """Log operation execution with timing"""
    logger.info(f"Starting {operation_name}")
    start = time.time()
    try:
        yield
        duration = time.time() - start
        logger.info(f"{operation_name} succeeded in {duration:.2f}s")
    except Exception as e:
        duration = time.time() - start
        logger.error(f"{operation_name} failed after {duration:.2f}s: {e}")
        raise

with log_execution("ETL pipeline", logger):
    extract_data()
    transform_data()
    load_data()

Protocols & Structural Typing

Protocols define behavioral contracts through structure rather than inheritance. Any object with the right methods satisfies the protocol, enabling flexible, duck-typed interfaces.

from typing import Protocol, runtime_checkable

@runtime_checkable
class SupportsContextManager(Protocol):
    """Protocol for context manager objects"""
    def __enter__(self) -> object: ...
    def __exit__(self, exc_type, exc_value, traceback) -> bool | None: ...

# Any object with these methods is a context manager
def use_resource(resource: SupportsContextManager):
    with resource:
        # Use resource
        pass

# Custom protocol for closeable objects
@runtime_checkable
class Closeable(Protocol):
    def close(self) -> None: ...

def ensure_closed(obj: Closeable):
    """Works with files, sockets, database connections, etc."""
    try:
        # Use obj
        pass
    finally:
        obj.close()

# Protocol for readable objects
class Readable(Protocol):
    def read(self, size: int = -1) -> bytes: ...

def process_stream(stream: Readable):
    """Works with files, network streams, BytesIO, etc."""
    data = stream.read()
    return process(data)
Design Principle: Protocols enable "structural subtyping", if it walks like a duck and quacks like a duck, it's a duck. This is more flexible than inheritance-based typing and is the foundation of Python's duck typing philosophy.

Async Context Managers

For async code, use __aenter__ and __aexit__ with async with statements. This is essential for async I/O operations like network requests or async database connections.

# Class-based async context manager
class AsyncConnection:
    async def __aenter__(self):
        await self.connect()
        print("Connection established")
        return self

    async def __aexit__(self, exc_type, exc_value, traceback):
        await self.close()
        print("Connection closed")
        return False

# Usage
async with AsyncConnection() as conn:
    await conn.query("SELECT * FROM users")

# Generator-based async context manager
from contextlib import asynccontextmanager

@asynccontextmanager
async def async_timer(name):
    """Async version of timer"""
    start = time.time()
    print(f"Starting {name}...")
    try:
        yield
    finally:
        duration = time.time() - start
        print(f"{name} took {duration:.2f} seconds")

async with async_timer("API call"):
    response = await fetch_data()

# Async database transaction
@asynccontextmanager
async def async_transaction(db):
    await db.begin()
    try:
        yield db
        await db.commit()
    except Exception:
        await db.rollback()
        raise

# Real-world example: connection pooling
@asynccontextmanager
async def get_connection(pool):
    """Get connection from pool, return on exit"""
    conn = await pool.acquire()
    try:
        yield conn
    finally:
        await pool.release(conn)

async with get_connection(db_pool) as conn:
    result = await conn.execute("SELECT ...")
Important: Never mix sync and async context managers. Use async with for async context managers and with for synchronous ones. Attempting to use with on an async context manager will fail.

Common Pitfalls & Anti-Patterns

# BAD: Not exception-safe
@contextmanager
def bad_file_handler(path):
    f = open(path)
    yield f
    f.close()  # Skipped if exception in with block!

# GOOD: Always use try/finally
@contextmanager
def good_file_handler(path):
    f = open(path)
    try:
        yield f
    finally:
        f.close()  # Always executes

# BAD: Suppressing all exceptions
class DangerousContext:
    def __exit__(self, *args):
        return True  # Silences KeyboardInterrupt, SystemExit!

# GOOD: Selective suppression
class SafeContext:
    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type is None:
            return False
        # Only suppress expected exceptions
        return issubclass(exc_type, (ValueError, KeyError))

# BAD: Stateful generator context manager
@contextmanager
def bad_counter():
    count = 0
    def increment():
        nonlocal count
        count += 1
    yield increment
    print(f"Final count: {count}")
# Doesn't work - generator is consumed after first use!

# GOOD: Use class for stateful context managers
class Counter:
    def __init__(self):
        self.count = 0
    
    def __enter__(self):
        return self
    
    def increment(self):
        self.count += 1
    
    def __exit__(self, *args):
        print(f"Final count: {self.count}")
        return False

Key Takeaways

  • Context managers encode lifecycle guarantees through __enter__ and __exit__
  • Always use try/finally in __exit__ to ensure cleanup happens
  • Prefer @contextmanager decorator for simplicity, classes for stateful resources
  • Use ExitStack for dynamic or conditional resource management
  • Exception suppression is powerful but dangerous - always check exc_type
  • Protocols enable duck-typed interfaces without inheritance
  • Async context managers (async with) are essential for async I/O
  • Context managers compose cleanly and make error-prone resource management safe by default
What's Next?

You've mastered context managers! Now let's explore C extensions to achieve maximum performance by integrating with C/C++ code.

  • Cython - Write C extensions with Python-like syntax
  • ctypes & CFFI - Call C libraries from Python
  • Performance Optimization - Achieve C-level performance in Python