Multiprocessing & Parallelism

Process pools, shared memory, and true parallel execution.

True Parallel Processing in Python

Multiprocessing bypasses the Global Interpreter Lock (GIL) by creating separate Python processes, each with its own interpreter and memory space. This enables true parallel execution on multiple CPU cores, making it ideal for CPU-intensive computations.

When to Use Multiprocessing:

  • ✓ CPU-bound tasks: Data processing, calculations, image manipulation, ML training
  • ✓ Parallel computation: Tasks that can run independently across cores
  • ✓ Bypassing GIL: When threading doesn't provide speedup for CPU work
  • ✗ I/O-bound tasks: Use threading or asyncio instead (lower overhead)
  • ✗ Frequent data sharing: Inter-process communication is expensive

Threading vs Multiprocessing

Threading

Shared memory: All threads share memory

GIL limitation: One thread executes at a time

Lightweight: Low overhead, fast creation

Best for: I/O-bound operations

Memory: [███ Shared ███]
Thread 1 → Same GIL
Thread 2 → Same GIL
Thread 3 → Same GIL
Multiprocessing

Separate memory: Each process isolated

No GIL: True parallel execution

Heavyweight: Higher overhead

Best for: CPU-bound operations

Process 1: [██] Own memory
Process 2: [██] Own memory
Process 3: [██] Own memory
All run in parallel!

Basic Multiprocessing

Creating and managing processes with the multiprocessing module.

import multiprocessing as mp
import os
import time

def worker(name):
    """Function to run in separate process"""
    print(f"Worker {name} started in PID: {os.getpid()}")
    time.sleep(2)
    print(f"Worker {name} finished")

if __name__ == '__main__':
    # IMPORTANT: Always use if __name__ == '__main__' guard
    # Required on Windows, good practice everywhere

    print(f"Main process PID: {os.getpid()}")

    # Create process
    process = mp.Process(target=worker, args=('A',))

    # Start process (creates new Python interpreter)
    process.start()
    print("Main process continues...")

    # Wait for process to complete
    process.join()
    print("Process completed!")

    # Multiple processes
    processes = []
    for i in range(4):
        p = mp.Process(target=worker, args=(f"Worker-{i}",))
        processes.append(p)
        p.start()

    # Wait for all processes
    for p in processes:
        p.join()

    print("All processes completed!")
    print(f"CPU cores available: {mp.cpu_count()}")

How Processes Start: fork, spawn & forkserver

A subtle but important detail: how a child process is created affects correctness, performance, and portability. Python offers three start methods, and the default differs by platform.

fork

Clones the parent (copy-on-write). Very fast, Unix-only. Unsafe if the parent holds locks in other threads. Historically the Linux default.

spawn

Launches a fresh interpreter and re-imports your module. Safe and portable, the default on macOS and Windows. Slower startup; everything must be picklable.

forkserver

Forks children from a clean helper process with no extra threads. Unix-only, combines fork's speed with much of spawn's safety.

import multiprocessing as mp

def square(x):
    return x * x

if __name__ == '__main__':
    # Three ways to start a process:
    #   'fork'       - clone the parent process (fast; Unix only; unsafe with threads)
    #   'spawn'      - launch a fresh interpreter (safe; default on macOS & Windows)
    #   'forkserver' - fork from a clean helper process (Unix; safe with threads)

    # Option A: set the default once (guard it with __main__)
    mp.set_start_method('spawn')

    # Option B: get an explicit context WITHOUT changing the global default
    ctx = mp.get_context('spawn')
    with ctx.Pool(processes=4) as pool:
        print(pool.map(square, range(8)))

# Why it matters:
# - 'spawn' re-imports your module in every child, so everything the child
#   needs must be importable and behind if __name__ == '__main__'.
# - 'fork' is fast (copy-on-write memory) but can deadlock if the parent held
#   a lock in another thread. Python 3.14 stops defaulting to 'fork' on Linux.
Heads up: because spawn re-imports your module, module-level code runs again in every child, which is exactly why the if __name__ == '__main__' guard is mandatory. Python 3.14 changes the Linux default away from fork for safety, so writing spawn-compatible code today keeps you future-proof.

Pool Methods: map, imap, apply_async & chunksize

Pool offers several ways to dispatch work. Picking the right one, and tuning chunksize, has a big impact on throughput.

import multiprocessing as mp

def process(n):
    return n * n

if __name__ == '__main__':
    data = range(1000)

    with mp.Pool(processes=4) as pool:
        # map(): blocks, returns a list IN ORDER
        results = pool.map(process, data)

        # chunksize: send fewer, bigger batches to cut IPC overhead
        results = pool.map(process, data, chunksize=50)

        # imap(): lazy iterator, ordered - great for streaming huge inputs
        for r in pool.imap(process, data, chunksize=50):
            ...  # handle each result as it arrives

        # imap_unordered(): yields results AS THEY FINISH (fastest first result)
        for r in pool.imap_unordered(process, data):
            ...

        # apply_async(): run a single call in the background, collect later
        future = pool.apply_async(process, args=(10,))
        print(future.get(timeout=5))  # 100
MethodReturnsOrderBest for
maplist (blocks)Input orderSimple "process everything, wait" jobs
imaplazy iteratorInput orderStreaming huge inputs without buffering all results
imap_unorderedlazy iteratorAs completedGetting results ASAP when order does not matter
apply_asyncAsyncResultN/A (single call)One-off background calls; collect with .get()
chunksize is the hidden lever: each task dispatched to a worker costs a pickle + IPC round trip. For many small items, a larger chunksize batches them together and can turn a multiprocessing slowdown into a real speedup.

The Pickling Boundary

Because processes do not share memory, anything crossing the boundary, the target function, its arguments, and its return value, is serialized with pickle. This is the single most common source of confusing multiprocessing errors.

Not picklable
  • Lambdas and nested/local functions
  • Closures capturing non-picklable state
  • Open files, sockets, DB connections
  • Most locks and thread objects
Picklable
  • Top-level (module-level) functions
  • Plain data: numbers, strings, lists, dicts
  • Dataclasses / objects with picklable fields
  • Results returned as plain data
import multiprocessing as mp

# The function, its arguments, AND its return value all cross the process
# boundary - so they must be picklable.

# The offending function passed to a top-level lambda:
run = lambda x: x * x  # a lambda cannot be pickled

# Won't work: lambdas / local (nested) functions are not picklable
def build_bad():
    with mp.Pool() as pool:
        return pool.map(run, range(5))
        # PicklingError / AttributeError: Can't pickle ...

# Works: top-level (module-level) functions ARE picklable
def square(x):
    return x * x

if __name__ == '__main__':
    with mp.Pool() as pool:
        print(pool.map(square, range(5)))  # [0, 1, 4, 9, 16]

# Common non-picklable culprits: lambdas, closures, open files/sockets,
# database connections, and most locks. Pass plain data, not live handles.

Multiprocessing Best Practices

✓ Do This
  • Use it for CPU-bound work (I/O-bound → threading/asyncio)
  • Guard entry points with if __name__ == '__main__'
  • Prefer Pool / ProcessPoolExecutor over manual processes
  • Tune chunksize for many small tasks
  • Pass plain, picklable data, not live handles
  • Write spawn-compatible code for portability
  • Prefer Value/Array over Manager when speed matters
  • Profile: confirm the speedup beats the overhead
✗ Avoid This
  • Multiprocessing for I/O-bound tasks (overhead wins)
  • Spawning more processes than CPU cores for CPU work
  • Passing lambdas / closures to workers
  • Sharing large objects through Manager in hot loops
  • Forgetting the __main__ guard (spawn re-import loop)
  • Assuming shared memory, each process is isolated
  • Ignoring startup/pickle cost for tiny tasks
  • Relying on fork-specific behavior in portable code

Key Takeaways

  • Multiprocessing bypasses the GIL - true parallel execution on multiple CPU cores
  • Perfect for CPU-bound tasks - data processing, calculations, simulations
  • Use Pool for most cases - high-level interface, automatic worker management
  • Processes have separate memory - use Queue, Pipe, or shared memory for IPC
  • Always use if __name__ == '__main__' guard - prevents infinite process creation
  • Chunk your data - reduces overhead for large datasets
  • Value/Array for speed, Manager for flexibility - choose based on needs
  • Know your start method - fork (fast, Unix), spawn (safe, portable), forkserver; 3.14 moves off fork on Linux
  • Pick the right Pool method - map, imap, imap_unordered, apply_async; tune chunksize to cut IPC cost
  • Everything crossing the boundary must be picklable - no lambdas, closures, or live handles
  • Profile to verify speedup - overhead can negate benefits for small tasks

Practice Exercises

Exercise 1: Parallel Prime Finder

Create a parallel prime number finder that searches for all primes between 1 and 10,000,000. Use a process pool to divide the range into chunks. Compare performance with sequential implementation and calculate speedup.

Exercise 2: Matrix Multiplication

Implement parallel matrix multiplication for two 1000x1000 matrices using shared memory. Divide the work by rows, with each process computing a subset of rows.

Exercise 3: Log File Analyzer

Build a parallel log file analyzer that processes multiple large log files simultaneously. Each process should count error messages and extract statistics using Queue for results.

Additional Resources

  • Official Docs: docs.python.org/3/library/multiprocessing.html
  • concurrent.futures: docs.python.org/3/library/concurrent.futures.html (ProcessPoolExecutor)
  • Start methods: "Contexts and start methods" in the multiprocessing docs
  • Shared memory: multiprocessing.shared_memory for zero-copy arrays (3.8+)
  • Higher-level: Joblib and Dask for parallel data pipelines
  • Talk: "Parallel Python" - understanding when parallelism actually pays off
What's Next?

You've mastered multiprocessing! Now let's dive deep into Python's object model with metaclasses and descriptors for advanced metaprogramming.

  • Metaclasses - Control class creation and customize class behavior
  • Descriptors - Implement property access with __get__, __set__, __delete__
  • Metaprogramming - Build frameworks and DSLs with Python's object model
Python AdvancedLesson 4