Multiprocessing & Parallelism

Process pools, shared memory, IPC, and true parallel execution for CPU-bound tasks

True Parallel Processing in Python

Multiprocessing bypasses the Global Interpreter Lock (GIL) by creating separate Python processes, each with its own interpreter and memory space. This enables true parallel execution on multiple CPU cores, making it ideal for CPU-intensive computations.

When to Use Multiprocessing:

  • ✓ CPU-bound tasks: Data processing, calculations, image manipulation, ML training
  • ✓ Parallel computation: Tasks that can run independently across cores
  • ✓ Bypassing GIL: When threading doesn't provide speedup for CPU work
  • ✗ I/O-bound tasks: Use threading or asyncio instead (lower overhead)
  • ✗ Frequent data sharing: Inter-process communication is expensive

Threading vs Multiprocessing

Threading

Shared memory: All threads share memory

GIL limitation: One thread executes at a time

Lightweight: Low overhead, fast creation

Best for: I/O-bound operations

Memory: [███ Shared ███]
Thread 1 → Same GIL
Thread 2 → Same GIL
Thread 3 → Same GIL
Multiprocessing

Separate memory: Each process isolated

No GIL: True parallel execution

Heavyweight: Higher overhead

Best for: CPU-bound operations

Process 1: [██] Own memory
Process 2: [██] Own memory
Process 3: [██] Own memory
All run in parallel!

Basic Multiprocessing

Creating and managing processes with the multiprocessing module.

import multiprocessing as mp
import os
import time

def worker(name):
    """Function to run in separate process"""
    print(f"Worker {name} started in PID: {os.getpid()}")
    time.sleep(2)
    print(f"Worker {name} finished")

if __name__ == '__main__':
    # IMPORTANT: Always use if __name__ == '__main__' guard
    # Required on Windows, good practice everywhere

    print(f"Main process PID: {os.getpid()}")

    # Create process
    process = mp.Process(target=worker, args=('A',))

    # Start process (creates new Python interpreter)
    process.start()
    print("Main process continues...")

    # Wait for process to complete
    process.join()
    print("Process completed!")

    # Multiple processes
    processes = []
    for i in range(4):
        p = mp.Process(target=worker, args=(f"Worker-{i}",))
        processes.append(p)
        p.start()

    # Wait for all processes
    for p in processes:
        p.join()

    print("All processes completed!")
    print(f"CPU cores available: {mp.cpu_count()}")

Key Takeaways

  • Multiprocessing bypasses the GIL - true parallel execution on multiple CPU cores
  • Perfect for CPU-bound tasks - data processing, calculations, simulations
  • Use Pool for most cases - high-level interface, automatic worker management
  • Processes have separate memory - use Queue, Pipe, or shared memory for IPC
  • Always use if __name__ == '__main__' guard - prevents infinite process creation
  • Chunk your data - reduces overhead for large datasets
  • Value/Array for speed, Manager for flexibility - choose based on needs
  • Profile to verify speedup - overhead can negate benefits for small tasks

Practice Exercises

Exercise 1: Parallel Prime Finder

Create a parallel prime number finder that searches for all primes between 1 and 10,000,000. Use a process pool to divide the range into chunks. Compare performance with sequential implementation and calculate speedup.

Exercise 2: Matrix Multiplication

Implement parallel matrix multiplication for two 1000x1000 matrices using shared memory. Divide the work by rows, with each process computing a subset of rows.

Exercise 3: Log File Analyzer

Build a parallel log file analyzer that processes multiple large log files simultaneously. Each process should count error messages and extract statistics using Queue for results.

What's Next?

You've mastered multiprocessing! Now let's dive deep into Python's object model with metaclasses and descriptors for advanced metaprogramming.

  • Metaclasses - Control class creation and customize class behavior
  • Descriptors - Implement property access with __get__, __set__, __delete__
  • Metaprogramming - Build frameworks and DSLs with Python's object model
Python AdvancedLesson 4