Performance & Profiling

Write fast code by measuring first and optimizing smart

Why Performance Matters

Performance issues rarely stem from "slow hardware." They typically originate from inefficient algorithms, unnecessary work, or hidden bottlenecks. Professional developers measure performance scientifically, identify hot spots with data, and optimize only where it provides real value.

The Performance Optimization Process:

Measure: Profile to find the actual bottlenecks
Analyze: Understand why code is slow
Optimize: Apply targeted improvements
Verify: Measure again to confirm improvement
Repeat: Only if performance is still inadequate

The Golden Rules of Optimization

1️⃣

Make It Work

Write correct, readable code first. Don't optimize prematurely.

2️⃣

Make It Right

Test thoroughly. Bugs are worse than slow code.

3️⃣

Make It Fast

Only if needed. Measure first, optimize the bottlenecks.

Donald Knuth's famous quote: "Premature optimization is the root of all evil." Most code doesn't need optimization. Profile first to find the 5-10% that does.

1. Measuring Performance: Time & Timeit

Before optimizing, you need baseline measurements. Python provides several tools for timing code execution.

Using time.perf_counter()

import time

def slow_function():
    total = 0
    for i in range(1_000_000):
        total += i
    return total

# Basic timing
start = time.perf_counter()
result = slow_function()
end = time.perf_counter()

print(f"Elapsed time: {end - start:.4f} seconds")
# Output: Elapsed time: 0.0523 seconds

# Context manager for cleaner timing
from contextlib import contextmanager

@contextmanager
def timer(name="Operation"):
    start = time.perf_counter()
    yield
    end = time.perf_counter()
    print(f"{name} took {end - start:.4f}s")

# Usage
with timer("Slow function"):
    result = slow_function()
# Output: Slow function took 0.0521s

Using timeit Module

import timeit

# Compare different approaches
# Approach 1: List concatenation
time1 = timeit.timeit(
    stmt="result = []\nfor i in range(1000): result = result + [i]",
    number=1000
)

# Approach 2: List append
time2 = timeit.timeit(
    stmt="result = []\nfor i in range(1000): result.append(i)",
    number=1000
)

# Approach 3: List comprehension
time3 = timeit.timeit(
    stmt="result = [i for i in range(1000)]",
    number=1000
)

print(f"Concatenation: {time1:.4f}s")  # Slowest
print(f"Append:        {time2:.4f}s")  # Medium
print(f"Comprehension: {time3:.4f}s")  # Fastest

# Using repeat for more accurate results
times = timeit.repeat(
    stmt="sum(range(1000))",
    repeat=5,
    number=10000
)
print(f"Best of 5: {min(times):.6f}s")
print(f"Average:   {sum(times)/len(times):.6f}s")

When to use each:

time.perf_counter() - Quick timing for larger code blocks
timeit - Accurate microbenchmarks for small snippets
timeit.repeat() - Multiple runs to reduce variance

2. Profiling: Finding the Bottlenecks

Profilers show exactly where your program spends its time, revealing the functions that are actual bottlenecks rather than the ones you think are slow.

cProfile: Built-in Profiler

import cProfile
import pstats
from io import StringIO

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

def calculate_all():
    results = []
    for i in range(30):
        results.append(fibonacci(i))
    return results

# Method 1: Quick profiling
cProfile.run('calculate_all()')

# Method 2: Save and analyze results
profiler = cProfile.Profile()
profiler.enable()

calculate_all()

profiler.disable()

# Print stats
stats = pstats.Stats(profiler)
stats.strip_dirs()
stats.sort_stats('cumulative')
stats.print_stats(10)  # Top 10 functions

Expected Output:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
4356586/30    1.205    0.000    1.205    0.000 profiler.py:5(fibonacci)
         1    0.000    0.000    1.205    1.205 profiler.py:9(calculate_all)
...

🔥 fibonacci() is called 4.4M times and takes 100% of the time!

Line Profiler: Line-by-Line Analysis

# Install: pip install line_profiler

# Add @profile decorator to function you want to profile
# Run with: kernprof -l -v script.py
@profile
def process_data(items):
    result = []
    for item in items:
        # Expensive operation
        processed = complex_calculation(item)
        result.append(processed)
    return result

Expected Output:

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           def process_data(items):
     3         1          2.0      2.0      0.0      result = []
     4      1000       1205.0      1.2      0.5      for item in items:
     5      1000     245032.0    245.0     99.4          processed = complex_calculation(item)
     6      1000       1532.0      1.5      0.6          result.append(processed)
     7         1          1.0      1.0      0.0      return result

🔥 Line 5 takes 99.4% of the time!

Reading profiler output:

ncalls: Number of times function was called
tottime: Total time spent in this function (excluding subcalls)
cumtime: Cumulative time (including subcalls)
percall: Average time per call

Focus on functions with high cumtime and many ncalls.

3. Common Performance Bottlenecks

Understanding common performance killers helps you write better code from the start.

4. Proven Optimization Techniques

Use Built-in Functions and Data Structures

# Built-ins are implemented in C and highly optimized

# Bad: Manual implementation
def sum_slow(numbers):
    total = 0
    for n in numbers:
        total += n
    return total

# Good: Built-in
def sum_fast(numbers):
    return sum(numbers)

# Bad: List for membership testing
def has_duplicate_slow(items):
    seen = []
    for item in items:
        if item in seen:  # O(n) lookup
            return True
        seen.append(item)
    return False

# Good: Set for membership testing
def has_duplicate_fast(items):
    seen = set()
    for item in items:
        if item in seen:  # O(1) lookup
            return True
        seen.add(item)
    return False

Caching with functools.lru_cache

from functools import lru_cache

# Without caching - exponential time complexity
def fibonacci_slow(n):
    if n <= 1:
        return n
    return fibonacci_slow(n-1) + fibonacci_slow(n-2)

# fibonacci_slow(35) takes ~5 seconds

# With caching - linear time complexity
@lru_cache(maxsize=None)
def fibonacci_fast(n):
    if n <= 1:
        return n
    return fibonacci_fast(n-1) + fibonacci_fast(n-2)

# fibonacci_fast(35) takes ~0.0001 seconds (50,000x faster!)

# Check cache stats
print(fibonacci_fast.cache_info())
# CacheInfo(hits=33, misses=36, maxsize=None, currsize=36)

# Clear cache if needed
fibonacci_fast.cache_clear()

# Real-world example: API calls
@lru_cache(maxsize=128)
def fetch_user_data(user_id):
    # Expensive API call
    response = requests.get(f"https://api.example.com/users/{user_id}")
    return response.json()

# First call: makes API request
user = fetch_user_data(123)

# Second call: returns cached result instantly
user = fetch_user_data(123)  # No API call!

Generators for Memory Efficiency

# Bad: Load everything into memory
def process_large_file_slow(filename):
    with open(filename) as f:
        lines = f.readlines()  # Loads entire file!
    
    results = []
    for line in lines:
        results.append(process_line(line))
    return results

# Good: Process one line at a time
def process_large_file_fast(filename):
    def line_generator():
        with open(filename) as f:
            for line in f:  # Lazy iteration
                yield process_line(line)
    return line_generator()

# Usage
for result in process_large_file_fast('huge_file.txt'):
    print(result)  # Memory stays constant!

# Generator expressions vs list comprehensions
# Bad: Creates entire list in memory
squares_list = [x**2 for x in range(1_000_000)]

# Good: Creates values on demand
squares_gen = (x**2 for x in range(1_000_000))

# Use case: Sum of squares
total = sum(x**2 for x in range(1_000_000))  # Memory efficient!

List Comprehensions vs Loops

# Comprehensions are faster than equivalent loops

# Slow
result = []
for x in range(1000):
    if x % 2 == 0:
        result.append(x ** 2)

# Fast (30-40% faster)
result = [x ** 2 for x in range(1000) if x % 2 == 0]

# Dict comprehension
# Slow
word_lengths = {}
for word in words:
    word_lengths[word] = len(word)

# Fast
word_lengths = {word: len(word) for word in words}

# Set comprehension
# Slow
unique_lengths = set()
for word in words:
    unique_lengths.add(len(word))

# Fast
unique_lengths = {len(word) for word in words}

Performance hierarchy (fastest to slowest):

Built-in C functions (sum, min, max, etc.)
List/dict/set comprehensions
Generator expressions (for memory efficiency)
map/filter functions
Regular for loops

5. Memory Profiling

Sometimes performance issues stem from excessive memory usage, causing swapping or out-of-memory errors.

tracemalloc (Built-in)

import tracemalloc

# Start tracking
tracemalloc.start()

# Your code here
data = [i ** 2 for i in range(1_000_000)]

# Get current memory usage
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory: {current / 1024 / 1024:.2f} MB")
print(f"Peak memory: {peak / 1024 / 1024:.2f} MB")

# Get top memory allocations
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

print("Top 10 memory allocations:")
for stat in top_stats[:10]:
    print(stat)

tracemalloc.stop()

memory_profiler (External)

# Install: pip install memory_profiler
# Run with: python -m memory_profiler script.py

from memory_profiler import profile

@profile
def memory_heavy_function():
    # Creates large list
    a = [1] * (10 ** 6)

    # Creates another large list
    b = [2] * (2 * 10 ** 7)

    # Delete first list
    del a

    return sum(b)

Expected Output:

Line #    Mem usage    Increment  Occurrences   Line Contents
============================================================
     3     38.0 MiB     38.0 MiB           1   @profile
     4                                         def memory_heavy_function():
     5     45.7 MiB      7.7 MiB           1       a = [1] * (10 ** 6)
     6    198.4 MiB    152.7 MiB           1       b = [2] * (2 * 10 ** 7)
     7    190.7 MiB     -7.7 MiB           1       del a
     8    190.7 MiB      0.0 MiB           1       return sum(b)

Memory optimization tips:

Use generators instead of lists when possible
Process data in chunks for large files
Delete large objects when no longer needed
Use __slots__ for classes with many instances
Consider numpy arrays for numerical data (more memory efficient)

Understanding Time Complexity (Big O)

Understanding algorithmic complexity helps you predict performance as data grows.

Complexity	Name	Example	10 items	1000 items
O(1)	Constant	Dict lookup, array access	1 op	1 op
O(log n)	Logarithmic	Binary search	3 ops	10 ops
O(n)	Linear	Loop through list	10 ops	1,000 ops
O(n log n)	Linearithmic	Efficient sorting	30 ops	10,000 ops
O(n²)	Quadratic	Nested loops	100 ops	1,000,000 ops
O(2ⁿ)	Exponential	Recursive fibonacci	1,024 ops	💥 Unusable

# Common Python operations and their complexity

# O(1) - Constant
item = my_dict[key]           # Dict/set lookup
item = my_list[5]             # List index access
my_list.append(item)          # List append
my_set.add(item)              # Set add

# O(log n) - Logarithmic
bisect.bisect_left(sorted_list, x)  # Binary search

# O(n) - Linear
sum(my_list)                  # Iterate through list
max(my_list)                  # Find maximum
item in my_list               # List membership (use set instead!)
my_list.count(item)           # Count occurrences

# O(n log n) - Linearithmic
sorted(my_list)               # Efficient sort
my_list.sort()                # In-place sort

# O(n²) - Quadratic (avoid!)
for i in range(n):
    for j in range(n):
        # nested loop operations

# Choosing the right data structure matters!
# List: O(n) for 'in', O(1) for append
# Set:  O(1) for 'in', O(1) for add
# Dict: O(1) for lookup, O(1) for insert

# Bad: O(n) membership test in loop = O(n²)
def filter_items_slow(items, allowed):
    return [item for item in items if item in allowed]  # allowed is list

# Good: O(1) membership test in loop = O(n)
def filter_items_fast(items, allowed):
    allowed_set = set(allowed)
    return [item for item in items if item in allowed_set]

Performance Best Practices

✓ Do This

Profile before optimizing
Use appropriate data structures
Cache expensive computations
Use built-in functions
Process data in batches
Use generators for large datasets
Avoid premature optimization
Measure after optimizing

✗ Avoid This

Optimizing without measuring
Using lists for membership tests
String concatenation in loops
Repeated expensive operations
Loading entire files into memory
Nested loops when unnecessary
Sacrificing readability for tiny gains
Ignoring algorithmic complexity

Real-World Optimization Example

Let's see a complete example of identifying and fixing a performance bottleneck.

# Problem: Process user activity logs and find top 10 most active users

# SLOW VERSION (takes 5 seconds for 100k records)
def find_top_users_slow(logs):
    user_activity = {}

    # Count activities per user
    for log in logs:
        user_id = log['user_id']
        if user_id in user_activity:
            user_activity[user_id] += 1
        else:
            user_activity[user_id] = 1

    # Sort to find top 10
    sorted_users = []
    for user_id, count in user_activity.items():
        sorted_users.append((count, user_id))

    sorted_users.sort(reverse=True)
    return sorted_users[:10]

# Profile reveals: 60% time in dict operations, 40% in sorting

# OPTIMIZED VERSION 1: Use defaultdict and better sorting
from collections import defaultdict

def find_top_users_faster(logs):
    user_activity = defaultdict(int)

    # Count activities - defaultdict is faster
    for log in logs:
        user_activity[log['user_id']] += 1

    # sorted() with key is faster than manual sorting
    return sorted(user_activity.items(),
                  key=lambda x: x[1],
                  reverse=True)[:10]

# Result: 3 seconds (40% improvement)

# OPTIMIZED VERSION 2: Use Counter and heapq
from collections import Counter
import heapq

def find_top_users_fastest(logs):
    # Counter is optimized for counting
    user_activity = Counter(log['user_id'] for log in logs)

    # heapq.nlargest is faster than full sort for small k
    return user_activity.most_common(10)

# Result: 1.5 seconds (70% improvement!)

# OPTIMIZED VERSION 3: Process in batches for memory efficiency
def find_top_users_memory_efficient(log_file):
    user_activity = Counter()

    # Process file in chunks instead of loading all into memory
    BATCH_SIZE = 10000
    batch = []

    with open(log_file) as f:
        for line in f:
            batch.append(json.loads(line))

            if len(batch) >= BATCH_SIZE:
                # Process batch
                user_activity.update(log['user_id'] for log in batch)
                batch = []

        # Process remaining
        if batch:
            user_activity.update(log['user_id'] for log in batch)

    return user_activity.most_common(10)

# Result: Same speed but uses 90% less memory!

# Lessons learned:
# 1. Use specialized data structures (Counter, defaultdict)
# 2. Use optimized algorithms (heapq.nlargest vs full sort)
# 3. Process data in batches for large files
# 4. Profile to verify improvements

Optimization results:

Version 1: 40% faster using better data structures
Version 2: 70% faster using specialized algorithms
Version 3: Same speed, 90% less memory usage

Total improvement: 3.3x faster with better memory efficiency!

Key Takeaways

Always measure first - profiling reveals the real bottlenecks
Focus optimization efforts - 90% of time is spent in 10% of code
Choose the right algorithm - O(n) vs O(n²) matters more than micro-optimizations
Use appropriate data structures - sets for membership, dicts for lookups
Leverage built-in functions - they're optimized in C
Cache expensive operations - don't recompute what you already know
Consider memory efficiency - use generators for large datasets
Readable code first - optimize only when necessary

Practice Exercises

Exercise 1: Profile and Optimize

Write a function that finds all pairs of numbers in a list that sum to a target. First implement a naive O(n²) solution, profile it, then optimize to O(n) using a set. Compare the performance with timeit for lists of 100, 1000, and 10000 elements.

Exercise 2: Memory Optimization

Create two versions of a function that processes a large CSV file (1M+ rows): one that loads everything into memory, and one that uses generators. Use tracemalloc to compare memory usage.

Exercise 3: Caching Benefits

Implement a function that calculates expensive statistics on data (mean, median, mode). Add @lru_cache and measure the speedup when the function is called multiple times with the same inputs. Check cache_info() to see hit rates.

Additional Resources

Books: "High Performance Python" by Gorelick & Ozsvald
Tools: py-spy (sampling profiler), scalene (CPU+memory profiler)
Visualization: snakeviz (visualize cProfile output)
Documentation: docs.python.org/3/library/profile.html
Time Complexity: wiki.python.org/moin/TimeComplexity

What's Next?

Your code is now fast and efficient! Next, learn how to package and share your Python projects with others.

Package Structure - Organize your code into installable packages
setup.py & pyproject.toml - Configure your package for distribution
Publishing to PyPI - Share your package with the Python community

Back to "Testing & Debugging"

Python IntermediateLesson 8

Next: "Packaging & Distribution"

Performance & Profiling

Why Performance Matters

The Golden Rules of Optimization

Make It Work

Make It Right

Make It Fast

1. Measuring Performance: Time & Timeit

Using time.perf_counter()

Using timeit Module

2. Profiling: Finding the Bottlenecks

cProfile: Built-in Profiler

Expected Output:

Line Profiler: Line-by-Line Analysis

Expected Output:

3. Common Performance Bottlenecks

1. Inefficient Algorithms

2. Repeated Work in Loops

3. String Concatenation

4. Excessive I/O Operations

4. Proven Optimization Techniques

Use Built-in Functions and Data Structures

Caching with functools.lru_cache

Generators for Memory Efficiency

List Comprehensions vs Loops

5. Memory Profiling

tracemalloc (Built-in)

memory_profiler (External)

Expected Output:

Understanding Time Complexity (Big O)

Performance Best Practices

✓ Do This

✗ Avoid This

Real-World Optimization Example

Key Takeaways

Practice Exercises

Exercise 1: Profile and Optimize

Exercise 2: Memory Optimization

Exercise 3: Caching Benefits

Additional Resources

What's Next?