Advanced Capstone Project

Build a high-performance, concurrent, production-ready system using asyncio and the producer-consumer pattern.

Introduction

Welcome to the advanced capstone project! This challenge will test your mastery of Python's most powerful features: asynchronous programming, concurrency, and advanced design patterns. You'll build a production-grade system that handles multiple tasks concurrently using the producer-consumer pattern with asyncio and queues. This project demonstrates real-world architectures used in high-performance applications, microservices, and distributed systems.

Project Purpose

The goal is to create a concurrent task processing system using the producer-consumer pattern. Producers will generate tasks and place them in an asyncio queue, while consumers will process these tasks concurrently. The system must handle multiple producers and consumers running simultaneously.

Why This Pattern Matters

The producer-consumer pattern is fundamental in modern software architecture. You'll be evaluated on:

  • Asynchronous Programming, Using async/await, coroutines, and event loops effectively
  • Concurrency Management, Handling multiple tasks without race conditions or deadlocks
  • Queue Operations, Properly using asyncio.Queue for thread-safe communication
  • Error Handling, Gracefully managing exceptions in concurrent contexts
  • Performance, Optimizing for throughput and resource utilization

This pattern is used in message queues, web scrapers, data pipelines, real-time processors, and countless production systems. Mastering it is essential for building scalable applications.

First Steps

Step 1: Create Your Project Folder

Start by creating a dedicated folder for your advanced capstone project:

$ mkdir async_producer_consumer
$ cd async_producer_consumer

Step 2: Create the Virtual Environment

Set up an isolated Python environment for your project:

$ pip install --upgrade pip virtualenv
$ virtualenv --python=python3.11 .venv
# or python -m virtualenv --python=python3.11 .venv
$ source .venv/bin/activate

Step 3: Install Required Dependencies

This project requires click and environs for CLI and configuration. Create a requirements.txt file:

click
environs

Then install them using:

$ pip install -r requirements.txt

click

For creating command-line interfaces with professional argument parsing and command organization.

environs

For environment variable management with validation and type parsing from .env files.

Project Structure

Let's create a well-organized structure for our concurrent task processing system. We'll separate concerns into dedicated modules for producers, consumers, and the queue management.

Create the Service Folder and Core Files

Create a service folder with files for our async components:

$ mkdir service
$ cd service
$ touch __init__.py producer.py consumer.py main.py

producer.py

Contains the Producer class that generates tasks and adds them to the queue asynchronously.

consumer.py

Contains the Consumer class that processes tasks from the queue concurrently.

main.py

The orchestrator that creates the queue, spawns producers/consumers, and runs the async system.

Implementing producer.py

The Producer class generates tasks asynchronously and adds them to the shared queue. This demonstrates async/await patterns and queue operations.

producer.py - Complete Implementation

# -*- coding: utf-8 -*-

import asyncio
import logging
from datetime import datetime


class Producer:
    """Asynchronous producer that generates tasks"""

    def __init__(
        self,
        queue: asyncio.Queue,
        name: str,
        num_tasks: int = 5,
        delay: float = 0.5
    ):
        self.queue = queue
        self.name = name
        self.num_tasks = num_tasks
        self.delay = delay
        self.logger = logging.getLogger(self.name)

    async def produce(self) -> None:
        """Generate and enqueue tasks asynchronously"""
        self.logger.info(f"Producer {self.name} starting...")

        for i in range(self.num_tasks):
            task_data = {
                "id": f"{self.name}-task-{i}",
                "timestamp": datetime.now().isoformat(),
                "data": f"Data from {self.name}"
            }

            await self.queue.put(task_data)
            self.logger.info(
                f"Producer {self.name} produced: {task_data['id']}"
            )

            # Simulate work with async sleep
            await asyncio.sleep(self.delay)

        self.logger.info(f"Producer {self.name} finished")

What's Happening Here?

  • Async Method, The produce() method is defined as async def, making it a coroutine that can be awaited and run concurrently with other coroutines.
  • Queue Operations, Uses await self.queue.put() to add tasks to the asyncio.Queue, which is thread-safe and designed for async operations.
  • Non-Blocking Delays, await asyncio.sleep() yields control back to the event loop, allowing other tasks to run concurrently instead of blocking.
  • Type Hints, Proper typing with asyncio.Queue, str, int, and float makes the code more maintainable and IDE-friendly.

Implementing consumer.py

The Consumer class retrieves tasks from the queue and processes them asynchronously. Multiple consumers can run concurrently, processing tasks as they become available.

consumer.py - Complete Implementation

# -*- coding: utf-8 -*-

import asyncio
import logging
from typing import Dict, Any


class Consumer:
    """Asynchronous consumer that processes tasks from queue"""

    def __init__(
        self,
        queue: asyncio.Queue,
        name: str,
        processing_time: float = 1.0
    ):
        self.queue = queue
        self.name = name
        self.processing_time = processing_time
        self.logger = logging.getLogger(self.name)
        self.tasks_processed = 0

    async def consume(self) -> None:
        """Consume and process tasks from the queue"""
        self.logger.info(f"Consumer {self.name} starting...")

        while True:
            try:
                # Wait for a task with timeout
                task_data = await asyncio.wait_for(
                    self.queue.get(),
                    timeout=3.0
                )

                # Process the task
                await self._process_task(task_data)

                # Mark task as done
                self.queue.task_done()
                self.tasks_processed += 1

            except asyncio.TimeoutError:
                # No more tasks in queue, exit
                self.logger.info(
                    f"Consumer {self.name} finished. "
                    f"Processed {self.tasks_processed} tasks"
                )
                break
            except Exception as error:
                self.logger.error(
                    f"Consumer {self.name} error: {error}"
                )

    async def _process_task(self, task_data: Dict[str, Any]) -> None:
        """Process a single task"""
        self.logger.info(
            f"Consumer {self.name} processing: {task_data['id']}"
        )

        # Simulate processing time
        await asyncio.sleep(self.processing_time)

        self.logger.info(
            f"Consumer {self.name} completed: {task_data['id']}"
        )

What's Happening Here?

  • Infinite Loop with Timeout, Uses while True with asyncio.wait_for()to continuously process tasks until the queue is empty for 3 seconds.
  • Queue.get() and task_done(), await self.queue.get() retrieves tasks, andself.queue.task_done() signals completion, enabling proper queue tracking.
  • Exception Handling, Catches asyncio.TimeoutError to gracefully exit and generic Exception to log errors without crashing the consumer.
  • Task Tracking, Maintains a counter of processed tasks to provide useful metrics and demonstrate state management in async code.

Implementing main.py - Orchestration

The main.py file orchestrates the entire async system, creating the queue, spawning multiple producers and consumers, and running them concurrently with asyncio.gather().

main.py - Complete Implementation

# -*- coding: utf-8 -*-

import asyncio
import logging
import os

from .producer import Producer
from .consumer import Consumer


async def execute() -> None:
    """Main async execution orchestrator"""
    logger = logging.getLogger(
        name=os.getenv("APP_NAME", "AsyncProducerConsumer")
    )

    try:
        logger.info("Starting async producer-consumer system...")

        # Create shared queue with max size
        queue = asyncio.Queue(maxsize=10)

        # Create multiple producers
        producers = [
            Producer(queue, name=f"Producer-{i}", num_tasks=5, delay=0.3)
            for i in range(3)
        ]

        # Create multiple consumers
        consumers = [
            Consumer(queue, name=f"Consumer-{i}", processing_time=0.8)
            for i in range(2)
        ]

        # Run all producers and consumers concurrently
        await asyncio.gather(
            *[p.produce() for p in producers],
            *[c.consume() for c in consumers]
        )

        # Wait for all tasks to be processed
        await queue.join()

        logger.info("All tasks completed successfully!")

    except Exception as error:
        logger.error(f"System error: {error}")
        raise

    finally:
        logger.info("Async system shutdown complete")

What's Happening Here?

  • Shared Queue, Creates a single asyncio.Queue(maxsize=10) that all producers and consumers share, with a maximum size to prevent memory issues.
  • List Comprehensions, Uses list comprehensions to create multiple producer and consumer instances with unique names and configurations.
  • asyncio.gather(), Runs all coroutines concurrently using await asyncio.gather()with unpacked lists, enabling true parallel execution.
  • Queue.join(), await queue.join() waits until all tasks are marked as done via task_done(), ensuring complete processing.

Concurrency Pattern Demonstrated

This demonstrates the producer-consumer pattern with multiple producers (3) and consumers (2) running concurrently. The queue acts as a buffer, decoupling production from consumption and enabling efficient parallel processing.

Creating the Entry Point - manager.py

Create manager.py at the root level to provide a CLI entry point that bridges synchronous CLI code with our async execution system.

manager.py - Complete Implementation

# -*- coding: utf-8 -*-

import asyncio
import logging
import os

from click.core import CommandCollection
from click.decorators import group
from environs import Env


# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)


@group()
def cli_main():
    pass


@cli_main.command("run")
def main():
    """Run the async producer-consumer system"""
    from service.main import execute

    # Run async code from sync context
    asyncio.run(execute())
    exit()


if __name__ == "__main__":
    Env().read_env(os.path.join(os.getcwd(), ".env"))
    CommandCollection(sources=[cli_main])()

What's Happening Here?

  • asyncio.run(), Bridges synchronous CLI code with async execution using asyncio.run(), which creates an event loop, runs the coroutine, and cleans up automatically.
  • Logging Configuration, Sets up logging at the root level with a clear format showing timestamps, logger names, and levels for all async components.
  • Lazy Import, Imports execute inside the function to isolate errors during service initialization, preventing failures in one component from crashing the entire CLI.

How to Run Your Project

Execute your async producer-consumer system from the command line. When you run your async system, you'll see interleaved logs from producers and consumers demonstrating concurrent execution:

$ python manager.py run

This will start 3 producers and 2 consumers running concurrently, processing tasks through the shared queue.

Expected Output:
2024-01-19 14:32:10 - Producer-0 - INFO - Producer Producer-0 starting...
2024-01-19 14:32:10 - Producer-1 - INFO - Producer Producer-1 starting...
2024-01-19 14:32:10 - Producer-2 - INFO - Producer Producer-2 starting...
2024-01-19 14:32:10 - Consumer-0 - INFO - Consumer Consumer-0 starting...
2024-01-19 14:32:10 - Consumer-1 - INFO - Consumer Consumer-1 starting...
2024-01-19 14:32:10 - Producer-0 - INFO - Producer Producer-0 produced: Producer-0-task-0
2024-01-19 14:32:10 - Producer-1 - INFO - Producer Producer-1 produced: Producer-1-task-0
2024-01-19 14:32:10 - Consumer-0 - INFO - Consumer Consumer-0 processing: Producer-0-task-0
2024-01-19 14:32:10 - Consumer-1 - INFO - Consumer Consumer-1 processing: Producer-1-task-0
2024-01-19 14:32:11 - Producer-2 - INFO - Producer Producer-2 produced: Producer-2-task-0
2024-01-19 14:32:11 - Consumer-0 - INFO - Consumer Consumer-0 completed: Producer-0-task-0
2024-01-19 14:32:11 - Consumer-0 - INFO - Consumer Consumer-0 processing: Producer-2-task-0
...
2024-01-19 14:32:15 - Consumer-0 - INFO - Consumer Consumer-0 finished. Processed 8 tasks
2024-01-19 14:32:15 - Consumer-1 - INFO - Consumer Consumer-1 finished. Processed 7 tasks
2024-01-19 14:32:15 - AsyncProducerConsumer - INFO - All tasks completed successfully!
2024-01-19 14:32:15 - AsyncProducerConsumer - INFO - Async system shutdown complete

What You're Seeing

Notice how the logs are interleaved - producers and consumers run simultaneously. This proves true concurrent execution. The consumers process tasks as soon as they're available, demonstrating the efficiency of the producer-consumer pattern.

With 3 producers creating 5 tasks each (15 total) and 2 consumers processing them, the workload is distributed efficiently, showcasing async Python's power for I/O-bound operations.

Bonus: Python Project Template

Want to see a complete, production-ready example? Check out our Python project template that demonstrates best practices, proper structure, and professional patterns for real-world applications.

The template includes:

  • Docker configuration - Containerized development and deployment
  • CI/CD pipelines - Automated testing and deployment workflows
  • Comprehensive testing - Unit tests, integration tests, and coverage reports
  • Enterprise architecture - Professional project organization and structure