Advanced Capstone Project
Build a high-performance, concurrent, production-ready system using asyncio and the producer-consumer pattern.
Introduction
Welcome to the advanced capstone project! This challenge will test your mastery of Python's most powerful features: asynchronous programming, concurrency, and advanced design patterns. You'll build a production-grade system that handles multiple tasks concurrently using the producer-consumer pattern with asyncio and queues. This project demonstrates real-world architectures used in high-performance applications, microservices, and distributed systems.
Project Purpose
The goal is to create a concurrent task processing system using the producer-consumer pattern. Producers will generate tasks and place them in an asyncio queue, while consumers will process these tasks concurrently. The system must handle multiple producers and consumers running simultaneously.
Why This Pattern Matters
The producer-consumer pattern is fundamental in modern software architecture. You'll be evaluated on:
- Asynchronous Programming, Using async/await, coroutines, and event loops effectively
- Concurrency Management, Handling multiple tasks without race conditions or deadlocks
- Queue Operations, Properly using asyncio.Queue for thread-safe communication
- Error Handling, Gracefully managing exceptions in concurrent contexts
- Performance, Optimizing for throughput and resource utilization
This pattern is used in message queues, web scrapers, data pipelines, real-time processors, and countless production systems. Mastering it is essential for building scalable applications.
First Steps
Step 1: Create Your Project Folder
Start by creating a dedicated folder for your advanced capstone project:
mkdir async_producer_consumercd async_producer_consumerStep 2: Create the Virtual Environment
Set up an isolated Python environment for your project:
pip install --upgrade pip virtualenvvirtualenv --python=python3.11 .venvor python -m virtualenv --python=python3.11 .venvsource .venv/bin/activateStep 3: Install Required Dependencies
This project requires click and environs for CLI and configuration. Create a requirements.txt file:
click environs
Then install them using:
pip install -r requirements.txtclick
For creating command-line interfaces with professional argument parsing and command organization.
environs
For environment variable management with validation and type parsing from .env files.
Project Structure
Let's create a well-organized structure for our concurrent task processing system. We'll separate concerns into dedicated modules for producers, consumers, and the queue management.
Create the Service Folder and Core Files
Create a service folder with files for our async components:
mkdir servicecd servicetouch __init__.py producer.py consumer.py main.pyproducer.py
Contains the Producer class that generates tasks and adds them to the queue asynchronously.
consumer.py
Contains the Consumer class that processes tasks from the queue concurrently.
main.py
The orchestrator that creates the queue, spawns producers/consumers, and runs the async system.
Implementing producer.py
The Producer class generates tasks asynchronously and adds them to the shared queue. This demonstrates async/await patterns and queue operations.
producer.py - Complete Implementation
# -*- coding: utf-8 -*-
import asyncio
import logging
from datetime import datetime
class Producer:
"""Asynchronous producer that generates tasks"""
def __init__(
self,
queue: asyncio.Queue,
name: str,
num_tasks: int = 5,
delay: float = 0.5
):
self.queue = queue
self.name = name
self.num_tasks = num_tasks
self.delay = delay
self.logger = logging.getLogger(self.name)
async def produce(self) -> None:
"""Generate and enqueue tasks asynchronously"""
self.logger.info(f"Producer {self.name} starting...")
for i in range(self.num_tasks):
task_data = {
"id": f"{self.name}-task-{i}",
"timestamp": datetime.now().isoformat(),
"data": f"Data from {self.name}"
}
await self.queue.put(task_data)
self.logger.info(
f"Producer {self.name} produced: {task_data['id']}"
)
# Simulate work with async sleep
await asyncio.sleep(self.delay)
self.logger.info(f"Producer {self.name} finished")What's Happening Here?
- Async Method, The
produce()method is defined asasync def, making it a coroutine that can be awaited and run concurrently with other coroutines. - Queue Operations, Uses
await self.queue.put()to add tasks to the asyncio.Queue, which is thread-safe and designed for async operations. - Non-Blocking Delays,
await asyncio.sleep()yields control back to the event loop, allowing other tasks to run concurrently instead of blocking. - Type Hints, Proper typing with
asyncio.Queue,str,int, andfloatmakes the code more maintainable and IDE-friendly.
Implementing consumer.py
The Consumer class retrieves tasks from the queue and processes them asynchronously. Multiple consumers can run concurrently, processing tasks as they become available.
consumer.py - Complete Implementation
# -*- coding: utf-8 -*-
import asyncio
import logging
from typing import Dict, Any
class Consumer:
"""Asynchronous consumer that processes tasks from queue"""
def __init__(
self,
queue: asyncio.Queue,
name: str,
processing_time: float = 1.0
):
self.queue = queue
self.name = name
self.processing_time = processing_time
self.logger = logging.getLogger(self.name)
self.tasks_processed = 0
async def consume(self) -> None:
"""Consume and process tasks from the queue"""
self.logger.info(f"Consumer {self.name} starting...")
while True:
try:
# Wait for a task with timeout
task_data = await asyncio.wait_for(
self.queue.get(),
timeout=3.0
)
# Process the task
await self._process_task(task_data)
# Mark task as done
self.queue.task_done()
self.tasks_processed += 1
except asyncio.TimeoutError:
# No more tasks in queue, exit
self.logger.info(
f"Consumer {self.name} finished. "
f"Processed {self.tasks_processed} tasks"
)
break
except Exception as error:
self.logger.error(
f"Consumer {self.name} error: {error}"
)
async def _process_task(self, task_data: Dict[str, Any]) -> None:
"""Process a single task"""
self.logger.info(
f"Consumer {self.name} processing: {task_data['id']}"
)
# Simulate processing time
await asyncio.sleep(self.processing_time)
self.logger.info(
f"Consumer {self.name} completed: {task_data['id']}"
)What's Happening Here?
- Infinite Loop with Timeout, Uses
while Truewithasyncio.wait_for()to continuously process tasks until the queue is empty for 3 seconds. - Queue.get() and task_done(),
await self.queue.get()retrieves tasks, andself.queue.task_done()signals completion, enabling proper queue tracking. - Exception Handling, Catches
asyncio.TimeoutErrorto gracefully exit and genericExceptionto log errors without crashing the consumer. - Task Tracking, Maintains a counter of processed tasks to provide useful metrics and demonstrate state management in async code.
Implementing main.py - Orchestration
The main.py file orchestrates the entire async system, creating the queue, spawning multiple producers and consumers, and running them concurrently with asyncio.gather().
main.py - Complete Implementation
# -*- coding: utf-8 -*-
import asyncio
import logging
import os
from .producer import Producer
from .consumer import Consumer
async def execute() -> None:
"""Main async execution orchestrator"""
logger = logging.getLogger(
name=os.getenv("APP_NAME", "AsyncProducerConsumer")
)
try:
logger.info("Starting async producer-consumer system...")
# Create shared queue with max size
queue = asyncio.Queue(maxsize=10)
# Create multiple producers
producers = [
Producer(queue, name=f"Producer-{i}", num_tasks=5, delay=0.3)
for i in range(3)
]
# Create multiple consumers
consumers = [
Consumer(queue, name=f"Consumer-{i}", processing_time=0.8)
for i in range(2)
]
# Run all producers and consumers concurrently
await asyncio.gather(
*[p.produce() for p in producers],
*[c.consume() for c in consumers]
)
# Wait for all tasks to be processed
await queue.join()
logger.info("All tasks completed successfully!")
except Exception as error:
logger.error(f"System error: {error}")
raise
finally:
logger.info("Async system shutdown complete")What's Happening Here?
- Shared Queue, Creates a single
asyncio.Queue(maxsize=10)that all producers and consumers share, with a maximum size to prevent memory issues. - List Comprehensions, Uses list comprehensions to create multiple producer and consumer instances with unique names and configurations.
- asyncio.gather(), Runs all coroutines concurrently using
await asyncio.gather()with unpacked lists, enabling true parallel execution. - Queue.join(),
await queue.join()waits until all tasks are marked as done viatask_done(), ensuring complete processing.
Concurrency Pattern Demonstrated
This demonstrates the producer-consumer pattern with multiple producers (3) and consumers (2) running concurrently. The queue acts as a buffer, decoupling production from consumption and enabling efficient parallel processing.
Creating the Entry Point - manager.py
Create manager.py at the root level to provide a CLI entry point that bridges synchronous CLI code with our async execution system.
manager.py - Complete Implementation
# -*- coding: utf-8 -*-
import asyncio
import logging
import os
from click.core import CommandCollection
from click.decorators import group
from environs import Env
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
@group()
def cli_main():
pass
@cli_main.command("run")
def main():
"""Run the async producer-consumer system"""
from service.main import execute
# Run async code from sync context
asyncio.run(execute())
exit()
if __name__ == "__main__":
Env().read_env(os.path.join(os.getcwd(), ".env"))
CommandCollection(sources=[cli_main])()What's Happening Here?
- asyncio.run(), Bridges synchronous CLI code with async execution using
asyncio.run(), which creates an event loop, runs the coroutine, and cleans up automatically. - Logging Configuration, Sets up logging at the root level with a clear format showing timestamps, logger names, and levels for all async components.
- Lazy Import, Imports
executeinside the function to isolate errors during service initialization, preventing failures in one component from crashing the entire CLI.
How to Run Your Project
Execute your async producer-consumer system from the command line. When you run your async system, you'll see interleaved logs from producers and consumers demonstrating concurrent execution:
python manager.py runThis will start 3 producers and 2 consumers running concurrently, processing tasks through the shared queue.
Expected Output:
2024-01-19 14:32:10 - Producer-0 - INFO - Producer Producer-0 starting... 2024-01-19 14:32:10 - Producer-1 - INFO - Producer Producer-1 starting... 2024-01-19 14:32:10 - Producer-2 - INFO - Producer Producer-2 starting... 2024-01-19 14:32:10 - Consumer-0 - INFO - Consumer Consumer-0 starting... 2024-01-19 14:32:10 - Consumer-1 - INFO - Consumer Consumer-1 starting... 2024-01-19 14:32:10 - Producer-0 - INFO - Producer Producer-0 produced: Producer-0-task-0 2024-01-19 14:32:10 - Producer-1 - INFO - Producer Producer-1 produced: Producer-1-task-0 2024-01-19 14:32:10 - Consumer-0 - INFO - Consumer Consumer-0 processing: Producer-0-task-0 2024-01-19 14:32:10 - Consumer-1 - INFO - Consumer Consumer-1 processing: Producer-1-task-0 2024-01-19 14:32:11 - Producer-2 - INFO - Producer Producer-2 produced: Producer-2-task-0 2024-01-19 14:32:11 - Consumer-0 - INFO - Consumer Consumer-0 completed: Producer-0-task-0 2024-01-19 14:32:11 - Consumer-0 - INFO - Consumer Consumer-0 processing: Producer-2-task-0 ... 2024-01-19 14:32:15 - Consumer-0 - INFO - Consumer Consumer-0 finished. Processed 8 tasks 2024-01-19 14:32:15 - Consumer-1 - INFO - Consumer Consumer-1 finished. Processed 7 tasks 2024-01-19 14:32:15 - AsyncProducerConsumer - INFO - All tasks completed successfully! 2024-01-19 14:32:15 - AsyncProducerConsumer - INFO - Async system shutdown complete
What You're Seeing
Notice how the logs are interleaved - producers and consumers run simultaneously. This proves true concurrent execution. The consumers process tasks as soon as they're available, demonstrating the efficiency of the producer-consumer pattern.
With 3 producers creating 5 tasks each (15 total) and 2 consumers processing them, the workload is distributed efficiently, showcasing async Python's power for I/O-bound operations.
Bonus: Python Project Template
Want to see a complete, production-ready example? Check out our Python project template that demonstrates best practices, proper structure, and professional patterns for real-world applications.
The template includes:
- Docker configuration - Containerized development and deployment
- CI/CD pipelines - Automated testing and deployment workflows
- Comprehensive testing - Unit tests, integration tests, and coverage reports
- Enterprise architecture - Professional project organization and structure