Production-Ready Deployment Project

Deploy a production-ready serverless application using AWS CDK and Terraform

Introduction

Welcome to the Infrastructure Deployment Capstone Project! In this hands-on project, you'll deploy a production-ready serverless application on AWS that combines batch processing and event-driven architectures. You'll learn how to use Infrastructure as Code (IaC) to provision, configure, and manage cloud resources in a repeatable, version-controlled manner. This project demonstrates real-world deployment patterns used by engineering teams to ship reliable, scalable applications to production.

Project Overview

You'll build and deploy a Data Processing Pipeline that consists of two AWS components working together to handle different types of workloads:

1. ECS Fargate Task (Batch Processing)

Purpose: Processes large datasets on a scheduled basis (e.g., nightly data aggregation, report generation)

Runs as a containerized task on AWS ECS Fargate
Triggered by EventBridge (CloudWatch Events) on a schedule
Reads data from S3, processes it, and writes results back
Auto-scales based on workload requirements
Publishes completion events to SNS for notifications

2. Lambda Function (Event-Driven Processing)

Purpose: Processes individual records in real-time as they arrive (e.g., user uploads, API requests)

Triggered by messages in an SQS queue
Processes one record at a time with automatic retries
Validates, transforms, and enriches incoming data
Stores processed results in DynamoDB
Dead Letter Queue (DLQ) for failed messages

System Architecture

               ┌─────────────────────────────────────────────────┐
               │             AWS Cloud Environment               │
               └─────────────────────────────────────────────────┘

  ┌─────────────────────────────────┐      ┌──────────────────────────────┐
  │   BATCH PROCESSING PIPELINE     │      │  EVENT-DRIVEN PIPELINE       │
  └─────────────────────────────────┘      └──────────────────────────────┘

  [ EventBridge Rule ]                     [ S3 Upload / API Gateway ]
         │ (cron)                                     │
         ▼                                            ▼
  ┌──────────────┐                          ┌─────────────────┐
  │  ECS Fargate │                          │   SQS Queue     │
  │     Task     │─────────┐                └────────┬────────┘
  └──────┬───────┘         │                         │
         │                 │                         │ (trigger)
         │                 ▼                         ▼
         │          ┌────────────┐           ┌──────────────┐
         │          │ SNS Topic  │           │    Lambda    │
         │          │(Notif.)    │           │   Function   │
         │          └────────────┘           └──────┬───────┘
         │                                          │
         ▼                                          ▼
  ┌────────────┐                           ┌───────────────┐
  │ S3 Bucket  │                           │   DynamoDB    │
  │  (Data)    │                           │    Table      │
  └────────────┘                           └───────────────┘
         │                                          │
         │                                          │
         └────────── CloudWatch Logs ───────────────┘
                     (Monitoring)

// Pattern: Hybrid Batch + Event-Driven Architecture | Serverless Compute | Managed Services

Why This Architecture? This design showcases the two most common processing patterns in modern cloud applications: scheduled batch processing for large datasets and event-driven processing for real-time individual records. Both use serverless technologies to minimize operational overhead and costs.

What You'll Learn

Infrastructure as Code (IaC)

Define your entire infrastructure using code with AWS CDK and Terraform

Serverless Architecture

Deploy scalable applications without managing servers using Lambda and ECS Fargate

Event-Driven Patterns

Implement queue-based processing with SQS, Lambda triggers, and DLQ patterns

Container Orchestration

Package and deploy applications using Docker containers on ECS

CI/CD Integration

Automate deployments with GitHub Actions and AWS service integrations

Monitoring & Observability

Set up CloudWatch metrics, logs, and alarms for production monitoring

Prerequisites

Required Tools & Accounts

Development Tools

Node.js 18+ (for AWS CDK)
Python 3.11+ (for Lambda function)
Docker Desktop (for container builds)
AWS CLI v2 configured
Git for version control

Cloud Accounts

AWS Account with admin access
AWS credentials configured locally
GitHub account (for CI/CD)
Basic understanding of AWS services

Cost Warning: This project uses AWS services that may incur charges. Most resources are eligible for AWS Free Tier, but always monitor your AWS billing dashboard and tear down resources when done.

Expected Project Structure

Before diving into the implementation, here's what your final project structure will look like. Understanding this organization will help you navigate through the steps ahead.

Project Directory Structure

data-pipeline-cdk/
├── app.py                              # CDK app entry point
├── requirements.txt                    # Python dependencies for CDK
├── requirements-dev.txt                # Development dependencies
├── README.md                           # Project documentation
├── cdk.json                            # CDK configuration
├── source.bat                          # Environment activation script
│
├── infrastructure/                     # CDK Stack Definitions
│   ├── __init__.py
│   ├── batch_processing_stack.py       # ECS Fargate batch processing
│   └── event_driven_stack.py           # Lambda + SQS event processing
│
├── project/                            # Application Code
│   ├── __init__.py
│   │
│   ├── lambdas/                        # Lambda Functions
│   │   ├── __init__.py
│   │   └── lambda_one/                 # Event-driven processor
│   │       ├── __init__.py
│   │       └── handler.py              # Lambda handler code
│   │
│   └── tasks/                          # ECS Tasks
│       ├── __init__.py
│       └── task_one/                   # Batch processor
│           ├── __init__.py
│           ├── main.py                 # Python batch processing script
│           ├── Dockerfile              # Container definition
│           └── requirements.txt        # Task dependencies (boto3)
│
└── tests/                              # Unit Tests
    ├── __init__.py
    └── unit/
        └── __init__.py

Key Directories Explained

infrastructure/

Contains CDK stack definitions that define your AWS resources. Each stack file represents a logical grouping of related infrastructure components.

project/lambdas/

Houses all Lambda function code. Each subdirectory represents a separate Lambda function with its handler and dependencies.

project/tasks/

Contains ECS task definitions and Docker containers. Each task has its own Dockerfile, application code, and dependencies.

app.py

The main CDK application file that instantiates and synthesizes your stacks. This is the entry point for CDK commands like deploy and destroy.

Organization Tip: This structure separates infrastructure code (CDK stacks) from application code (Lambda/ECS), following the principle of separation of concerns. As your project grows, this organization makes it easy to locate and modify specific components.

Part 1: Deployment with AWS CDK

AWS Cloud Development Kit (CDK) lets you define cloud infrastructure using familiar programming languages. We'll use Python to define our entire stack, which CDK will synthesize into CloudFormation templates.

Step 1: Initialize CDK Project

$ mkdir data-pipeline-cdk && cd data-pipeline-cdk

$ npm install -g aws-cdk

$ cdk init app --language python

$ source .venv/bin/activate

$ pip install --upgrade pip

$ pip install -r requirements.txt

$ rm -R data_pipeline_cdk

$ mkdir infrastructure

Step 2: Define Lambda + SQS Stack

Create infrastructure/event_driven_stack.py to define the event-driven processing pipeline:

from aws_cdk import (
    Stack,
    Duration,
    RemovalPolicy,
    CfnOutput,
    aws_lambda as lambda_,
    aws_sqs as sqs,
    aws_dynamodb as dynamodb,
    aws_lambda_event_sources as lambda_event_sources,
)
from constructs import Construct


class EventDrivenStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # DynamoDB table for processed data
        table = dynamodb.Table(
            self, "ProcessedDataTable",
            partition_key=dynamodb.Attribute(
                name="id",
                type=dynamodb.AttributeType.STRING
            ),
            sort_key=dynamodb.Attribute(
                name="timestamp",
                type=dynamodb.AttributeType.NUMBER
            ),
            billing_mode=dynamodb.BillingMode.PAY_PER_REQUEST,
            removal_policy=RemovalPolicy.DESTROY,  # For demo only
        )

        # Dead Letter Queue
        dlq = sqs.Queue(
            self, "ProcessingDLQ",
            queue_name="data-processing-dlq",
            retention_period=Duration.days(14),
        )

        # Main processing queue
        queue = sqs.Queue(
            self, "ProcessingQueue",
            queue_name="data-processing-queue",
            visibility_timeout=Duration.seconds(300),
            dead_letter_queue=sqs.DeadLetterQueue(
                queue=dlq,
                max_receive_count=3
            ),
        )

        # Lambda function
        processor = lambda_.Function(
            self, "DataProcessor",
            runtime=lambda_.Runtime.PYTHON_3_11,
            handler="handler.handler",
            code=lambda_.Code.from_asset("project/lambdas/lambda_one"),
            environment={
                "TABLE_NAME": table.table_name,
            },
            timeout=Duration.seconds(60),
            memory_size=512,
        )

        # Grant permissions
        table.grant_write_data(processor)

        # Connect SQS trigger to Lambda
        processor.add_event_source(
            lambda_event_sources.SqsEventSource(
                queue,
                batch_size=10,
                report_batch_item_failures=True,
            )
        )

        # Outputs
        CfnOutput(self, "QueueURL", value=queue.queue_url)
        CfnOutput(self, "TableName", value=table.table_name)

Key CDK Patterns Explained

Dead Letter Queue (DLQ): Failed messages after 3 retries go to the DLQ for debugging
Event Source Mapping: Automatically polls SQS and invokes Lambda with batches of messages
Batch Item Failures: Enables partial batch responses - only failed items are retried
IAM Permissions: grantWriteData() automatically creates least-privilege IAM policies

Step 3: Lambda Function Implementation

Create project/lambdas/lambda_one/handler.py with the processing logic:

import json
import os
import time
import boto3
from datetime import datetime, timezone

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ['TABLE_NAME'])

def handler(event, context):
    """
    Process messages from SQS queue and store in DynamoDB.
    Implements partial batch failure handling.
    """
    failed_records = []

    for record in event['Records']:
        try:
            # Parse message body
            body = json.loads(record['body'])

            # Validate required fields
            if 'id' not in body or 'data' not in body:
                raise ValueError("Missing required fields: id or data")

            # Process and enrich data
            item = {
                'id': body['id'],
                'timestamp': int(time.time()),
                'data': body['data'],
                'processed_at': datetime.now(timezone.utc).isoformat(),
                'status': 'processed'
            }

            # Store in DynamoDB
            table.put_item(Item=item)

            print(f"Successfully processed record: {body['id']}")

        except Exception as e:
            print(f"Failed to process record: {str(e)}")
            # Add to failed records for retry
            failed_records.append({
                'itemIdentifier': record['messageId']
            })

    # Return batch item failures for SQS to retry
    return {
        'batchItemFailures': failed_records
    }

Step 4: Define ECS Batch Processing Stack

Create infrastructure/batch_processing_stack.py for the scheduled batch job:

from aws_cdk import (
    Stack,
    Aws,
    RemovalPolicy,
    CfnOutput,
    aws_ecs as ecs,
    aws_ec2 as ec2,
    aws_s3 as s3,
    aws_sns as sns,
    aws_events as events,
    aws_events_targets as targets,
)
from constructs import Construct


class BatchProcessingStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # VPC for ECS tasks
        vpc = ec2.Vpc(
            self, "BatchVPC",
            max_azs=2,
            nat_gateways=0,  # Use VPC endpoints to save costs
            enable_dns_hostnames=True,  # Required for VPC endpoints
            enable_dns_support=True,     # Required for VPC endpoints
        )

        # Security group for VPC endpoints
        vpc_endpoint_sg = ec2.SecurityGroup(
            self, "VPCEndpointSecurityGroup",
            vpc=vpc,
            description="Security group for VPC endpoints",
            allow_all_outbound=True,
        )

        # Allow HTTPS traffic from within the VPC to VPC endpoints
        vpc_endpoint_sg.add_ingress_rule(
            peer=ec2.Peer.ipv4(vpc.vpc_cidr_block),
            connection=ec2.Port.tcp(443),
            description="Allow HTTPS from VPC",
        )

        # VPC Endpoints for ECR (required for ECS tasks to pull images)
        vpc.add_interface_endpoint(
            "EcrDockerEndpoint",
            service=ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
            security_groups=[vpc_endpoint_sg],
            private_dns_enabled=True,
        )

        vpc.add_interface_endpoint(
            "EcrApiEndpoint",
            service=ec2.InterfaceVpcEndpointAwsService.ECR,
            security_groups=[vpc_endpoint_sg],
            private_dns_enabled=True,
        )

        vpc.add_interface_endpoint(
            "CloudWatchLogsEndpoint",
            service=ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
            security_groups=[vpc_endpoint_sg],
            private_dns_enabled=True,
        )

        # SNS endpoint for task notifications
        vpc.add_interface_endpoint(
            "SnsEndpoint",
            service=ec2.InterfaceVpcEndpointAwsService.SNS,
            security_groups=[vpc_endpoint_sg],
            private_dns_enabled=True,
        )

        # S3 Gateway endpoint (required for ECR to pull Docker layers from S3)
        vpc.add_gateway_endpoint(
            "S3Endpoint",
            service=ec2.GatewayVpcEndpointAwsService.S3,
        )

        # S3 bucket for data
        data_bucket = s3.Bucket(
            self, "DataBucket",
            bucket_name=f"batch-data-{Aws.ACCOUNT_ID}",
            versioned=False,
            removal_policy=RemovalPolicy.DESTROY,
            auto_delete_objects=True,
        )

        # SNS topic for notifications
        notification_topic = sns.Topic(
            self, "BatchNotifications",
            display_name="Batch Processing Notifications",
        )

        # Security group for ECS tasks
        task_security_group = ec2.SecurityGroup(
            self, "TaskSecurityGroup",
            vpc=vpc,
            description="Security group for ECS tasks",
            allow_all_outbound=True,
        )

        # ECS cluster
        cluster = ecs.Cluster(
            self, "BatchCluster",
            vpc=vpc,
            cluster_name="batch-processing-cluster",
        )

        # Task definition
        task_definition = ecs.FargateTaskDefinition(
            self, "BatchTask",
            memory_limit_mib=2048,
            cpu=1024,
        )

        # Container definition
        task_definition.add_container(
            "BatchProcessor",
            image=ecs.ContainerImage.from_asset("./project/tasks/task_one"),
            logging=ecs.LogDrivers.aws_logs(stream_prefix="batch-processor"),
            environment={
                "BUCKET_NAME": data_bucket.bucket_name,
                "SNS_TOPIC_ARN": notification_topic.topic_arn,
            },
        )

        # Grant S3 and SNS permissions
        data_bucket.grant_read_write(task_definition.task_role)
        notification_topic.grant_publish(task_definition.task_role)

        # EventBridge rule to trigger daily at 2 AM UTC
        rule = events.Rule(
            self, "DailyBatchRule",
            schedule=events.Schedule.cron(
                minute="0",
                hour="2",
                week_day="*",
            ),
        )

        # Add ECS task as target
        # Use isolated subnets with VPC endpoints (no NAT gateway needed)
        rule.add_target(
            targets.EcsTask(
                cluster=cluster,
                task_definition=task_definition,
                subnet_selection=ec2.SubnetSelection(
                    subnet_type=ec2.SubnetType.PRIVATE_ISOLATED
                ),
                security_groups=[task_security_group],
            )
        )

        # Outputs
        CfnOutput(self, "BucketName", value=data_bucket.bucket_name)
        CfnOutput(self, "TopicArn", value=notification_topic.topic_arn)

Step 5: Batch Processor Implementation

Create project/tasks/task_one/main.py for the batch processing logic:

import os
import json
import boto3
from datetime import datetime, timedelta, timezone

s3 = boto3.client('s3')
sns = boto3.client('sns')

BUCKET_NAME = os.environ['BUCKET_NAME']
SNS_TOPIC_ARN = os.environ['SNS_TOPIC_ARN']

def process_batch():
    """
    Batch processing job that runs daily.
    Aggregates previous day's data from S3.
    """
    print("Starting batch processing job...")

    try:
        # Calculate yesterday's date for partitioning
        yesterday = (datetime.now() - timedelta(days=1)).strftime('%Y-%m-%d')
        prefix = f"raw-data/{yesterday}/"

        # List all files from yesterday
        response = s3.list_objects_v2(
            Bucket=BUCKET_NAME,
            Prefix=prefix
        )

        if 'Contents' not in response:
            print(f"No data found for {yesterday}")
            return

        # Process each file
        total_records = 0
        for obj in response['Contents']:
            file_key = obj['Key']

            # Download and parse file
            file_obj = s3.get_object(Bucket=BUCKET_NAME, Key=file_key)
            data = json.loads(file_obj['Body'].read())

            # Aggregate logic here
            total_records += len(data)
            print(f"Processed {file_key}: {len(data)} records")

        # Generate summary report
        summary = {
            'date': yesterday,
            'total_records': total_records,
            'processed_at': datetime.now(timezone.utc).isoformat(),
            'status': 'success'
        }

        # Save summary to S3
        summary_key = f"reports/{yesterday}/summary.json"
        s3.put_object(
            Bucket=BUCKET_NAME,
            Key=summary_key,
            Body=json.dumps(summary),
            ContentType='application/json'
        )

        # Send notification
        sns.publish(
            TopicArn=SNS_TOPIC_ARN,
            Subject=f"Batch Job Completed: {yesterday}",
            Message=json.dumps(summary, indent=2)
        )

        print(f"Batch processing completed successfully: {total_records} records")

    except Exception as e:
        error_msg = f"Batch processing failed: {str(e)}"
        print(error_msg)

        # Send failure notification
        sns.publish(
            TopicArn=SNS_TOPIC_ARN,
            Subject="Batch Job Failed",
            Message=error_msg
        )
        raise

if __name__ == '__main__':
    process_batch()

Create project/tasks/task_one/Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

CMD ["python", "main.py"]

Create project/tasks/task_one/requirements.txt:

boto3>=1.28.0
botocore>=1.31.0

Step 6: Deploy to AWS

Before deploying, we need to update app.py to include both stacks we've created. Replace the content of app.py with the following:

import os

import aws_cdk as cdk

from infrastructure.batch_processing_stack import BatchProcessingStack
from infrastructure.event_driven_stack import EventDrivenStack

app = cdk.App()

env = cdk.Environment(
    account=os.getenv("CDK_DEFAULT_ACCOUNT"),
    region=os.getenv("CDK_DEFAULT_REGION"))


batch_stack = BatchProcessingStack(
    scope=app,
    construct_id="BatchProcessingStack",
    env=env,
)

event_stack = EventDrivenStack(
    scope=app,
    construct_id="EventDrivenStack",
    env=env,
)

app.synth()

Understanding app.py

Environment Configuration: Uses environment variables to set AWS account and region
Stack Instantiation: Creates both BatchProcessingStack and EventDrivenStack with the same environment
Synthesize: The app.synth() command generates CloudFormation templates from your CDK code

Now you're ready to deploy both stacks to AWS:

# Bootstrap CDK (first time only, ensure AWS credentials are configured)

$ cdk bootstrap

# Synthesize CloudFormation template

$ cdk synth

# Deploy all stacks

$ cdk deploy --all

Deployment Complete! After deployment completes, you'll see the CloudFormation outputs including queue URLs, bucket names, and other resource ARNs. Save these for testing.

Bonus: Simplify Your CDK with Production-Ready Constructs

Level Up Your CDK Skills with Reusable Constructs

While you've learned to build CDK infrastructure from scratch, professional teams use pre-built, battle-tested constructs to speed up development and enforce best practices. The ByteCode Solutions Core AWS CDK library provides production-ready components that handle common infrastructure patterns with built-in security and operational best practices.

Core AWS CDK Library Features

The core-aws-cdk library provides common elements and constructs to create infrastructure in AWS using AWS CDK with Python, including:

Base Stacks

Pre-configured CDK stacks with automatic tagging and organization

Lambda Functions

Simplified Lambda creation with automatic packaging and dependencies

S3 Buckets

S3 bucket creation with built-in security best practices and encryption

SQS Queues

Queue creation with automatic dead-letter queue configuration

SNS Topics

Topic creation with subscription management and filtering

Network Stack

VPC and networking resource management with best practices

ZIP Asset Packaging

Automatic Lambda ZIP creation with dependencies and caching

View Repository Read Documentation

How This Could Simplify Your Project

Instead of manually defining every CDK construct, you could use the library to:

Leverage pre-configured base stacks with automatic tagging and resource organization
Use the Lambda construct with automatic ZIP packaging instead of manual asset handling
Create SQS queues with built-in DLQ patterns in fewer lines of code
Deploy SNS topics with standardized subscription management
Follow security best practices automatically enforced by the library

Pro Tip: After completing this project with raw CDK constructs, try refactoring it using the core-aws-cdk library to see how production teams accelerate infrastructure development while maintaining best practices!

Step 7: Testing Your Deployment

Test Event-Driven Pipeline (Lambda + SQS)

Important: Ensure you use the resource names exactly as they were created by CloudFormation!

Send Test Message to SQS

aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/data-processing-queue \
  --message-body '{"id": "test-001", "data": "Hello from SQS!"}'

Expected Output:

{
    "MD5OfMessageBody": "979b14da0634d5f4b19c4ab55828f2d2",
    "MessageId": "b45d02eb-3fdc-4450-9dcd-80439bdf2466"
}

Check Lambda Logs

aws logs tail /aws/lambda/EventDrivenStack-DataProcessor --follow

Expected Output:

2024-01-22T17:14:06.005000+00:00/[$LATEST]586f05e33e INIT_START Runtime Version: python:3.11.v109    Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:49f73325b4590c
2024-01-22T17:14:06/[$LATEST]586f05e33e START RequestId: b3bb3e97-d58f-5b07-9d0a-5f7b889be30f Version: $LATEST
2024-01-22T17:14:06/[$LATEST]586f05e33e Successfully processed record: test-001
2024-01-22T17:14:06/[$LATEST]586f05e33e END RequestId: b3bb3e97-d58f-5b07-9d0a-5f7b889be30f
2024-01-22T17:14:06/[$LATEST]586f05e33e REPORT RequestId: b3bb3e97-d58f-5b07-9d0a-5f7b889be30f    Duration: 55.99 ms    Billed Duration: 579 ms    Memory Size: 512 MB    Max Memory Used: 86 MB    Init Duration: 522.64 ms

Look for the "Successfully processed record: test-001" message confirming successful processing.

Query DynamoDB to Verify Processing

aws dynamodb scan --table-name ProcessedDataTable

Expected Output:

{
    "Items": [
        {
            "id": {
                "S": "test-001"
            },
            "processed_at": {
                "S": "2026-01-22T17:14:06.532243"
            },
            "data": {
                "S": "Hello from SQS!"
            },
            "status": {
                "S": "processed"
            },
            "timestamp": {
                "N": "1769102046"
            }
        }
    ],
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

Your test message should appear in the DynamoDB table with status "processed" and the correct data.

Test Batch Processing (ECS)

Upload Test Data to S3

# Create test data file
echo '[{"id": 1, "value": "test"}]' > test-data.json

# Upload to S3 (replace ACCOUNT with your account ID, and adjust date to be your yesterday)
aws s3 cp test-data.json s3://batch-data-ACCOUNT/raw-data/2024-01-21/data.json

Manually Trigger ECS Task

Don't wait for the scheduled execution, trigger the task manually for immediate testing, and use the appropriate subnet and security group:

aws ecs run-task \
  --cluster batch-processing-cluster \
  --task-definition BatchProcessingStack-BatchTask \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-***],securityGroups=[sg-***]}"

Check ECS Task Logs

aws logs tail /ecs/batch-processor --follow

Expected Output:

2024-01-22T18:15:07 batch-processor/BatchProcessor/8a317aeb015 Starting batch processing job...
2024-01-22T18:15:07 batch-processor/BatchProcessor/8a317aeb015 Processed raw-data/2024-01-21/data.json: 1 records
2024-01-22T18:15:07 batch-processor/BatchProcessor/8a317aeb015 Batch processing completed successfully: 1 records

Look for the "Batch processing completed successfully" message confirming the ECS task processed the data and generated the summary report.

Testing Complete! You've successfully tested both the event-driven (Lambda + SQS) and batch processing (ECS Fargate) components. Your serverless data processing pipeline is now fully operational!

Part 2: Bonus - Deployment with Terraform

Why Terraform?

While AWS CDK is excellent for AWS-specific infrastructure, Terraform is a cloud-agnostic IaC tool that works across AWS, Azure, GCP, and 1000+ providers. It uses HCL (HashiCorp Configuration Language) and maintains infrastructure state to enable safe updates and drift detection.

✅ Terraform Advantages

Multi-cloud support
Large ecosystem & modules
State management & planning
Industry standard

✅ AWS CDK Advantages

Use familiar languages (Python/TypeScript)
Better IDE support & type safety
AWS service integration
Higher-level abstractions

Terraform Deployment Commands

# Initialize Terraform

$ terraform init

# Preview changes

$ terraform plan

# Apply changes

$ terraform apply

# Destroy infrastructure

$ terraform destroy

Bonus Challenge: Deploy with Production-Ready Terraform Modules

Advanced Exercise: Use Reusable Terraform Modules

Ready to take your Terraform skills to the next level? Instead of writing infrastructure code from scratch, try using production-ready, reusable Terraform modules from the ByteCode Solutions Core Terraform library. This is how real engineering teams build scalable infrastructure!

Core Terraform Modules Repository

The core-terraform repository provides a collection of battle-tested, modular infrastructure components that follow HashiCorp and industry best practices:

ECS Cluster Module

Provisions ECS clusters with Fargate/EC2 support, auto-scaling, and IAM roles

Lambda Function Module

Deploys Lambda functions with runtime config, IAM, and logging best practices

Lambda Builder Module

Build pipeline for packaging Lambda code with dependencies and caching

Folder Hash Module

Generate hashes to trigger updates when source files change

View Repository Read Documentation

Your Challenge

Try replicating the same infrastructure using these production-ready modules:

Use the Lambda Function and Lambda Builder modules for the event-driven pipeline
Use the ECS Cluster module for the batch processing infrastructure
Reference the module documentation for usage examples and input variables
Compare the module approach with raw Terraform resource definitions

CDK vs Terraform: Quick Comparison

Feature	AWS CDK	Terraform
Language	Python, TypeScript, Java, C#, Go	HCL (declarative)
Cloud Support	AWS only	Multi-cloud (AWS, Azure, GCP, etc.)
State Management	CloudFormation handles state	Terraform state file (local or remote)
Learning Curve	Moderate (need to know programming)	Easy (simple declarative syntax)
Type Safety	Yes (type hints with Python/TypeScript)	Limited
Ecosystem	AWS-focused constructs	Massive provider ecosystem

Part 3: Automated Deployment with GitLab CI/CD

Why CI/CD for Infrastructure?

Manual deployments from your local machine work for learning, but production teams use CI/CD pipelines to automate infrastructure deployment. This ensures consistency, enables team collaboration, and provides audit trails for all infrastructure changes.

✅ Consistency

Same deployment process every time, regardless of who triggers it

✅ Security

AWS credentials stay in CI/CD, never on developer machines

✅ Auditability

Complete history of who deployed what and when

Step 1: Create Infrastructure Dockerfile

Create infrastructure/Dockerfile to containerize your CDK deployment environment:

FROM alpine:3.18.5

LABEL authors="Your Name <your.email@example.com>"
LABEL version="1.0"

ARG CI_JOB_TOKEN
ARG CDK_DEFAULT_ACCOUNT
ARG CDK_DEFAULT_REGION

ARG AWS_ACCESS_KEY_ID
ARG AWS_DEFAULT_REGION=us-east-1
ARG AWS_SECRET_ACCESS_KEY

ENV APP_NAME=data-pipeline-cdk
ENV VIRTUAL_ENV=/home/$APP_USER/.venv

# Install dependencies
RUN apk update && \
    apk add --update npm curl git openssh gcc python3 py3-pip docker-cli && \
    npm i -g aws-cdk && \
    rm -rf /var/cache/apk/*

# Configure SSH for GitLab
RUN mkdir -p ~/.ssh/ && \
    touch ~/.ssh/known_hosts && \
    chmod 644 ~/.ssh/known_hosts && \
    ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts

# Create working directory
RUN mkdir /$APP_NAME && \
    chmod g+rwx /$APP_NAME

WORKDIR /$APP_NAME

# Copy project files
COPY infrastructure ./infrastructure
COPY project ./project
COPY app.py ./
COPY cdk.json ./
COPY cdk.context.json ./

# Install Python dependencies
RUN echo "machine gitlab.com" > ~/.netrc && \
    echo "login gitlab-ci-token" >> ~/.netrc && \
    echo "password ${CI_JOB_TOKEN}" >> ~/.netrc && \
    pip install --upgrade pip && \
    pip install -r infrastructure/requirements.txt && \
    rm ~/.netrc

# Verify installations
RUN echo "pip version: $(pip --version)" && \
    echo "AWS CDK version: $(cdk --version)"

Dockerfile Purpose: This creates a consistent deployment environment with all necessary tools (Node.js, Python, AWS CDK, Docker CLI) that GitLab CI will use to deploy your infrastructure.

Step 2: Create Infrastructure Requirements

Create infrastructure/requirements.txt for the CDK dependencies needed in the container:

aws-cdk-lib>=2.0.0
constructs>=10.0.0

Note:This requirements file is specifically for the infrastructure container and contains only CDK dependencies. Your Lambda and ECS tasks have their own separate requirements files in their respective directories.

Step 3: Configure GitLab CI/CD Pipeline

Create .gitlab-ci.yml in your project root to define the CI/CD pipeline:

image: docker:20.10.21-dind

services:
  - docker:20.10.21-dind

variables:
  APP_NAME: data-pipeline-cdk
  CI_JOB_TOKEN: "${ACCESS_TOKEN}"
  TEMPLATE_REGISTRY_HOST: "registry.gitlab.com"
  DOCKER_HOST: tcp://docker:2375

stages:
  - deploy

# Deploy infrastructure on main branch
deploy:
  stage: deploy

  environment:
    name: $CI_COMMIT_BRANCH
    on_stop: stop_deploy

  variables:
    AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}"
    AWS_DEFAULT_REGION: "${AWS_DEFAULT_REGION}"
    AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}"
    CDK_DEFAULT_ACCOUNT: "${CDK_DEFAULT_ACCOUNT}"
    CDK_DEFAULT_REGION: "${CDK_DEFAULT_REGION}"
    CI_JOB_TOKEN: "${ACCESS_TOKEN}"

  script:
    # Build the infrastructure container
    - docker build -t $APP_NAME-infrastructure --build-arg CI_JOB_TOKEN=${CI_JOB_TOKEN} -f infrastructure/Dockerfile .

    # Synthesize CloudFormation templates
    - docker run --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} --env AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} --env CDK_DEFAULT_ACCOUNT=${CDK_DEFAULT_ACCOUNT} --env CDK_DEFAULT_REGION=${CDK_DEFAULT_REGION} $APP_NAME-infrastructure cdk synth

    # Deploy all stacks
    - docker run --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} --env AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} --env CDK_DEFAULT_ACCOUNT=${CDK_DEFAULT_ACCOUNT} --env CDK_DEFAULT_REGION=${CDK_DEFAULT_REGION} $APP_NAME-infrastructure cdk deploy --all --require-approval never

  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# Destroy infrastructure (manual trigger only)
stop_deploy:
  stage: deploy

  environment:
    name: $CI_COMMIT_BRANCH
    action: stop

  script:
    - docker build -t $APP_NAME-infrastructure --build-arg CI_JOB_TOKEN=${CI_JOB_TOKEN} -f infrastructure/Dockerfile .
    - docker run --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} --env AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} --env CDK_DEFAULT_ACCOUNT=${CDK_DEFAULT_ACCOUNT} --env CDK_DEFAULT_REGION=${CDK_DEFAULT_REGION} $APP_NAME-infrastructure cdk destroy --all --force

  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Pipeline Breakdown

Docker-in-Docker: Uses Docker inside GitLab CI to build and run your infrastructure container
deploy job: Automatically triggers on pushes to main branch, builds container, and deploys infrastructure
stop_deploy job: Manual-only job to destroy infrastructure when needed
Environment variables: AWS credentials and configuration passed from GitLab CI/CD variables

Step 4: Configure GitLab CI/CD Variables

In your GitLab project, go to Settings → CI/CD → Variables and add the following variables:

Variable Name	Description	Masked
`AWS_ACCESS_KEY_ID`	AWS IAM access key for deployment	Yes
`AWS_SECRET_ACCESS_KEY`	AWS IAM secret access key	Yes
`AWS_DEFAULT_REGION`	AWS region (e.g., us-east-1)	No
`CDK_DEFAULT_ACCOUNT`	Your AWS account ID	No
`CDK_DEFAULT_REGION`	CDK deployment region	No
`ACCESS_TOKEN`	GitLab access token (if using private dependencies)	Yes

Security Best Practice: Always mark sensitive variables (credentials, tokens) as Masked and Protected in GitLab to prevent them from appearing in CI/CD logs.

Step 5: Trigger Automated Deployment

Once everything is configured, push your code to the main branch:

git add .
git commit -m "Add CDK infrastructure with GitLab CI/CD"
git push origin main

GitLab CI/CD will automatically:

Build the infrastructure Docker container
Run cdk synth to generate CloudFormation templates
Run cdk deploy --all to provision AWS resources

Monitor Progress: View your pipeline execution in GitLab at CI/CD → Pipelines. You'll see real-time logs of the deployment process.

Benefits of CI/CD Infrastructure Deployment

For Teams

Consistent deployments across all team members
No "works on my machine" issues
Easy rollbacks by reverting git commits
Collaboration through merge requests

For Production

Audit trail of all infrastructure changes
Secrets managed centrally in CI/CD
Automated testing before deployment
Environment parity (dev, staging, prod)

Production Monitoring & Best Practices

CloudWatch Monitoring

Lambda invocations, errors, duration
SQS queue depth & age of oldest message
ECS task CPU/memory utilization
DynamoDB read/write capacity
Custom metrics for business logic

Alerting Strategy

Lambda error rate > 5%
DLQ message count > 0
ECS task failures
Queue age > 5 minutes
S3 batch job completion status

Production Best Practices

Security

Use IAM roles, never hardcode credentials
Enable encryption at rest (S3, DynamoDB)
Use VPC endpoints for private communication
Implement least privilege IAM policies
Enable CloudTrail for audit logging

Reliability

Use Dead Letter Queues for failed messages
Implement exponential backoff retries
Set appropriate Lambda timeouts
Enable X-Ray for distributed tracing
Tag all resources for cost tracking

Cleaning Up Resources

Important: Don't forget to tear down your infrastructure when you're done to avoid unnecessary AWS charges!

CDK Cleanup

cdk destroy --all

Terraform Cleanup

terraform destroy

Key Takeaways

Infrastructure as Code enables version-controlled, repeatable deployments
AWS CDK uses programming languages for type-safe infrastructure definitions
Terraform provides cloud-agnostic IaC with a large ecosystem
Serverless architecture reduces operational overhead for batch and event-driven workloads
ECS Fargate is ideal for containerized batch jobs that run on schedules
Lambda + SQS provides scalable, reliable event-driven processing
Dead Letter Queues are essential for handling failures in production
Monitoring and alerting with CloudWatch ensures system reliability
IAM roles and policies implement security through least privilege access
Always tag resources and clean up unused infrastructure to control costs

Additional Resources

AWS CDK

Terraform

Another Practical Guide: Lambda with Custom Dependencies

Practical Guide: Deploying Lambda Functions with Terraform

For a more focused, step-by-step walkthrough of deploying Lambda functions written in Python with custom dependencies (not included in AWS Lambda Runtime), check out this detailed Medium article. This guide demonstrates a simpler deployment scenario using Terraform as Infrastructure as Code (IaC), perfect for understanding the fundamentals before tackling complex multi-service architectures.

What You'll Learn:

Packaging Python Lambda functions with external dependencies
Creating Lambda deployment packages with libraries not in AWS Runtime
Using Terraform to deploy Lambda functions to AWS
Managing IAM roles and permissions for Lambda

Read Full Article on Medium

Back to "Security & Compliance"

Infrastructure for EngineersLesson Final Project

Course Completed!