Cloud Platforms
AWS, GCP, and Azure, Storage, compute, databases, serverless, and cloud architecture
The Cloud: Infinite Infrastructure on Demand
Cloud computing revolutionized infrastructure. Instead of buying servers, racking them, and managing data centers, you provision resources with an API call and pay only for what you use. The big three, AWS (Amazon Web Services), GCP (Google Cloud Platform), and Azure (Microsoft Azure), dominate the market. This lesson focuses on AWS (the market leader with ~32% market share) while showing equivalent services in GCP and Azure. You'll learn storage, compute, databases, serverless, and how to architect cloud-native applications.
Cloud Fundamentals, Why Cloud?
Before diving into services, let's understand what cloud computing actually provides.
Speed & Agility
- Provision resources in minutes (not months)
- Experiment cheaply, fail fast
- Global reach in 30+ regions
- Deploy worldwide instantly
Cost Efficiency
- Pay-as-you-go (no upfront costs)
- Scale down = pay less
- No hardware to maintain
- Economies of scale
Elasticity
- Auto-scale based on demand
- Handle traffic spikes automatically
- Scale to zero when idle
- No capacity planning needed
Reliability
- 99.99% SLAs for many services
- Built-in redundancy
- Automatic backups
- Disaster recovery built-in
The Big Three
| Provider | Market Share | Strengths | Best For |
|---|---|---|---|
| AWS | ~32% | Most mature, largest service catalog, huge ecosystem | Startups, enterprises, general purpose |
| Azure | ~23% | Enterprise integration, hybrid cloud, Windows/.NET workloads | Enterprises with Microsoft stack |
| GCP | ~10% | Data analytics, ML/AI, Kubernetes, developer experience | Data-heavy workloads, ML projects, startups |
Storage Services, Objects, Blocks, and Files
Cloud storage comes in three flavors: object storage (files/blobs),block storage (virtual hard drives), and file storage (shared file systems).
| Service Type | AWS | GCP | Azure |
|---|---|---|---|
| Object Storage | S3 | Cloud Storage | Blob Storage |
| Block Storage | EBS | Persistent Disk | Managed Disks |
| File Storage | EFS | Filestore | Azure Files |
| Archive Storage | S3 Glacier | Cloud Storage Archive | Archive Blob Storage |
AWS S3, Object Storage
S3 (Simple Storage Service) is AWS's flagship object storage. It's infinitely scalable, highly durable (99.999999999% - eleven 9s), and incredibly cheap.
# AWS CLI - S3 operations
# Create bucket $ aws s3 mb s3://my-app-bucket # Upload file $ aws s3 cp image.jpg s3://my-app-bucket/images/ # Upload directory $ aws s3 sync ./dist s3://my-app-bucket/website/ # Download file $ aws s3 cp s3://my-app-bucket/data.csv ./ # List objects $ aws s3 ls s3://my-app-bucket/ # Delete object $ aws s3 rm s3://my-app-bucket/old-file.txt # Make bucket public (careful!) $ aws s3api put-bucket-policy --bucket my-app-bucket --policy file://policy.json # Enable versioning $ aws s3api put-bucket-versioning \ --bucket my-app-bucket \ --versioning-configuration Status=Enabled
# Python boto3 - S3 SDK
import boto3
from botocore.exceptions import ClientError
s3 = boto3.client('s3')
# Upload file
s3.upload_file('local.jpg', 'my-bucket', 'images/photo.jpg')
# Upload with metadata
s3.put_object(
Bucket='my-bucket',
Key='data.json',
Body=json.dumps(data),
ContentType='application/json',
Metadata={'uploaded-by': 'user123'}
)
# Download file
s3.download_file('my-bucket', 'data.csv', 'local-data.csv')
# Generate presigned URL (temporary access)
url = s3.generate_presigned_url(
'get_object',
Params={'Bucket': 'my-bucket', 'Key': 'private.pdf'},
ExpiresIn=3600 # 1 hour
)
# List objects with pagination
paginator = s3.get_paginator('list_objects_v2')
for page in paginator.paginate(Bucket='my-bucket', Prefix='logs/'):
for obj in page.get('Contents', []):
print(obj['Key'], obj['Size'])
# Delete multiple objects
s3.delete_objects(
Bucket='my-bucket',
Delete={
'Objects': [
{'Key': 'file1.txt'},
{'Key': 'file2.txt'},
]
}
)S3 Storage Classes (Cost Optimization)
| Class | Use Case | Cost | Retrieval Time |
|---|---|---|---|
| S3 Standard | Frequently accessed data | $$$$ | Instant |
| S3 Intelligent-Tiering | Unknown/changing access patterns | Auto-optimized | Instant |
| S3 Standard-IA | Infrequent access (backups) | $$$ | Instant |
| S3 One Zone-IA | Infrequent, non-critical | $$ | Instant |
| S3 Glacier Instant | Archive, instant retrieval | $ | Instant |
| S3 Glacier Flexible | Long-term archive | $ | Minutes to hours |
| S3 Glacier Deep Archive | Compliance, rarely accessed | ¢ | 12 hours |
AWS EBS, Block Storage
EBS (Elastic Block Store) provides persistent block storage volumes for EC2 instances. Think of it as virtual hard drives.
# Create and attach EBS volume
# Create volume $ aws ec2 create-volume \ --availability-zone us-east-1a \ --size 100 \ --volume-type gp3 \ --iops 3000 \ --throughput 125 # Attach to instance $ aws ec2 attach-volume \ --volume-id vol-1234567890abcdef0 \ --instance-id i-1234567890abcdef0 \ --device /dev/sdf # Create snapshot (backup) $ aws ec2 create-snapshot \ --volume-id vol-1234567890abcdef0 \ --description "Backup before upgrade" # Restore from snapshot $ aws ec2 create-volume \ --snapshot-id snap-1234567890abcdef0 \ --availability-zone us-east-1a
• gp3/gp2 (General Purpose SSD): Balanced price/performance, most workloads
• io2/io1 (Provisioned IOPS SSD): High-performance databases, low latency
• st1 (Throughput Optimized HDD): Big data, data warehouses
• sc1 (Cold HDD): Infrequently accessed data
Compute Services, VMs, Containers, and Serverless
Compute is the backbone of cloud: virtual machines, containers, and serverless functions.
| Service Type | AWS | GCP | Azure |
|---|---|---|---|
| Virtual Machines | EC2 | Compute Engine | Virtual Machines |
| Container Service | ECS / EKS | Cloud Run / GKE | Container Instances / AKS |
| Serverless Functions | Lambda | Cloud Functions | Azure Functions |
| Serverless Containers | Fargate | Cloud Run | Container Apps |
AWS EC2, Virtual Machines
# Launch EC2 instance
# Via AWS CLI
$ aws ec2 run-instances \
--image-id ami-0c55b159cbfafe1f0 \
--instance-type t3.micro \
--key-name my-key \
--security-group-ids sg-1234567890 \
--subnet-id subnet-1234567890 \
--user-data file://install.sh \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=web-server}]'
# List instances
$ aws ec2 describe-instances \
--filters "Name=tag:Environment,Values=production"
# Stop instance
$ aws ec2 stop-instances --instance-ids i-1234567890abcdef0
# Terminate instance
$ aws ec2 terminate-instances --instance-ids i-1234567890abcdef0# Python boto3 - EC2
import boto3
ec2 = boto3.resource('ec2')
# Launch instance
instances = ec2.create_instances(
ImageId='ami-0c55b159cbfafe1f0',
MinCount=1,
MaxCount=1,
InstanceType='t3.micro',
KeyName='my-key',
SecurityGroupIds=['sg-1234567890'],
SubnetId='subnet-1234567890',
UserData='''#!/bin/bash
yum update -y
yum install -y httpd
systemctl start httpd
systemctl enable httpd
''',
TagSpecifications=[
{
'ResourceType': 'instance',
'Tags': [
{'Key': 'Name', 'Value': 'web-server'},
{'Key': 'Environment', 'Value': 'production'}
]
}
]
)
# Wait for instance to be running
instance = instances[0]
instance.wait_until_running()
# Get public IP
instance.reload()
print(f"Instance running at {instance.public_ip_address}")
# Stop instance
instance.stop()
# Terminate instance
instance.terminate()EC2 Instance Types
| Family | Use Case | Examples |
|---|---|---|
| T (Burstable) | Low-cost, variable workloads, dev/test | t3.micro, t3.medium |
| M (General Purpose) | Balanced compute, memory, networking | m6i.large, m6i.xlarge |
| C (Compute Optimized) | CPU-intensive, batch processing, ML inference | c6i.xlarge, c7g.2xlarge |
| R (Memory Optimized) | In-memory databases, caches, analytics | r6i.xlarge, r7g.4xlarge |
| P/G (GPU) | ML training, graphics rendering | p4d.24xlarge, g5.xlarge |
AWS Lambda, Serverless Functions
Lambda runs code without managing servers. Pay only for compute time used (billed per 100ms). Perfect for event-driven architectures.
# lambda_function.py
import json
import boto3
from datetime import datetime
s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users')
def lambda_handler(event, context):
"""
Lambda handler function
event: trigger data (API Gateway, S3 event, etc.)
context: runtime information
"""
# Example 1: API Gateway event
if event.get('httpMethod'):
body = json.loads(event['body'])
# Process request
user_id = body.get('user_id')
user = table.get_item(Key={'id': user_id})
return {
'statusCode': 200,
'headers': {'Content-Type': 'application/json'},
'body': json.dumps(user.get('Item', {}))
}
# Example 2: S3 event (file uploaded)
if event.get('Records'):
for record in event['Records']:
if record['eventName'].startswith('ObjectCreated'):
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
# Process uploaded file
print(f"Processing {key} from {bucket}")
# Get file
obj = s3.get_object(Bucket=bucket, Key=key)
data = obj['Body'].read()
# Do something with data...
return {'statusCode': 200}
# Example 3: Scheduled event (CloudWatch Events/EventBridge)
if event.get('source') == 'aws.events':
# Daily cleanup job
print(f"Running cleanup at {datetime.now()}")
# Cleanup logic...
return {'statusCode': 200}
return {
'statusCode': 400,
'body': json.dumps({'error': 'Unknown event type'})
}# Deploy Lambda with AWS SAM
# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
ApiFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.lambda_handler
Runtime: python3.11
CodeUri: ./src
MemorySize: 512
Timeout: 30
Environment:
Variables:
TABLE_NAME: !Ref UsersTable
Events:
ApiEvent:
Type: Api
Properties:
Path: /users/{id}
Method: GET
Policies:
- DynamoDBCrudPolicy:
TableName: !Ref UsersTable
UsersTable:
Type: AWS::DynamoDB::Table
Properties:
BillingMode: PAY_PER_REQUEST
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
# Deploy
$ sam build
$ sam deploy --guided• API backends (with API Gateway)
• File processing (S3 uploads trigger Lambda)
• Scheduled tasks (cron jobs)
• Stream processing (Kinesis, DynamoDB Streams)
• Webhooks and integrations
Limits: 15-minute max execution, 10GB memory, 10GB ephemeral storage (512MB default)
Database Services, Relational, NoSQL, and Caching
Cloud providers offer managed databases: no patching, automated backups, high availability built-in.
| Database Type | AWS | GCP | Azure |
|---|---|---|---|
| Relational (SQL) | RDS (MySQL, PostgreSQL, etc.) | Cloud SQL | Azure SQL Database |
| NoSQL (Key-Value) | DynamoDB | Firestore, Bigtable | Cosmos DB |
| Document Database | DocumentDB | Firestore | Cosmos DB |
| In-Memory Cache | ElastiCache (Redis, Memcached) | Memorystore | Azure Cache for Redis |
| Data Warehouse | Redshift | BigQuery | Synapse Analytics |
AWS RDS, Managed Relational Databases
# Create RDS instance (PostgreSQL)
$ aws rds create-db-instance \ --db-instance-identifier myapp-db \ --db-instance-class db.t4g.micro \ --engine postgres \ --engine-version 15.3 \ --master-username admin \ --master-user-password MySecurePassword123! \ --allocated-storage 20 \ --storage-type gp3 \ --backup-retention-period 7 \ --preferred-backup-window "03:00-04:00" \ --multi-az \ # High availability --publicly-accessible false \ --vpc-security-group-ids sg-1234567890 \ --db-subnet-group-name my-db-subnet-group \ --storage-encrypted \ --enable-cloudwatch-logs-exports '["postgresql"]' # Create read replica $ aws rds create-db-instance-read-replica \ --db-instance-identifier myapp-db-replica \ --source-db-instance-identifier myapp-db \ --db-instance-class db.t4g.micro # Create snapshot $ aws rds create-db-snapshot \ --db-instance-identifier myapp-db \ --db-snapshot-identifier myapp-db-snapshot-2024-01-20
# Connect to RDS from application
# Python with psycopg2
import psycopg2
# Get connection string from environment or Secrets Manager
conn = psycopg2.connect(
host="myapp-db.c1234567890.us-east-1.rds.amazonaws.com",
port=5432,
database="myapp",
user="admin",
password="MySecurePassword123!"
)
cursor = conn.cursor()
# Query
cursor.execute("SELECT * FROM users WHERE email = %s", (email,))
user = cursor.fetchone()
# Insert
cursor.execute(
"INSERT INTO users (name, email) VALUES (%s, %s)",
(name, email)
)
conn.commit()
cursor.close()
conn.close()AWS DynamoDB, Serverless NoSQL
DynamoDB is a fully managed NoSQL database. Single-digit millisecond latency, auto-scales to trillions of requests per day, pay per request.
# Python boto3 - DynamoDB
import boto3
from boto3.dynamodb.conditions import Key, Attr
from decimal import Decimal
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('users')
# Put item
table.put_item(
Item={
'user_id': '12345',
'name': 'John Doe',
'email': 'john@example.com',
'age': 30,
'tags': ['premium', 'verified'],
'created_at': '2024-01-20T10:30:00Z'
}
)
# Get item
response = table.get_item(
Key={'user_id': '12345'}
)
user = response.get('Item')
# Update item
table.update_item(
Key={'user_id': '12345'},
UpdateExpression='SET age = :age, tags = list_append(tags, :tag)',
ExpressionAttributeValues={
':age': 31,
':tag': ['active']
}
)
# Query (requires partition key)
response = table.query(
KeyConditionExpression=Key('user_id').eq('12345')
)
# Scan (expensive, avoid in production)
response = table.scan(
FilterExpression=Attr('age').gt(25) & Attr('tags').contains('premium')
)
# Batch write
with table.batch_writer() as batch:
for i in range(100):
batch.put_item(
Item={
'user_id': f'user-{i}',
'name': f'User {i}',
'score': Decimal(str(i * 10))
}
)
# Delete item
table.delete_item(
Key={'user_id': '12345'}
)• Partition Key: Required, used for data distribution
• Sort Key: Optional, enables range queries
• GSI (Global Secondary Index): Query on non-key attributes
• LSI (Local Secondary Index): Alternative sort key
• Billing: On-demand (pay per request) or provisioned capacity
Best for: High-scale apps, serverless, gaming leaderboards, IoT
Networking, VPCs, Load Balancers, and CDN
Cloud networking provides isolation, security, and global distribution.
| Service | AWS | GCP | Azure |
|---|---|---|---|
| Virtual Network | VPC | VPC | Virtual Network |
| Load Balancer | ALB / NLB / ELB | Cloud Load Balancing | Load Balancer |
| CDN | CloudFront | Cloud CDN | Azure CDN |
| DNS | Route 53 | Cloud DNS | Azure DNS |
| API Gateway | API Gateway | API Gateway | API Management |
AWS VPC, Virtual Private Cloud
VPC Architecture (10.0.0.0/16)
Security: Security Groups, Network ACLs, VPC Flow Logs — NAT Gateway enables private subnet outbound access
Load Balancers
| Type | Layer | Use Case |
|---|---|---|
| ALB (Application) | Layer 7 (HTTP/HTTPS) | Web apps, microservices, path-based routing |
| NLB (Network) | Layer 4 (TCP/UDP) | Ultra-low latency, millions of requests/sec, static IP |
| ELB (Classic) | Layer 4/7 | Legacy, use ALB/NLB instead |
# Create Application Load Balancer
$ aws elbv2 create-load-balancer \ --name my-alb \ --subnets subnet-12345 subnet-67890 \ --security-groups sg-12345 \ --scheme internet-facing \ --type application # Create target group $ aws elbv2 create-target-group \ --name my-targets \ --protocol HTTP \ --port 80 \ --vpc-id vpc-12345 \ --health-check-path /health \ --health-check-interval-seconds 30 # Register targets $ aws elbv2 register-targets \ --target-group-arn arn:aws:... \ --targets Id=i-12345 Id=i-67890 # Create listener $ aws elbv2 create-listener \ --load-balancer-arn arn:aws:... \ --protocol HTTP \ --port 80 \ --default-actions Type=forward,TargetGroupArn=arn:aws:...
Cloud Architecture Patterns
Common patterns for building cloud-native applications.
1. Three-Tier Web Application
Three-Tier Web Application
2. Serverless Architecture
Serverless Architecture
3. Microservices with ECS/EKS
Microservices with ECS / EKS
• Independently deployable
• Own database (data isolation)
• Communicates via APIs/events
• Scales independently
Service Discovery: Route 53, AWS Cloud Map
Message Queue: SQS, SNS, EventBridge
Observability: CloudWatch, X-Ray
4. Event-Driven Architecture
Event-Driven Architecture
Benefits: Loose coupling, scalability, resilience
• Design for failure (everything fails eventually)
• Implement elasticity (scale up/down automatically)
• Decouple components (loose coupling enables independent scaling)
• Think parallel (async processing, queues)
• Use managed services (less operational overhead)
• Implement security in depth (multiple layers)
• Cache aggressively (CloudFront, ElastiCache, DAX)
Cost Optimization
Cloud bills can spiral out of control. Here's how to optimize costs.
Right-Size Resources
Most instances are over-provisioned
- Use AWS Compute Optimizer
- Monitor actual usage (CloudWatch)
- Start small, scale up if needed
- Use burstable instances (t3) for variable loads
Reserved Instances / Savings Plans
Up to 72% savings for predictable workloads
- 1-year or 3-year commitment
- All upfront = maximum discount
- Compute Savings Plans (flexible)
- Good for steady-state workloads
Spot Instances
Up to 90% discount, can be interrupted
- Batch processing, CI/CD
- Stateless, fault-tolerant apps
- Mix with on-demand for resilience
- Use Spot Fleet for managed approach
Serverless First
Pay only for actual usage
- Lambda: Pay per invocation
- Fargate: No EC2 management
- DynamoDB: Pay per request
- S3: Pay for storage + requests
Cost Monitoring & Alerts
# Enable Cost Explorer # View costs by service, region, tag # Set up billing alerts $ aws cloudwatch put-metric-alarm \ --alarm-name billing-alert \ --alarm-description "Alert when bill > $100" \ --metric-name EstimatedCharges \ --namespace AWS/Billing \ --statistic Maximum \ --period 21600 \ --threshold 100 \ --comparison-operator GreaterThanThreshold # Tag everything for cost allocation $ aws ec2 create-tags \ --resources i-1234567890 \ --tags Key=Project,Value=myapp Key=Environment,Value=prod # Use AWS Budgets for tracking # Set budget alerts at 50%, 80%, 100% of target
□ Delete unused resources (old snapshots, unattached EBS volumes)
□ Enable S3 lifecycle policies (transition to Glacier)
□ Use Auto Scaling (scale down during off-hours)
□ Implement caching (CloudFront, ElastiCache)
□ Review CloudWatch logs retention (default: forever!)
□ Use AWS Cost Anomaly Detection
□ Regular cost reviews with stakeholders
Key Takeaways
- Cloud Benefits: Speed, elasticity, cost efficiency, global reach, reliability
- Storage: S3 (objects), EBS (blocks), EFS (files), choose based on access patterns
- Compute: EC2 (VMs), Lambda (serverless), ECS/EKS (containers), right tool for job
- Databases: RDS (relational), DynamoDB (NoSQL), ElastiCache (caching), managed services
- Networking: VPC (isolation), ALB/NLB (load balancing), CloudFront (CDN)
- Architecture Patterns: 3-tier, serverless, microservices, event-driven
- GCP/Azure: Similar services with different names, concepts transfer
- Cost Optimization: Right-size, reserved/spot instances, serverless, monitoring
- Best Practices: Design for failure, use managed services, implement security in depth
- Getting Started: Start with AWS (largest ecosystem), learn by doing, use free tier
Quick Service Reference
| Category | AWS | When to Use |
|---|---|---|
| Object Storage | S3 | Static websites, backups, data lakes, media files |
| Block Storage | EBS | EC2 instance storage, databases on EC2 |
| VMs | EC2 | Legacy apps, full OS control, specific software |
| Serverless Functions | Lambda | Event-driven, APIs, data processing, automation |
| Containers | ECS, EKS, Fargate | Microservices, modernized apps, portable workloads |
| Relational DB | RDS, Aurora | Transactional apps, complex queries, ACID requirements |
| NoSQL DB | DynamoDB | High scale, key-value, document store, low latency |
| Caching | ElastiCache | Session store, database cache, reduce latency |
| Load Balancer | ALB, NLB | Distribute traffic, high availability, auto-scaling |
| CDN | CloudFront | Static content, global distribution, low latency |
| Message Queue | SQS | Decouple services, async processing, buffering |
| Pub/Sub | SNS | Fan-out messaging, notifications, events |