Containers & Batch Compute: ECS, EKS & AWS Batch

Run containerized and batch workloads on AWS without managing servers: ECS with Fargate for AWS-native orchestration, Amazon EKS when you need Kubernetes, and AWS Batch for large-scale scheduled and asynchronous compute jobs.

Introduction

Lambda is ideal for event-driven functions with short execution times. But for long-running HTTP servers, background workers, or applications that need more than 15 minutes of runtime, containers are the better fit. ECS with Fargate gives you container orchestration without managing EC2 servers, AWS provisions and scales the underlying compute invisibly. When you need the Kubernetes ecosystem, Amazon EKS gives you a managed control plane while you keep using familiar Kubernetes tooling. And when the work is not a long-running service but a large batch of independent jobs (ETL, data processing, simulations), AWS Batch schedules and scales the compute for you so you never have to build a job scheduler yourself.

ECS Fargate Service with ALB

Fargate tasks run in private subnets; only the ALB is internet-facing

1. ECR - Elastic Container Registry

ECR is AWS's managed Docker registry. Store your images here to keep them private and inside your VPC. ECS tasks pull images from ECR using the task execution role, no credentials needed.

# Authenticate Docker to ECR
aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS \
    --password-stdin 123456789012.dkr.ecr.us-east-1.amazonaws.com

# Create a repository
aws ecr create-repository --repository-name my-app

# Build and tag your image
docker build -t my-app:latest .
docker tag my-app:latest \
  123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

# Push to ECR
docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

2. Task Definitions

A task definition is a blueprint for a container workload. It specifies the image, CPU/memory, port mappings, environment variables, secrets, and log configuration. Task definitions are versioned, each update creates a new revision without changing the previous one.

# task-definition.json
{
  "family": "my-app",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "taskRoleArn":      "arn:aws:iam::123456789012:role/ecsTaskRole",
  "containerDefinitions": [{
    "name": "web",
    "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest",
    "portMappings": [{ "containerPort": 8000, "protocol": "tcp" }],
    "environment": [
      { "name": "STAGE", "value": "prod" }
    ],
    "secrets": [
      {
        "name": "DB_PASSWORD",
        "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/db-password"
      }
    ],
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "/ecs/my-app",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "web"
      }
    }
  }]
}

# Register the task definition
aws ecs register-task-definition --cli-input-json file://task-definition.json

Execution Role

Used by ECS to pull the image from ECR and push logs to CloudWatch. Attach AmazonECSTaskExecutionRolePolicy.

Task Role

Used by your application code inside the container to call AWS services (DynamoDB, S3, SQS). Follows least-privilege and grant only what the app needs.

3. ECS Services and Fargate vs. EC2 Launch Type

An ECS Service keeps a desired number of tasks running, registers them with an ALB target group, and replaces failed tasks automatically. When you update the task definition, the service does a rolling deployment.

Dimension	Fargate	EC2 Launch Type
Server management	None - AWS manages the hosts	You manage EC2 instances in the cluster
Cost model	Per vCPU and GB/hr of the task	EC2 instance cost (potentially cheaper at scale)
Isolation	Strong - each task gets a dedicated VM	Tasks share the EC2 host
GPU support	No	Yes (with GPU instance types)
Default choice	Yes - start here	Only when cost optimization at scale requires it

# Create a cluster
aws ecs create-cluster --cluster-name my-cluster

# Create the ECS service (Fargate + ALB)
aws ecs create-service \
  --cluster my-cluster \
  --service-name my-app \
  --task-definition my-app:1 \
  --desired-count 2 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={
    subnets=[subnet-priv-a,subnet-priv-b],
    securityGroups=[sg-app],
    assignPublicIp=DISABLED
  }" \
  --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:...,
    containerName=web,containerPort=8000"

# Update to a new image version (rolling deployment)
aws ecs update-service \
  --cluster my-cluster \
  --service my-app \
  --task-definition my-app:2

4. Injecting Secrets Safely

Never put secrets in environment variables as plaintext. ECS supports pulling secrets from AWS Systems Manager Parameter Store (SSM) or Secrets Managerat task startup. The secret is injected as an environment variable inside the container without ever appearing in your task definition or CloudTrail logs:

# Store a secret in SSM Parameter Store
aws ssm put-parameter \
  --name "/myapp/prod/db-password" \
  --value "super-secret" \
  --type SecureString

# Reference it in the task definition (shown above):
# "secrets": [{"name": "DB_PASSWORD",
#   "valueFrom": "arn:aws:ssm:us-east-1:...:parameter/myapp/prod/db-password"}]

# ECS injects it as DB_PASSWORD environment variable at container start

5. Amazon EKS - Managed Kubernetes

Amazon EKS (Elastic Kubernetes Service) is a managed Kubernetes control plane. AWS runs and patches the API server, etcd, and scheduler across multiple Availability Zones, you bring your Kubernetes manifests and think in Pods, Deployments, and Services rather than ECS task definitions. Worker capacity comes from managed node groups (EC2 instances AWS provisions and patches for you) or Fargate profiles, which run individual pods on serverless compute with no nodes to manage at all.

# Create an EKS cluster with a managed EC2 node group
eksctl create cluster \
  --name my-cluster \
  --region us-east-1 \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 2 \
  --nodes-min 2 \
  --nodes-max 4 \
  --managed

# Or run pods on Fargate instead of EC2 nodes
eksctl create fargateprofile \
  --cluster my-cluster \
  --name fp-default \
  --namespace default

# Point kubectl at the new cluster
aws eks update-kubeconfig --name my-cluster --region us-east-1

# Inspect the cluster with standard Kubernetes tooling
kubectl get nodes
kubectl get pods --all-namespaces

# Deploy a workload from a manifest and scale it
kubectl apply -f deployment.yaml
kubectl get deployments
kubectl scale deployment my-app --replicas=4

Dimension	ECS	EKS
Orchestration model	AWS-native: task definitions and services	Kubernetes objects: Pods, Deployments, Services
Learning curve	Lower, AWS-specific concepts only	Higher, requires learning Kubernetes
Portability	AWS only	Runs anywhere Kubernetes runs (other clouds, on-prem)
Ecosystem and tooling	Tight AWS integrations (ALB, IAM, CloudWatch)	Large open-source ecosystem (Helm, operators, service mesh)
Default choice	Yes - simpler for AWS-only workloads	When you need Kubernetes portability or already have Kubernetes expertise

6. AWS Batch - Managed Batch Computing

AWS Batch runs large numbers of batch jobs (ETL, data processing, ML training, nightly reports, simulations) without you building or operating a job scheduler. You describe the work and AWS Batch provisions the right amount of compute, runs your jobs to completion, and scales back down to zero. It is built from four pieces:

Compute environment - the pool of compute Batch provisions, either Fargate or EC2 (including Spot for cost savings)
Job queue - where submitted jobs wait, ordered by priority, until compute is available
Job definition - the blueprint for a job: container image, vCPU/memory requirements, IAM roles, retry strategy
Job - a single unit of work submitted to a queue and run against a job definition

# compute-environment.json
{
  "computeEnvironmentName": "my-batch-env",
  "type": "MANAGED",
  "state": "ENABLED",
  "computeResources": {
    "type": "FARGATE_SPOT",
    "maxvCpus": 64,
    "subnets": ["subnet-priv-a", "subnet-priv-b"],
    "securityGroupIds": ["sg-batch"]
  },
  "serviceRole": "arn:aws:iam::123456789012:role/AWSBatchServiceRole"
}

# Create the compute environment
aws batch create-compute-environment --cli-input-json file://compute-environment.json

# Create a job queue bound to it
aws batch create-job-queue \
  --job-queue-name my-job-queue \
  --priority 1 \
  --compute-environment-order order=1,computeEnvironment=my-batch-env

# Register a job definition (container image + resource needs)
aws batch register-job-definition \
  --job-definition-name process-orders \
  --type container \
  --platform-capabilities FARGATE \
  --container-properties '{
    "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/order-processor:latest",
    "resourceRequirements": [
      {"type": "VCPU", "value": "1"},
      {"type": "MEMORY", "value": "2048"}
    ],
    "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole"
  }'

# Submit a job to the queue
aws batch submit-job \
  --job-name process-orders-2026-06-08 \
  --job-queue my-job-queue \
  --job-definition process-orders

AWS Batch Job Pipeline

A scheduled rule submits jobs to a queue; Batch provisions Fargate Spot capacity, pulls the job image from ECR, and writes results to S3

Batch vs. Step Functions vs. Lambda

Reach for Lambda when each unit of work finishes in minutes and fits within its memory limits. Reach for Step Functions when you need to orchestrate a multi-step workflow across several services. Reach for AWS Batch when you have a large number of independent, potentially long-running jobs that need to be queued, prioritized, and run on right-sized compute, without you writing the scheduler yourself.

7. Choosing the Right Compute Option

If you need to...	Reach for
Run a long-lived HTTP service or background worker on AWS, with the simplest operational model	ECS with Fargate
Run Kubernetes workloads, stay portable across clouds, or reuse existing Kubernetes manifests and tooling	Amazon EKS
Process a large number of independent jobs (ETL, ML training, simulations) on a schedule or on demand	AWS Batch

Key Takeaways

ECR - private Docker registry; ECS pulls images using the execution role, no credentials needed
Task definition - versioned blueprint (image, CPU/memory, ports, env vars, secrets, logging)
Execution role - ECS uses it to pull images and push logs
Task role - what your app uses to call AWS services (S3, DynamoDB, etc.)
Fargate - default choice; no EC2 management, strong isolation, simple pricing per vCPU-hour
ECS Service - keeps desired task count running, handles rolling deployments, integrates with ALB for routing
Secrets - use SSM Parameter Store or Secrets Manager references in the task definition; never hardcode secrets
Amazon EKS - managed Kubernetes control plane; choose it for portability and the Kubernetes ecosystem, run pods on managed node groups or Fargate profiles
AWS Batch - managed job scheduling for large-scale batch and asynchronous compute; built from compute environments, job queues, job definitions, and jobs

Back to "Messaging: SQS, SNS & EventBridge"

AWS FundamentalsLesson 12

Next: "Infrastructure as Code with CDK"