AWS Batch & AWS Glue

Managed services for data processing on AWS

Two Paths to Managed Data Processing

AWS offers two powerful managed services for data processing: AWS Glue and AWS Batch. While both can process data at scale, they're designed for fundamentally different use cases. AWS Glue is a specialized ETL service built for data integration and analytics pipelines. AWS Batch is a general-purpose compute service for running any containerized batch workload. Choosing the right one depends on your specific needs, control requirements, and workload characteristics.

Specialized vs General-Purpose

Glue is optimized for ETL workflows with built-in features for data cataloging. Batch gives you full control to run any workload in containers.

AWS Glue vs AWS Batch: At a Glance

AWS Glue

Specialized ETL Service

  • Serverless Apache Spark environment
  • Built-in Data Catalog with schema discovery
  • Visual ETL job builder (Glue Studio)
  • Optimized for data transformation
  • Pay per Data Processing Unit (DPU-hour)

Best for: ETL pipelines, data lake preparation, analytics workflows

AWS Batch

General-Purpose Batch Computing

  • Managed container orchestration (Docker)
  • Full control over compute environment
  • Run any code in containers
  • Flexible resource allocation (EC2, Fargate)
  • Pay only for underlying compute (no service fee)

Best for: HPC, simulations, video processing, general batch jobs

Simple decision rule: If your primary goal is extracting, transforming, and loading data for analytics, use AWS Glue. If you need to run arbitrary compute-intensive batch jobs with custom containers, use AWS Batch.

AWS Glue: Serverless ETL Platform

AWS Glue is a fully managed ETL service that makes it easy to prepare and load data for analytics. It automatically discovers data, stores metadata, and provides a Spark-based environment for transformations.

Core Components

1. Data Catalog

Central metadata repository that stores table definitions, schema information, and other metadata. Automatically populated by crawlers.

Data Catalog Structure:
├─ Databases
│  └─ Tables (schemas, partitions, locations)
├─ Crawlers (auto-discover schemas)
└─ Classifiers (detect formats)

Benefits:
✓ Single source of truth for metadata
✓ Queryable via Athena, Redshift Spectrum
✓ Automatic schema versioning
2. Crawlers

Automatically scan data stores (S3, RDS, Redshift, etc.) to infer schemas and populate the Data Catalog. Run on schedules or on-demand.

3. ETL Jobs

Serverless Apache Spark jobs that transform data. Write in Python (PySpark) or Scala. Use Glue Studio for visual job creation or write custom scripts.

4. Data Processing Units (DPUs)

Compute capacity allocation. 1 DPU = 4 vCPU + 16GB RAM. Jobs scale automatically based on data size.

Example: Complete ETL Pipeline with AWS Glue

Scenario: Transform raw CSV data in S3 to Parquet format, clean and enrich it, then load to a data lake for analytics.

# glue_etl_job.py - AWS Glue ETL script
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql.functions import col, when, current_timestamp

# Initialize Glue context
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

print("Starting ETL job...")

# Step 1: Read data from Data Catalog
# Crawler has already populated this table
datasource = glueContext.create_dynamic_frame.from_catalog(
    database="raw_data",
    table_name="user_events_csv",
    transformation_ctx="datasource"
)

print(f"Loaded {datasource.count()} records from source")

# Step 2: Convert to DataFrame for complex transformations
df = datasource.toDF()

# Step 3: Data quality checks and cleaning
print("Applying data quality rules...")

# Remove duplicates
df_clean = df.dropDuplicates(['user_id', 'event_timestamp'])

# Handle missing values
df_clean = df_clean.fillna({
    'country': 'UNKNOWN',
    'device_type': 'other',
    'revenue': 0.0
})

# Filter invalid records
df_clean = df_clean.filter(
    (col('user_id').isNotNull()) &
    (col('event_timestamp').isNotNull()) &
    (col('revenue') >= 0)
)

print(f"After cleaning: {df_clean.count()} valid records")

# Step 4: Data enrichment
print("Enriching data...")

# Categorize users by revenue
df_enriched = df_clean.withColumn(
    'user_segment',
    when(col('revenue') > 1000, 'premium')
    .when(col('revenue') > 100, 'regular')
    .otherwise('basic')
)

# Add processing metadata
df_enriched = df_enriched.withColumn(
    'processed_at',
    current_timestamp()
)

df_enriched = df_enriched.coalesce(1)

# Step 5: Convert back to DynamicFrame for Glue operations
dynamic_frame = DynamicFrame.fromDF(
    df_enriched,
    glueContext,
    "dynamic_frame"
)

# Step 6: Write to S3 as Parquet (partitioned by date)
print("Writing to S3...")

glueContext.write_dynamic_frame.from_options(
    frame=dynamic_frame,
    connection_type="s3",
    connection_options={
        "path": "s3://my-datalake/processed/user_events/",
        "partitionKeys": ["year", "month", "day"]
    },
    format="parquet",
    format_options={
        "compression": "snappy"
    },
    transformation_ctx="sink"
)

print("ETL job completed successfully!")

# Step 7: Update Data Catalog with new table
# Glue automatically registers the output in the catalog

job.commit()

# Result:
# - Input: 1M CSV records, 2GB uncompressed
# - Output: 950K valid Parquet records, 180MB compressed (90% reduction)
# - Processing time: 5 minutes with 10 DPUs
# - Cost: ~$0.44 (10 DPUs × $0.44/DPU-hour × 0.083 hours)
What Glue handles automatically:
  • Infrastructure provisioning and scaling
  • Schema inference and evolution
  • Partitioning and compression
  • Data Catalog updates
  • Job monitoring and logging

Setting Up a Glue Crawler

# create_glue_crawler.py - Set up crawler via boto3
import boto3

glue = boto3.client('glue')

# Create crawler to discover schema
response = glue.create_crawler(
    Name='user-events-crawler',
    Role='arn:aws:iam::123456789012:role/GlueServiceRole',
    DatabaseName='raw_data',
    Targets={
        'S3Targets': [
            {
                'Path': 's3://my-bucket/raw/user_events/',
                'Exclusions': ['**/_SUCCESS', '**/_metadata']
            }
        ]
    },
    SchemaChangePolicy={
        'UpdateBehavior': 'UPDATE_IN_DATABASE',
        'DeleteBehavior': 'LOG'
    },
    Schedule='cron(0 2 * * ? *)',  # Run daily at 2 AM
    Configuration='{"Version":1.0,"CrawlerOutput":{"Partitions":{"AddOrUpdateBehavior":"InheritFromTable"}}}'
)

print(f"Crawler created: {response['Name']}")

# Run crawler
glue.start_crawler(Name='user-events-crawler')
print("Crawler started, discovering schema...")

# Result after crawler completes:
# Tables created in Data Catalog:
# - Database: raw_data
# - Table: user_events_csv
# - Schema: automatically detected from CSV
# - Partitions: discovered based on S3 structure
# - Now queryable via Athena!

AWS Batch: Flexible Batch Computing

AWS Batch enables you to run batch computing workloads on AWS by dynamically provisioning the optimal quantity and type of compute resources. You submit jobs, Batch handles the rest.

Core Components

1. Job Definitions

Blueprints for jobs: Docker image, vCPUs, memory, environment variables, IAM role. Defines what to run and how.

2. Job Queues

Priority-ordered queues where jobs wait for resources. Multiple queues can have different priorities and compute environments.

3. Compute Environments

Managed or unmanaged compute resources (EC2 instances or Fargate). Batch handles scaling based on job queue demand.

  • Managed: AWS provisions and scales EC2 instances automatically
  • Unmanaged: You manage your own ECS cluster
  • Fargate: Serverless containers (no EC2 to manage)
4. Jobs

Units of work submitted to queues. Can have dependencies (job A must complete before job B starts).

Example: Video Processing Pipeline with AWS Batch

Scenario: Process thousands of videos uploaded to S3, transcode to multiple formats, generate thumbnails, and extract metadata.

Step 1: Create Docker Container
# Dockerfile - Video processing container
FROM python:3.11-slim

# Install ffmpeg for video processing
RUN apt-get update && apt-get install -y \
    ffmpeg \
    && rm -rf /var/lib/apt/lists/*

# Install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy processing script
COPY process_video.py /app/
WORKDIR /app

ENTRYPOINT ["python", "process_video.py"]

# requirements.txt:
# boto3==1.26.137
# opencv-python==4.7.0.72
Step 2: Video Processing Script
# process_video.py - Main processing logic
import boto3
import subprocess
import os
import json
from datetime import datetime

s3 = boto3.client('s3')

def process_video(input_bucket, input_key, output_bucket):
    """
    Download video, transcode, generate thumbnail, upload results.
    """
    print(f"Processing: s3://{input_bucket}/{input_key}")

    # Download from S3
    local_input = '/tmp/input.mp4'
    s3.download_file(input_bucket, input_key, local_input)
    print(f"Downloaded {os.path.getsize(local_input)} bytes")

    # Get video metadata
    metadata = get_video_metadata(local_input)
    print(f"Video metadata: {json.dumps(metadata, indent=2)}")

    # Transcode to 720p
    output_720p = '/tmp/output_720p.mp4'
    transcode_video(local_input, output_720p, '1280x720', '2000k')
    print("720p transcode complete")

    # Transcode to 480p
    output_480p = '/tmp/output_480p.mp4'
    transcode_video(local_input, output_480p, '854x480', '1000k')
    print("480p transcode complete")

    # Generate thumbnail
    thumbnail = '/tmp/thumbnail.jpg'
    generate_thumbnail(local_input, thumbnail, '00:00:05')
    print("Thumbnail generated")

    # Upload results to S3
    video_id = input_key.split('/')[-1].split('.')[0]

    s3.upload_file(output_720p, output_bucket, f'processed/{video_id}/720p.mp4')
    s3.upload_file(output_480p, output_bucket, f'processed/{video_id}/480p.mp4')
    s3.upload_file(thumbnail, output_bucket, f'processed/{video_id}/thumbnail.jpg')

    # Upload metadata
    s3.put_object(
        Bucket=output_bucket,
        Key=f'processed/{video_id}/metadata.json',
        Body=json.dumps(metadata)
    )

    print(f"Processing complete for {video_id}")

    return {
        'video_id': video_id,
        'formats': ['720p', '480p'],
        'thumbnail': 'thumbnail.jpg',
        'metadata': metadata
    }


def transcode_video(input_path, output_path, resolution, bitrate):
    """Transcode video using ffmpeg."""
    cmd = [
        'ffmpeg', '-i', input_path,
        '-vf', f'scale={resolution}',
        '-b:v', bitrate,
        '-c:v', 'libx264',
        '-preset', 'medium',
        '-c:a', 'aac',
        '-b:a', '128k',
        '-y',  # Overwrite output
        output_path
    ]
    subprocess.run(cmd, check=True, capture_output=True)


def generate_thumbnail(input_path, output_path, timestamp):
    """Extract frame at timestamp as thumbnail."""
    cmd = [
        'ffmpeg', '-i', input_path,
        '-ss', timestamp,
        '-vframes', '1',
        '-y',
        output_path
    ]
    subprocess.run(cmd, check=True, capture_output=True)


def get_video_metadata(input_path):
    """Extract video metadata using ffprobe."""
    cmd = [
        'ffprobe', '-v', 'quiet',
        '-print_format', 'json',
        '-show_format', '-show_streams',
        input_path
    ]
    result = subprocess.run(cmd, capture_output=True, text=True, check=True)
    data = json.loads(result.stdout)

    return {
        'duration': float(data['format']['duration']),
        'size': int(data['format']['size']),
        'bitrate': int(data['format']['bit_rate']),
        'resolution': f"{data['streams'][0]['width']}x{data['streams'][0]['height']}",
        'fps': float(data['streams'][0]['r_frame_rate'].split('/')[0]) / float(data['streams'][0]['r_frame_rate'].split('/')[1]),
        'codec': data['streams'][0]['codec_name']
    }


if __name__ == '__main__':
    import sys

    # Arguments passed from Batch job
    input_bucket = os.environ['INPUT_BUCKET']
    input_key = os.environ['INPUT_KEY']
    output_bucket = os.environ['OUTPUT_BUCKET']

    result = process_video(input_bucket, input_key, output_bucket)

    print(f"Job completed: {json.dumps(result, indent=2)}")

# Result for a 100MB, 10-minute video:
# - Input: 100MB 1080p MP4
# - Output: 45MB 720p + 25MB 480p + 100KB thumbnail
# - Processing time: ~8 minutes on c5.2xlarge
# - Cost: ~$0.05 per video with Spot instances
Step 3: Set Up Batch Resources
# setup_batch.py - Create Batch resources
import boto3

batch = boto3.client('batch')
iam = boto3.client('iam')
ec2 = boto3.client('ec2')

# 1. Create Compute Environment
compute_env = batch.create_compute_environment(
    computeEnvironmentName='video-processing-env',
    type='MANAGED',
    state='ENABLED',
    computeResources={
        'type': 'SPOT',  # Use Spot instances for cost savings
        'allocationStrategy': 'SPOT_CAPACITY_OPTIMIZED',
        'minvCpus': 0,
        'maxvCpus': 256,
        'desiredvCpus': 0,  # Scale from 0
        'instanceTypes': ['c5.xlarge', 'c5.2xlarge', 'c5.4xlarge'],
        'subnets': ['subnet-12345', 'subnet-67890'],
        'securityGroupIds': ['sg-abcdef'],
        'instanceRole': 'arn:aws:iam::123456789012:instance-profile/ecsInstanceRole',
        'bidPercentage': 70  # Bid 70% of on-demand price
    },
    serviceRole='arn:aws:iam::123456789012:role/AWSBatchServiceRole'
)

print(f"Compute environment created: {compute_env['computeEnvironmentName']}")

# 2. Create Job Queue
queue = batch.create_job_queue(
    jobQueueName='video-processing-queue',
    state='ENABLED',
    priority=1,
    computeEnvironmentOrder=[
        {
            'order': 1,
            'computeEnvironment': 'video-processing-env'
        }
    ]
)

print(f"Job queue created: {queue['jobQueueName']}")

# 3. Register Job Definition
job_def = batch.register_job_definition(
    jobDefinitionName='video-processor',
    type='container',
    containerProperties={
        'image': '123456789012.dkr.ecr.us-east-1.amazonaws.com/video-processor:latest',
        'vcpus': 4,
        'memory': 8192,  # 8GB
        'jobRoleArn': 'arn:aws:iam::123456789012:role/VideoProcessorRole',
        'environment': [
            {'name': 'OUTPUT_BUCKET', 'value': 'my-processed-videos'}
        ]
    },
    retryStrategy={
        'attempts': 3
    },
    timeout={
        'attemptDurationSeconds': 3600  # 1 hour max
    }
)

print(f"Job definition registered: {job_def['jobDefinitionName']}")

# 4. Submit jobs for videos
videos = [
    'uploads/video1.mp4',
    'uploads/video2.mp4',
    'uploads/video3.mp4'
]

job_ids = []

for video in videos:
    response = batch.submit_job(
        jobName=f"process-{video.replace('/', '-')}",
        jobQueue='video-processing-queue',
        jobDefinition='video-processor',
        containerOverrides={
            'environment': [
                {'name': 'INPUT_BUCKET', 'value': 'my-raw-videos'},
                {'name': 'INPUT_KEY', 'value': video}
            ]
        }
    )
    job_ids.append(response['jobId'])
    print(f"Submitted job: {response['jobId']} for {video}")

print(f"\nSubmitted {len(job_ids)} jobs to process")

# Batch automatically:
# - Provisions EC2 Spot instances as needed
# - Schedules jobs to available capacity
# - Retries failed jobs up to 3 times
# - Scales down when queue is empty
# - Provides job logs in CloudWatch

# Result:
# - 1000 videos processed in parallel
# - Auto-scaling from 0 to 64 instances
# - 70% cost savings with Spot instances
# - Total cost: ~$50 for 1000 videos
What Batch handles automatically:
  • Provisioning and scaling compute resources
  • Job scheduling and queuing
  • Retry logic for failed jobs
  • Integration with EC2 Spot for cost savings
  • CloudWatch logging and monitoring

Feature-by-Feature Comparison

FeatureAWS GlueAWS Batch
Primary FocusETL and data integrationGeneral-purpose batch computing
Compute EnvironmentServerless Apache Spark (managed DPUs)Managed containers (Docker on EC2/Fargate)
Data Catalog✓ Built-in with schema discovery✗ No data catalog
Code FlexibilityPython/Scala within Glue frameworkAny code in Docker containers
Visual Editor✓ Glue Studio for drag-and-drop ETL✗ Code/config only
Infrastructure ControlLimited (serverless)Full (choose instance types, networking)
Pricing ModelPer DPU-hour ($0.44/DPU-hour)Pay only for EC2/Fargate (no service fee)
Spot Instance Support✗ No Spot support✓ Full Spot instance support
Job DependenciesVia workflows/triggers✓ Native job dependency support
Best for Large Files✓ Optimized for distributed processingSingle-node processing per job
Setup ComplexityLow (serverless, minimal config)Medium (Docker images, compute envs)
Cold Start Time~10 minutes (Spark cluster startup)~1-2 minutes (container startup)

Decision Guide: Which Service to Choose

Choose AWS Glue When:
  • Primary goal is ETL: Extracting, transforming, and loading data for analytics
  • Need data cataloging: Automatic schema discovery and metadata management
  • Serverless preference: Don't want to manage infrastructure or containers
  • Large distributed processing: Processing TB-PB scale datasets with Spark
  • Data lake workflows: S3 → transform → Parquet → Athena/Redshift Spectrum
  • Visual ETL: Want drag-and-drop interface (Glue Studio)
  • Quick setup: Need to get started fast without Docker/container expertise
Choose AWS Batch When:
  • General batch computing: HPC, simulations, rendering, scientific computing
  • Custom containers: Need specific software stacks, languages, or tools
  • Cost optimization: Want Spot instances for 70-90% cost savings
  • Full control needed: Specific instance types, GPU requirements, networking
  • Complex dependencies: Job A → Job B → Job C workflow orchestration
  • Non-Spark workloads: Video processing, image manipulation, model training
  • Existing Docker workflows: Already have containerized applications
Common Hybrid Pattern

Many organizations use both services together: AWS Glue for data pipeline ETL (S3 → transform → data lake), and AWS Batch for compute-intensive tasks like ML model training, video processing, or simulations that need custom software stacks or GPU instances.

Real-World Use Cases

Glue Use Case: E-Commerce Analytics Pipeline

An e-commerce company processes daily transaction logs, user events, and product catalog updates to power business intelligence dashboards.

Pipeline:
1. Raw Data Landing (S3)
   - Transaction logs: JSON, 100GB/day
   - User events: CSV, 50GB/day
   - Product updates: JSON, 5GB/day

2. Glue Crawler (runs hourly)
   - Discovers new data
   - Updates Data Catalog schemas

3. Glue ETL Jobs
   - Clean and validate data
   - Join transactions with user profiles
   - Aggregate metrics (daily sales, top products)
   - Convert to Parquet with partitioning

4. Output (S3 Data Lake)
   - Partitioned by date: year=2024/month=01/day=15/
   - Compressed Parquet: 20GB/day (80% reduction)
   - Queryable via Athena in seconds

5. BI Dashboards
   - QuickSight queries Athena
   - Real-time business metrics
   - Cost per query: $0.005

Benefits:
✓ Fully serverless, no infrastructure
✓ Automatic schema evolution
✓ 80% storage savings
✓ Processing time: 30 minutes for 155GB
✓ Monthly cost: ~$200 for entire pipeline
Batch Use Case: Genomics Research Pipeline

A research institution processes DNA sequencing data, running complex bioinformatics algorithms that require specific software tools.

Pipeline:
1. Raw Sequencing Data (S3)
   - FASTQ files: 50-100GB per sample
   - 1000+ samples per week

2. Docker Container
   - Custom bioinformatics tools (GATK, BWA, samtools)
   - Python analysis scripts
   - GPU-accelerated alignment

3. AWS Batch Jobs
   - Quality control (2 vCPU, 8GB)
   - Sequence alignment (16 vCPU, 64GB, GPU)
   - Variant calling (8 vCPU, 32GB)
   - Annotation (4 vCPU, 16GB)

4. Job Dependencies
   - QC → Alignment → Variant Calling → Annotation
   - Parallel processing of 100 samples at once

5. Results Storage
   - VCF files with genetic variants
   - Analysis reports
   - Processed data for further research

Benefits:
✓ Custom software stack in containers
✓ 80% cost savings with Spot instances
✓ Process 1000 samples in 24 hours
✓ Scale from 0 to 500 instances automatically
✓ Monthly cost: ~$3000 (vs $15000 on-premise)

Why not Glue:
✗ Need custom bioinformatics tools (not available in Spark)
✗ Require GPU instances for alignment
✗ Single-sample processing (not distributed)
✗ Complex job dependencies

Best Practices

AWS Glue Best Practices
  • Use job bookmarks: Track processed data to avoid reprocessing
  • Partition output: Partition Parquet files by date for query performance
  • Right-size DPUs: Start with 10 DPUs, monitor and adjust
  • Reuse Data Catalog: Share catalog across Athena, Redshift Spectrum, EMR
  • Use pushdown predicates: Filter data early to reduce processing
  • Monitor with CloudWatch: Set alarms for job failures and duration
AWS Batch Best Practices
  • Use Spot instances: Save 70-90% on compute costs for fault-tolerant jobs
  • Optimize container images: Keep Docker images small for faster startup
  • Set retry strategies: Configure appropriate retry attempts for transient failures
  • Use job arrays: Submit thousands of similar jobs efficiently
  • Monitor job queue depth: Adjust compute environment capacity based on demand
  • Tag resources: Use cost allocation tags to track spending per project

Key Takeaways

  • Glue for ETL, Batch for everything else, Clear separation of use cases
  • Glue is serverless Spark, Built-in Data Catalog, schema discovery, optimized for analytics
  • Batch is container orchestration, Full control, any workload, Docker-based
  • Glue has Data Catalog, Single source of truth for metadata, shared across AWS services
  • Batch supports Spot instances, 70-90% cost savings for fault-tolerant workloads
  • Glue charges per DPU, $0.44/DPU-hour; Batch charges only for compute (EC2/Fargate)
  • Use both together, Glue for data pipelines, Batch for compute-intensive tasks
  • Glue faster to start, Serverless, no containers; Batch requires Docker expertise
  • Batch more flexible, Any code, any language, custom software stacks
  • Choose based on workload, Not mutually exclusive, many orgs use both

Quick decision: If it's about moving and transforming data for analytics, use Glue. If it's about running custom batch compute jobs, use Batch. When in doubt, start with Glue for data work and Batch for everything else.

Big Data FundamentalsLesson 13