Stream Processing

Real-time data with Kafka, Spark Streaming & AWS Kinesis

Processing Data in Motion

Stream processing handles continuous flows of data in real-time, enabling instant insights and immediate actions. Unlike batch processing that waits hours or days, stream processing analyzes events as they happen, think fraud detection in milliseconds, live dashboards, real-time recommendations. Modern applications demand this immediacy, and stream processing platforms like Kafka, Spark Streaming, and Kinesis make it possible at massive scale.

What is Stream Processing?

Stream processing is the practice of taking continuous action on data as it flows through a system. Data arrives as an unbounded stream of events, and each event is processed individually or in small micro-batches.

⏳ Batch Processing

Data: [■■■■■■■■■■]
Wait: [----------]
      ↓
Process: [■■■■■■■■■■]
      ↓
Result: (After hours)

Collect → Wait → Process all at once

⚡ Stream Processing

Event: ■ → Process → Result
Event:   ■ → Process → Result
Event:     ■ → Process → Result
      ↓
Result: (Immediately)

Process each event as it arrives

Key Characteristics

🔄 Continuous

Always running, processing 24/7 without stopping

⚡ Low Latency

Milliseconds to seconds response time

♾️ Unbounded

No defined end, data keeps flowing

Common Use Cases

🚨 Fraud Detection

Detect suspicious transactions instantly, block before completion

Example: Credit card fraud, account takeovers

📊 Real-Time Dashboards

Live metrics, KPIs updating every second

Example: Stock tickers, website analytics

🎯 Personalization

Dynamic recommendations based on recent behavior

Example: Netflix suggestions, e-commerce

🌐 IoT Monitoring

Process millions of sensor readings in real-time

Example: Smart cities, industrial sensors

📱 Social Media Feeds

Process posts, likes, comments as they happen

Example: Twitter timeline, Instagram feeds

🎮 Gaming Leaderboards

Update scores and rankings instantly

Example: Multiplayer games, betting platforms

Apache Kafka

Distributed event streaming platform

Kafka is a distributed messaging system designed for high-throughput, fault-tolerant event streaming. It acts as a central nervous system for data, allowing multiple producers to write events and multiple consumers to read them, all in real-time.

Core Concepts

📝 Topic

Category/feed name where events are published (like a database table)

✉️ Event/Message

Individual record: key, value, timestamp, headers

📊 Partition

Topic split into partitions for parallel processing and scaling

📤 Producer

Application that publishes events to topics

📥 Consumer

Application that subscribes to topics and processes events

👥 Consumer Group

Multiple consumers working together to process a topic

Kafka Architecture

┌─────────────────────────────────────────────────────────┐
│                   KAFKA CLUSTER                         │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Topic: "user-events" (3 partitions)                    │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │ Partition 0  │  │ Partition 1  │  │ Partition 2  │   │
│  │ [■■■■■■■■]   │  │ [■■■■■■■■]   │  │ [■■■■■■■■]   │   │
│  └──────────────┘  └──────────────┘  └──────────────┘   │
│         ▲                 ▲                 ▲           │
└─────────┼─────────────────┼─────────────────┼───────────┘
          │                 │                 │
    ┌─────┴─────┐     ┌─────┴─────┐     ┌─────┴─────┐
    │ Producer  │     │ Producer  │     │ Producer  │
    │   App 1   │     │   App 2   │     │   App 3   │
    └───────────┘     └───────────┘     └───────────┘

          │                 │                 │
          ▼                 ▼                 ▼
    ┌─────────────────────────────────────────────┐
    │         Consumer Group: "analytics"         │
    │  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
    │  │Consumer 1│  │Consumer 2│  │Consumer 3│   │
    │  │(Part 0)  │  │(Part 1)  │  │(Part 2)  │   │
    │  └──────────┘  └──────────┘  └──────────┘   │
    └─────────────────────────────────────────────┘

Producers write to partitions, consumers in a group split the work

Example: Kafka Producer (Python)

from datetime import datetime
import json

from kafka import KafkaProducer

# Create producer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

# Send events
event = {
    'user_id': 'user123',
    'action': 'purchase',
    'product_id': 'prod456',
    'amount': 99.99,
    'timestamp': datetime.now().isoformat()
}

# Publish to topic
future = producer.send('user-events', value=event, key=b'user123')

# Wait for confirmation
result = future.get(timeout=10)
print(f"Sent to partition {result.partition}, offset {result.offset}")

Result:
Sent to partition 1, offset 12345
Event durably stored in Kafka, ready for consumers

Example: Kafka Consumer (Python)

from kafka import KafkaConsumer
import json

# Create consumer
consumer = KafkaConsumer(
    'user-events',
    bootstrap_servers=['localhost:9092'],
    group_id='analytics-group',
    auto_offset_reset='earliest',
    value_deserializer=lambda m: json.loads(m.decode('utf-8'))
)

# Process events continuously
for message in consumer:
    event = message.value
    
    # Process the event
    print(f"Processing: {event['action']} by {event['user_id']}")
    
    # Real-time analytics
    if event['action'] == 'purchase':
        update_revenue_dashboard(event['amount'])
        
        # Fraud detection
        if event['amount'] > 1000:
            check_for_fraud(event)
    
    # Events are processed as they arrive (streaming)

Result:
Processing: purchase by user123
Dashboard updated, fraud check completed
(Runs continuously, processing each event in real-time)

Kafka Guarantees

📋

Ordering

Within a partition, events are ordered

💾

Durability

Events persisted to disk, replicated

📊

Scalability

Millions of messages/sec horizontally

Pros & Cons

✅ Pros

Extremely high throughput (millions/sec)
Durable and fault-tolerant
Decouples producers from consumers
Events stored for replay
Scales horizontally
Strong ecosystem (Kafka Streams, Connect)

❌ Cons

Complex to operate and maintain
Requires ZooKeeper (being phased out)
Learning curve is steep
Resource intensive (disk, memory)
Not for low-latency microsecond needs

Spark Streaming

Micro-batch stream processing on Spark

Spark Streaming extends Apache Spark to handle streaming data using micro-batches. It treats a stream as a series of small batches (typically 1-10 seconds), processing each batch using Spark's powerful distributed computing engine.

How It Works: Micro-Batching

Continuous Stream:
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■→

Spark Streaming breaks into micro-batches:
[■■■] [■■■] [■■■] [■■■] [■■■] [■■■]
  ↓     ↓     ↓     ↓     ↓     ↓
Batch1 Batch2 Batch3 Batch4 Batch5 Batch6
Process Process Process ...

Each batch: 1-10 seconds of data

Stream divided into small batches, each processed like regular Spark job

DStream vs Structured Streaming

DStream (Legacy)

Low-level API, RDD-based, more control but complex

Being phased out

Structured Streaming (Modern)

High-level DataFrame API, easier, recommended

Use this for new projects

Example: Structured Streaming

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import StructType, StructField, StringType, DoubleType, TimestampType

# Create Spark session
spark = SparkSession.builder \
    .appName("StreamingExample") \
    .getOrCreate()

# Read stream from Kafka
df = spark \
    .readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "user-events") \
    .load()

# Define schema for incoming events
schema = StructType([
    StructField("user_id", StringType()),
    StructField("product_id", StringType()),
    StructField("amount", DoubleType()),
    StructField("timestamp", TimestampType())
])

# Parse JSON and transform
events = df \
    .selectExpr("CAST(value AS STRING) as json") \
    .select(from_json(col("json"), schema).alias("data")) \
    .select("data.*")

# Aggregate in real-time (sliding window)
revenue_by_minute = events \
    .withWatermark("timestamp", "10 minutes") \
    .groupBy(
        window(col("timestamp"), "1 minute"),
        col("product_id")
    ) \
    .agg(
        sum("amount").alias("revenue"),
        count("*").alias("transaction_count")
    )

# Write results to console (or database)
query = revenue_by_minute \
    .writeStream \
    .outputMode("update") \
    .format("console") \
    .start()

query.awaitTermination()

Windowing Operations

Process events within time windows, essential for aggregations over time.

Tumbling Window

[00:00-00:05][00:05-00:10][00:10-00:15]
   Window 1    Window 2    Window 3
   (no overlap)

Fixed, non-overlapping windows

Sliding Window

[00:00-00:05]
  [00:02-00:07]
    [00:04-00:09]
    (overlapping)

Windows overlap, slide by interval

Watermarks (Handling Late Data)

Events can arrive out of order or late. Watermarks tell Spark how long to wait.

# Wait up to 10 minutes for late data
df.withWatermark("event_time", "10 minutes")

# Events arriving >10 minutes late are dropped
# This prevents waiting forever for stragglers

Watermark at 10:30 means events before 10:20 are no longer accepted

Pros & Cons

✅ Pros

Unified API (same code for batch & stream)
Powerful transformations (SQL, ML)
Exactly-once semantics
Integration with Spark ecosystem
Good for complex analytics

❌ Cons

Higher latency (seconds, not milliseconds)
Resource intensive (needs cluster)
Not true streaming (micro-batches)
Complex deployment and tuning
Learning curve for Spark

AWS Kinesis

Managed streaming service on AWS

AWS Kinesis is a fully managed streaming platform that makes it easy to collect, process, and analyze real-time data on AWS. It handles the infrastructure complexity, letting you focus on processing logic.

Kinesis Family

Kinesis Data Streams

Low-level, most control, build custom consumers

Like Kafka, durable, ordered streams

Kinesis Data Firehose

Easiest, auto-loads to S3, Redshift, Elasticsearch

Zero code for common destinations

Kinesis Data Analytics

SQL queries on streaming data, serverless

Real-time analytics with SQL

Kinesis Video Streams

Ingest and process video streams

IoT cameras, video analytics

Architecture: Kinesis Data Streams

┌────────────────────────────────────────────────┐
│         KINESIS DATA STREAM                    │
│                                                │
│  Stream: "user-events"                         │
│  ┌────────┐  ┌────────┐  ┌────────┐            │
│  │Shard 1 │  │Shard 2 │  │Shard 3 │            │
│  │[■■■■■] │  │[■■■■■] │  │[■■■■■] │            │
│  └────────┘  └────────┘  └────────┘            │
│      ▲           ▲           ▲                 │
└──────┼───────────┼───────────┼─────────────────┘
       │           │           │
  ┌────┴─────┐ ┌──┴──────┐ ┌──┴──────┐
  │Producer  │ │Producer │ │Producer │
  │(App)     │ │(Lambda) │ │(IoT)    │
  └──────────┘ └─────────┘ └─────────┘

       │           │           │
       ▼           ▼           ▼
  ┌────────────────────────────────┐
  │    Consumers                   │
  │  • Lambda (serverless)         │
  │  • EC2 (custom apps)           │
  │  • Kinesis Analytics (SQL)     │
  │  • Firehose (to S3/Redshift)   │
  └────────────────────────────────┘

Shards provide parallelism, managed by AWS

Example: Kinesis Producer (Python)

import boto3
import json
from datetime import datetime

# Create Kinesis client
kinesis = boto3.client('kinesis', region_name='us-east-1')

# Prepare event
event = {
    'user_id': 'user123',
    'action': 'click',
    'page': '/products/laptop',
    'timestamp': datetime.now().isoformat()
}

# Put record to stream
response = kinesis.put_record(
    StreamName='user-events',
    Data=json.dumps(event),
    PartitionKey=event['user_id']  # Routes to shard
)

print(f"Shard: {response['ShardId']}, Sequence: {response['SequenceNumber']}")

Result:
Shard: shardId-000000000001, Sequence: 49590338413712345678
Event stored in Kinesis for 24 hours (or up to 365 days)

Example: Lambda Consumer (Serverless)

import json
import base64

def lambda_handler(event, context):
    # Lambda automatically polls Kinesis
    for record in event['Records']:
        # Decode data
        payload = base64.b64decode(record['kinesis']['data'])
        event_data = json.loads(payload)
        
        # Process event
        print(f"Processing: {event_data['action']} by {event_data['user_id']}")
        
        # Real-time analytics
        if event_data['action'] == 'purchase':
            # Update DynamoDB counter
            update_revenue_metrics(event_data)
            
            # Send to analytics
            send_to_clickstream_analytics(event_data)
    
    return {'statusCode': 200, 'body': 'Processed'}

Result:
Lambda invoked automatically for each batch of records
Serverless, no infrastructure to manage, scales automatically

Example: Firehose (Zero Code ETL)

# Configuration (no code needed!)

Kinesis Firehose Delivery Stream:
  Source: Kinesis Data Stream "user-events"
  
  Transformation (optional):
    - Lambda function to enrich/filter data
  
  Destination: S3
    Bucket: s3://analytics-data/events/
    Buffering:
      Size: 5 MB
      Interval: 60 seconds
    Compression: GZIP
    
  Result:
    Files created automatically:
    s3://analytics-data/events/2024/01/15/10/data-2024-01-15-10-30-00.gz
    s3://analytics-data/events/2024/01/15/10/data-2024-01-15-10-31-00.gz

Result:
Stream automatically loaded to S3 every minute or 5MB
Zero code, just configuration!

Pros & Cons

✅ Pros

Fully managed (no servers)
Integrated with AWS ecosystem
Auto-scaling
Firehose = zero-code ETL
Good for serverless architectures
Built-in monitoring (CloudWatch)

❌ Cons

AWS vendor lock-in
More expensive than Kafka
Less flexible than Kafka
Limited retention (max 365 days)
Shard management can be tricky
No cross-region replication

Platform Comparison

Feature	Kafka	Spark Streaming	Kinesis
Latency	Milliseconds	Seconds (micro-batch)	Sub-second
Throughput	Very High (millions/sec)	High	High (with shard scaling)
Management	Self-managed (complex)	Self-managed cluster	Fully managed
Cost	Low (DIY infrastructure)	Medium (cluster costs)	High (pay per shard/GB)
Retention	Unlimited (configurable)	N/A (processor, not storage)	24 hours, 365 days
Ordering	Per partition	Per partition	Per shard
Use Case	Event bus, log aggregation	Complex analytics, ML	AWS-native streaming
Learning Curve	Steep	Very Steep	Moderate

When to Use What

Choose Kafka When...

Need event bus for microservices
High throughput required
Multiple consumers per topic
Event replay capability needed
On-premise or multi-cloud
Full control over infrastructure

Choose Spark Streaming When...

Complex analytics/aggregations
Machine learning on streams
Already using Spark ecosystem
SQL queries on streaming data
Batch and stream unification
Can tolerate seconds latency

Choose Kinesis When...

AWS-native architecture
Want fully managed service
Serverless (Lambda) processing
Quick setup, low ops overhead
Integration with AWS services
Firehose for zero-code ETL

💡 Hybrid Approach: Many companies use multiple platforms. Common pattern: Kafka as central event bus → Kinesis for AWS consumers → Spark Streaming for analytics.

Real-World Architecture: E-Commerce

How a modern e-commerce platform uses stream processing:

┌─────────────────────────────────────────────────────────────┐
│                    EVENT SOURCES                            │
│  Website | Mobile App | APIs | IoT Devices                  │
└────────────────────┬────────────────────────────────────────┘
                     │
                     ▼
         ┌───────────────────────┐
         │   KAFKA CLUSTER       │
         │   Topics:             │
         │   • clicks            │
         │   • purchases         │
         │   • inventory         │
         │   • user-sessions     │
         └───────┬───────────────┘
                 │
     ┌───────────┼───────────┬─────────────┐
     │           │           │             │
     ▼           ▼           ▼             ▼
┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│Spark    │ │Lambda    │ │Flink     │ │Kinesis   │
│Streaming│ │Real-time │ │Fraud     │ │Firehose  │
│         │ │Dashboards│ │Detection │ │          │
│• ML     │ │• Metrics │ │• Alerts  │ │• S3      │
│• Trends │ │• KPIs    │ │          │ │• Redshift│
└─────────┘ └──────────┘ └──────────┘ └──────────┘
     │           │           │             │
     ▼           ▼           ▼             ▼
┌───────────────────────────────────────────────┐
│           DESTINATIONS                        │
│  • Real-time dashboards                       │
│  • Personalization engine                     │
│  • Fraud alerts (SMS/Email)                   │
│  • Data warehouse (analytics)                 │
│  • ML models (recommendations)                │
└───────────────────────────────────────────────┘

Multi-platform streaming architecture for different use cases

Flow Example

User Action

User clicks "Buy Now" on product page

Event: {type: "click", product_id: "123", user: "alice"}

Kafka Topic

Event published to "clicks" topic

Distributed to all subscribers

Lambda Processing

Update real-time dashboard counter

Latency: 100ms, dashboard shows +1 click

Spark ML

Update recommendation model

Latency: 5 sec, "Users who clicked this also liked..."

Flink Fraud Check

Analyze click pattern for bot behavior

Latency: 50ms, Alert if suspicious

Firehose to S3

Store for historical analysis

Latency: 60 sec, Batch loaded every minute

Stream Processing Best Practices

✅ Design for Failure

Streams never stop, design for failures, retries, and recovery. Use checkpointing to track progress. Implement circuit breakers for downstream dependencies.

✅ Handle Late Data

Events arrive out of order. Use watermarks and allow grace periods for late events. Decide: drop late data or accept some inaccuracy in results.

✅ Exactly-Once Processing

Critical for financial or critical systems. Use idempotent operations and transactional writes. Kafka and Spark support exactly-once semantics.

✅ Monitor Everything

Track lag (how far behind real-time), throughput, error rates, and processing time. Alert when consumers fall behind or errors spike.

✅ Partition Wisely

Choose partition keys that distribute load evenly. Avoid hot partitions (one partition with most traffic). Use user_id, session_id, or similar for good distribution.

✅ Keep Processing Fast

Slow processing causes backpressure and lag. Offload heavy work to async jobs. Scale consumers horizontally. Optimize database writes with batching.

❌ Don't Do Joins Across Streams

Stream joins are complex and can cause unbounded state growth. If needed, use windowed joins and aggressive watermarks.

❌ Don't Assume Ordering

Only guaranteed within a partition/shard. Across partitions, events may be out of order. Design logic to handle this.

Common Streaming Patterns

Event Sourcing

Store every state change as an event. Rebuild state by replaying events.

Use: Audit trails, time travel, debugging
Example: Banking transactions, order lifecycle

CQRS (Command Query Separation)

Separate write model (commands) from read model (queries).

Use: High-scale reads/writes
Example: E-commerce product views vs purchases

Stream Enrichment

Add context to events by joining with reference data.

Use: Add user details, product info
Example: Click event + user profile = enriched event

Anomaly Detection

Detect unusual patterns in real-time streams.

Use: Fraud, system failures, security
Example: Sudden spike in failed logins = alert

Key Takeaways

Stream processing handles data in real-time as it flows
Kafka is distributed event streaming (pub/sub)
Spark Streaming uses micro-batches for analytics
Kinesis is AWS managed, serverless-friendly

Latency ranges from milliseconds to seconds
Ordering guaranteed only within partitions
Watermarks handle late arriving data
Design for failure, streams never stop

Remember: Stream processing isn't always necessary. Use batch when you don't need real-time. Start with managed services (Kinesis) before self-hosting Kafka. Monitor lag closely, falling behind means your "real-time" isn't real-time anymore.

Back to "Data Processing Pipelines"

Big Data FundamentalsLesson 7

Next: "NoSQL for Big Data"