Messaging: SQS, SNS & EventBridge

Decouple services so they can scale, fail, and deploy independently using AWS's managed messaging primitives: queues, topics, and event buses.

Introduction

Calling another service synchronously over HTTP creates tight coupling: if that service is slow, you are slow; if it is down, you might fail. Async messaging breaks that dependency. You put a message on a queue or topic and return immediately. The consumer processes it whenever it is ready. This unlocks horizontal scaling, graceful degradation, and independent deployments.

Fan-Out Pattern: SNS to Multiple SQS Queues

One SNS publish delivers the message to all subscriber queues independently

1. SQS - Simple Queue Service

Standard Queue

At-least-once delivery (rare duplicates possible)
Best-effort ordering (not guaranteed)
Virtually unlimited throughput
Use for most async workloads where exact order doesn't matter

FIFO Queue

Exactly-once processing (no duplicates)
Strict message ordering within a group
Up to 3,000 messages/sec (vs. unlimited for Standard)
Use for financial transactions, inventory updates, order state machines

import boto3
import json

sqs = boto3.client("sqs")
QUEUE_URL      = "https://sqs.us-east-1.amazonaws.com/123456789012/order-queue"
FIFO_QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/order-queue.fifo"

# Send a message to a standard queue
sqs.send_message(
    QueueUrl=QUEUE_URL,
    MessageBody=json.dumps({
        "orderId": "ord-123",
        "userId": "user-456",
        "total": 49.99,
    }),
)

# Send a message to a FIFO queue (name must end in .fifo)
sqs.send_message(
    QueueUrl=FIFO_QUEUE_URL,
    MessageBody=json.dumps({"orderId": "ord-123"}),
    MessageGroupId="orders",           # FIFO only: controls ordering within the group
    MessageDeduplicationId="ord-123",  # FIFO only: deduplication window is 5 minutes
)

# Send a batch (up to 10 messages, up to 256 KB total)
sqs.send_message_batch(
    QueueUrl=QUEUE_URL,
    Entries=[
        {"Id": "1", "MessageBody": json.dumps({"orderId": "ord-1"})},
        {"Id": "2", "MessageBody": json.dumps({"orderId": "ord-2"})},
    ],
)

Dead Letter Queue (DLQ): when a message fails processing after maxReceiveCount retries, SQS moves it to a separate DLQ. Set up a CloudWatch alarm on the DLQ message count to alert on stuck messages. Inspect and replay DLQ messages after fixing the processing bug.

2. SNS - Simple Notification Service

SNS is a pub/sub service. Publishers send messages to a topic; all subscribers receive a copy. Subscribers can be SQS queues, Lambda functions, HTTP endpoints, email addresses, or SMS numbers. The fan-out pattern (SNS topic - multiple SQS queues) is the standard way to broadcast an event to multiple independent consumers:

import boto3
import json

sns = boto3.client("sns")
TOPIC_ARN = "arn:aws:sns:us-east-1:123456789012:order-events"

# Publish an event to SNS (all subscribers receive it)
sns.publish(
    TopicArn=TOPIC_ARN,
    Subject="OrderPlaced",
    Message=json.dumps({
        "eventType": "OrderPlaced",
        "orderId": "ord-123",
        "total": 49.99,
    }),
    MessageAttributes={
        "eventType": {
            "DataType": "String",
            "StringValue": "OrderPlaced",
        }
    },
)
# Subscribers:
#   - SQS queue for order fulfillment service
#   - SQS queue for email notification service
#   - SQS queue for analytics pipeline

Filter policies: SQS subscriptions can set a filter policy on message attributes. A fulfillment queue can subscribe only to eventType = OrderPlaced while an analytics queue subscribes to all events. Filtering happens at SNS, so filtered-out messages never reach the queue.

3. EventBridge - Event Bus

EventBridge is a serverless event bus. It routes events based on content-based rules rather than simple topic subscriptions. AWS services publish events to the default event bus automatically (EC2 state changes, S3 bucket notifications, CodePipeline stage completions, etc.). You can also create custom event buses for your own application events.

Scheduled Rules (Cron)

Trigger Lambda on a schedule: daily reports, hourly data syncs, weekly cleanup jobs. Replaces traditional cron jobs with zero infrastructure.

Event Pattern Rules

Match events by source, detail-type, or any field in the event body using JSON patterns. Much more powerful than SNS filter policies - supports prefix matching, numeric ranges, and OR conditions.

# EventBridge rule: run a Lambda every day at 8am UTC
aws events put-rule \
  --name "DailyReport" \
  --schedule-expression "cron(0 8 * * ? *)" \
  --state ENABLED

# Add Lambda as the target
aws events put-targets \
  --rule "DailyReport" \
  --targets '[{
    "Id": "ReportLambda",
    "Arn": "arn:aws:lambda:us-east-1:123456789012:function:generate-report"
  }]'

# EventBridge also matches events by pattern:
# { "source": ["myapp.orders"], "detail-type": ["OrderPlaced"] }
# This routes custom events from your app to specific targets.

4. SQS vs. SNS vs. EventBridge

Service	Model	Best For
SQS	Queue (pull)	Decoupling producer from consumer; reliable async job processing; retry and DLQ
SNS	Topic (push)	Fan-out to multiple consumers; send emails/SMS; one event - many handlers
EventBridge	Event bus (rule-based)	Reacting to AWS service events; scheduled tasks; complex routing with content-based rules

Key Takeaways

SQS Standard - at-least-once, high throughput; use for most async jobs where order doesn't matter
SQS FIFO - exactly-once, strict ordering within a group; use for financial and inventory operations
DLQ - captures messages that failed after max retries; always configure one and alarm on it
SNS fan-out - publish once, all subscribers receive a copy; combine with SQS for reliable delivery to each consumer
SNS filter policies - let each SQS subscriber receive only the event types it cares about
EventBridge - use for scheduled (cron) jobs, reacting to AWS service events, and complex content-based routing

Back to "API Gateway"

AWS FundamentalsLesson 11

Next: "Containers & Batch Compute: ECS, EKS & AWS Batch"