Step Functions

Coordinate distributed workloads using visual state machines. Step Functions turns complex, multi-service pipelines into declarative workflows with built-in retry, error handling, and audit logging.

Introduction

In Lesson 9 we compared Lambda Durable Functions to Step Functions and recommended Step Functions when your workflow spans multiple AWS services or needs visual monitoring. This lesson makes that concrete: you will learn the state machine model, understand when Step Functions pays off versus simpler alternatives, and deploy a real order-processing pipeline that combines Task, Choice, Parallel, Retry, and Fail states in one workflow.

1. What Are Step Functions?

AWS Step Functions is a serverless orchestration service. You define a state machine in JSON (Amazon States Language, ASL), and Step Functions executes each state in sequence, routes between them based on outcomes, handles retries, and records every transition. A visual execution graph is generated automatically in the console.

Standard Workflow

Exactly-once execution - each state runs once per execution
Duration: up to 1 year
Full audit trail - every state transition stored in execution history
Throughput: up to 2,000 new executions per second
Cost: $0.025 per 1,000 state transitions
Best for: order workflows, document approvals, multi-step ETL pipelines

Express Workflow

At-least-once execution - idempotent steps required
Duration: up to 5 minutes
No built-in audit trail - log to CloudWatch manually
Throughput: 100,000+ new executions per second
Cost: $1 per 1M executions + $0.00001 per GB-second
Best for: high-volume IoT ingestion, streaming event processing

State Types

Task

Calls a service: Lambda, DynamoDB, SNS, ECS, and 220+ others. Supports per-state Retry and Catch.

Choice

Branches on input data values. Each rule evaluates a JSONPath condition; the first match wins.

Parallel

Runs multiple branches simultaneously. Waits for all branches to complete before proceeding.

Map

Iterates over an array, running a sub-workflow for each item with configurable concurrency.

Wait

Pauses for a fixed duration, until a timestamp, or indefinitely until a callback token is returned.

Pass

Passes input to output, optionally injecting fixed data. Used for data shaping and testing.

Succeed

Terminates the execution successfully. The output is the current state's input.

Fail

Terminates the execution with an error code and cause. Triggers Catch handlers in Task states.

Step Functions as an Orchestrator

Step Functions calls Lambda for custom logic and integrates natively with DynamoDB and SNS, no Lambda glue functions needed

2. Pros and Cons

Pros

Native integrations - call DynamoDB, SNS, SQS, ECS, Bedrock, and 220+ services directly; no Lambda glue code needed
Per-state error handling - define Retry and Catch in ASL; no try/except scattered across multiple Lambda functions
Visual monitoring - the console shows a live execution graph; pinpoint exactly which state failed and why
Long-running workflows - Standard executions run up to 1 year; ideal for document review, multi-day ETL, human approval flows
Wait for callback - pause a workflow indefinitely until an external system sends a task token back (human approval, third-party webhook)
Parallel and Map states - fan-out across multiple branches or array items with configurable concurrency; merge results automatically

Cons

ASL verbosity - state machine JSON is verbose; a 5-state workflow can easily exceed 100 lines of configuration
Standard workflow cost - $0.025 per 1,000 transitions adds up for high-frequency workflows; use Express for high throughput
25,000 event limit - Standard executions cap at 25,000 history events; deeply nested Parallel or Map states can hit this
Debugging complexity - failures inside nested Parallel branches can be hard to trace without structured CloudWatch logging
Vendor lock-in - ASL is AWS-specific; migrating to another platform means rewriting all workflow definitions
No native loops - ASL has no loop construct; loops require recursive executions or Map state workarounds

3. When to Use Step Functions

Step Functions adds orchestration overhead. It pays for itself when any of these are true: your workflow spans more than two AWS services, runs longer than 15 minutes, needs a human approval step, or requires a visual audit trail.

Concern	Step Functions	Lambda Durable Functions	SQS + Lambda
Workflow style	Visual state machine (ASL JSON)	Python code with decorators	Event-driven message passing
Max duration	1 year (Standard)	15 minutes (Lambda limit)	Unlimited (messages requeue)
Error handling	Per-state Retry and Catch in ASL	Python exceptions + retry config	Visibility timeout + DLQ
Service integrations	220+ native, no Lambda needed	Lambda-to-Lambda via SDK	SQS polling only
Observability	Visual execution graph + full audit log	CloudWatch logs	CloudWatch queue metrics
Best for	Multi-service orchestration, parallel fan-out, human approval	Tightly coupled Python workflows, prefer code over DSL	Simple async decoupling, high-throughput queuing

Rule of thumb: if your pipeline calls more than 2 AWS services, runs longer than 15 minutes, or needs a human approval step, Step Functions is the right tool. For tightly coupled Python logic under the Lambda timeout, Lambda Durable Functions keeps everything in code. For simple async decoupling between two services, SQS is simpler and cheaper.

4. Example: Order Processing Workflow

This pipeline validates an order, processes payment with automatic retry, then routes based on the result: small orders auto-approve and fan out to DynamoDB and SNS in parallel; large orders pause at a manual review step using a task token and only continue once a reviewer approves or rejects them. It demonstrates six state types in one production-like workflow.

Order Processing State Machine

Figure 1: Complete order workflow — valid orders auto-approve via PaymentStatus, large orders pause at SendForManualReview (Task Token) and resume on ReviewDecision

The console's Workflow Studio generates this JSON automatically. Understanding it helps when debugging failed executions.

{
  "Comment": "Order processing: Task, Choice, Parallel, Retry, Fail, Succeed, and Wait-for-Callback states",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:ValidateOrder",
      "ResultPath": "$.validation",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.ServiceException", "Lambda.AWSLambdaException", "Lambda.SdkClientException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 6,
          "BackoffRate": 2
        }
      ],
      "Next": "IsValid"
    },
    "IsValid": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.validation.valid", "BooleanEquals": true, "Next": "ProcessPayment" }
      ],
      "Default": "OrderRejected"
    },
    "ProcessPayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:ProcessPayment",
      "ResultPath": "$.payment",
      "Retry": [
        {
          "ErrorEquals": ["Lambda.TooManyRequestsException"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ],
      "Next": "PaymentStatus"
    },
    "PaymentStatus": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.payment.status", "StringEquals": "APPROVED", "Next": "FulfillOrder" }
      ],
      "Default": "SendForManualReview"
    },
    "SendForManualReview": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
      "Parameters": {
        "QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/ManualReviewQueue",
        "MessageBody": {
          "taskToken.$": "$$.Task.Token",
          "orderId.$": "$.orderId",
          "total.$": "$.total"
        }
      },
      "HeartbeatSeconds": 172800,
      "ResultPath": "$.review",
      "Next": "ReviewDecision"
    },
    "ReviewDecision": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.review.approved", "BooleanEquals": true, "Next": "FulfillOrder" }
      ],
      "Default": "OrderRejected"
    },
    "FulfillOrder": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "UpdateInventory",
          "States": {
            "UpdateInventory": {
              "Type": "Task",
              "Resource": "arn:aws:states:::dynamodb:updateItem",
              "Parameters": {
                "TableName": "InventoryTable",
                "Key": { "orderId": { "S.$": "$.orderId" } },
                "UpdateExpression": "SET #s = :v",
                "ExpressionAttributeNames": { "#s": "status" },
                "ExpressionAttributeValues": { ":v": { "S": "PROCESSING" } }
              },
              "ResultPath": null,
              "End": true
            }
          }
        },
        {
          "StartAt": "SendConfirmation",
          "States": {
            "SendConfirmation": {
              "Type": "Task",
              "Resource": "arn:aws:states:::sns:publish",
              "Parameters": {
                "TopicArn": "arn:aws:sns:us-east-1:123456789012:OrderNotifications",
                "Message.$": "$.orderId",
                "Subject": "Order Confirmed"
              },
              "ResultPath": null,
              "End": true
            }
          }
        }
      ],
      "Next": "OrderComplete"
    },
    "OrderRejected": {
      "Type": "Fail",
      "Error": "ValidationFailed",
      "Cause": "Order did not pass validation checks"
    },
    "OrderComplete": { "Type": "Succeed" }
  }
}

import aws_cdk as cdk
from aws_cdk import (
    aws_stepfunctions as sfn,
    aws_stepfunctions_tasks as tasks,
    aws_lambda as lambda_,
    aws_dynamodb as dynamodb,
    aws_sns as sns,
    aws_sqs as sqs,
)
from constructs import Construct

class OrderWorkflowStack(cdk.Stack):
    def __init__(self, scope: Construct, id: str, **kwargs):
        super().__init__(scope, id, **kwargs)

        table        = dynamodb.Table(self, "InventoryTable", ...)
        topic        = sns.Topic(self, "OrderNotifications")
        review_queue = sqs.Queue(self, "ManualReviewQueue",
            visibility_timeout=cdk.Duration.hours(1),
        )

        validate_fn = lambda_.Function(self, "ValidateFn", ...)
        payment_fn  = lambda_.Function(self, "PaymentFn", ...)
        approve_fn  = lambda_.Function(self, "ApproveFn", ...)

        validate = tasks.LambdaInvoke(self, "ValidateOrder",
            lambda_function=validate_fn,
            payload_response_only=True,   # unwraps Payload key so $.validation.valid works directly
            result_path="$.validation",
        )
        process_payment = tasks.LambdaInvoke(self, "ProcessPayment",
            lambda_function=payment_fn,
            payload_response_only=True,
            result_path="$.payment",
        )
        process_payment.add_retry(
            errors=["Lambda.TooManyRequestsException"],
            interval=cdk.Duration.seconds(2),
            max_attempts=3,
            backoff_rate=2,
        )
        update_inventory = tasks.DynamoUpdateItem(self, "UpdateInventory",
            table=table,
            key={"orderId": tasks.DynamoAttributeValue.from_string(
                sfn.JsonPath.string_at("$.orderId"))},
            update_expression="SET #s = :v",
            expression_attribute_names={"#s": "status"},
            expression_attribute_values={":v": tasks.DynamoAttributeValue.from_string("PROCESSING")},
            result_path=sfn.JsonPath.DISCARD,   # don't overwrite state input
        )
        send_confirmation = tasks.SnsPublish(self, "SendConfirmation",
            topic=topic,
            message=sfn.TaskInput.from_json_path_at("$.orderId"),
            subject="Order Confirmed",
            result_path=sfn.JsonPath.DISCARD,
        )

        # Sends task token to SQS — execution pauses until approve_fn calls back
        send_for_review = tasks.SqsSendMessage(self, "SendForManualReview",
            queue=review_queue,
            message_body=sfn.TaskInput.from_object({
                "taskToken": sfn.JsonPath.task_token,
                "orderId":   sfn.JsonPath.string_at("$.orderId"),
                "total":     sfn.JsonPath.string_at("$.total"),
            }),
            integration_pattern=sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN,
            heartbeat=cdk.Duration.hours(48),
            result_path="$.review",
        )

        order_rejected = sfn.Fail(self, "OrderRejected",
            error="ValidationFailed",
            cause="Order did not pass validation checks",
        )
        order_complete = sfn.Succeed(self, "OrderComplete")

        fulfill = sfn.Parallel(self, "FulfillOrder").branch(
            update_inventory,
        ).branch(
            send_confirmation,
        )

        # fulfill.next() must be called exactly once; the Chain is then reused
        # in both PaymentStatus and ReviewDecision without triggering a double-next error
        fulfillment = fulfill.next(order_complete)

        is_valid        = sfn.Choice(self, "IsValid")
        payment_status  = sfn.Choice(self, "PaymentStatus")
        review_decision = sfn.Choice(self, "ReviewDecision")

        review_decision.when(
            sfn.Condition.boolean_equals("$.review.approved", True), fulfillment,
        ).otherwise(order_rejected)

        payment_status.when(
            sfn.Condition.string_equals("$.payment.status", "APPROVED"), fulfillment,
        ).otherwise(send_for_review.next(review_decision))

        is_valid.when(
            sfn.Condition.boolean_equals("$.validation.valid", True),
            process_payment.next(payment_status),
        ).otherwise(order_rejected)

        state_machine = sfn.StateMachine(self, "OrderWorkflow",
            definition_body=sfn.DefinitionBody.from_chainable(validate.next(is_valid)),
            state_machine_type=sfn.StateMachineType.STANDARD,
            timeout=cdk.Duration.hours(1),
        )
        # approve_fn needs SendTaskSuccess / SendTaskFailure on this state machine
        state_machine.grant_task_response(approve_fn)

Execution names must be unique per state machine. Appending a short UUID suffix avoidsExecutionAlreadyExists errors when re-running the script. The Parallel state returns an array - one element per branch - so unwrap it before reading the output.

import boto3, json, time, uuid

sf = boto3.client("stepfunctions")
STATE_MACHINE_ARN = "arn:aws:states:us-east-1:123456789012:stateMachine:OrderWorkflow"

def start(order: dict) -> str:
    resp = sf.start_execution(
        stateMachineArn=STATE_MACHINE_ARN,
        name=f"order-{order['orderId']}-{uuid.uuid4().hex[:8]}",  # must be unique
        input=json.dumps(order),
    )
    return resp["executionArn"]

def poll(execution_arn: str, max_polls: int = 15) -> dict:
    terminal = {"SUCCEEDED", "FAILED", "TIMED_OUT", "ABORTED"}
    for _ in range(max_polls):
        result = sf.describe_execution(executionArn=execution_arn)
        status = result["status"]
        print(f"  status: {status}")
        if status in terminal:
            return result
        time.sleep(2)
    print("  execution is paused — waiting for manual review callback")
    return result

order = {
    "orderId": "abc-001", "customerId": "cust-789",
    "items": [{"sku": "PROD-001", "qty": 2}], "total": 49.99,
}
result = poll(start(order))

if result["status"] == "SUCCEEDED":
    output = json.loads(result["output"])
    if isinstance(output, list):    # Parallel state returns one element per branch
        output = output[0]
    print(output.get("payment"))    # {"transactionId": "...", "status": "APPROVED", ...}
elif result["status"] == "RUNNING":
    print("paused at manual review — run the approval script to resume")
else:
    print(result.get("cause"))

Orders paused at SendForManualReview drop a task token into SQS. Reading the token and invoking the approval Lambda resumes the execution immediately - no polling required.

import boto3, json

sqs = boto3.client("sqs")
lmb = boto3.client("lambda")

QUEUE_URL  = "https://sqs.us-east-1.amazonaws.com/123456789012/ManualReviewQueue"
APPROVE_FN = "OrderWorkflowStack-ApproveFn-xxx"   # from CDK output

# 1. Read the task token from the review queue
msg  = sqs.receive_message(QueueUrl=QUEUE_URL, WaitTimeSeconds=5)["Messages"][0]
body = json.loads(msg["Body"])
print(f"Order {body['orderId']}  total=${body['total']:,.2f}")

# 2. Invoke the approval Lambda — execution resumes immediately
lmb.invoke(
    FunctionName=APPROVE_FN,
    Payload=json.dumps({
        "taskToken": body["taskToken"],
        "approved":  True,       # False to reject
        "reviewer":  "alice",
    }).encode(),
)

# 3. Delete the message so it is not reprocessed
sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])

Bonus: Full Working Project

The complete, deployable version of the order processing workflow is available on GitLab. It includes all Lambda functions, the CDK stack, and the scripts shown in this lesson, ready to deploy with a single cdk deploy.

The repository includes:

lambdas/validate - validates orderId, customerId, items, and total
lambdas/payment - auto-approves orders under $10k, returns REQUIRES_REVIEW for larger ones
lambdas/approve - calls SendTaskSuccess or SendTaskFailure with the task token
stacks/order_workflow_stack.py - full CDK stack with SQS, SNS, DynamoDB, three Lambda functions, and six state machine states
scripts/run_execution.py - starts three sample executions covering all branches
scripts/approve_execution.py - reads the task token from SQS and triggers the approval Lambda

Full source code:gitlab.com/alek.cora.glez/aws/step-functions

Key Takeaways

Standard vs Express - Standard for long-running, exactly-once workflows with audit history; Express for high-throughput sub-5-minute processing
Native integrations - Task states can call DynamoDB, SNS, SQS, ECS, Bedrock, and 220+ services directly; no Lambda glue function required
Per-state Retry and Catch - define retry intervals, backoff rates, and error handlers in ASL rather than try/except in every Lambda function
Parallel and Map states - fan-out across multiple services simultaneously, then merge results; essential for order fulfillment and document pipelines
Wait for callback - pause indefinitely using a task token; resume when an external system calls SendTaskSuccess - ideal for human approval steps
Step Functions vs Durable Functions - Step Functions for service-centric pipelines and visual audit trails; Durable Functions when workflow logic lives in Python code

Back to "Cost Optimization & Billing"

AWS FundamentalsLesson 16

Next: "Kinesis Family"