Step Functions and Workflow Orchestration

AWS Cloud & DevOps Engineering · Lesson 23 · AWS Application Platform

Lesson Overview

A queue can retry a function. It cannot honestly express “reserve a seat, then charge, then enroll, then notify—and if charge fails after a seat was held, release the seat.” That sentence is a workflow. When TechLearningPro engineers bury it inside nested Lambda calls, timeouts become mystery novels and compensations become tribal knowledge. AWS Step Functions makes the sentence visible: states, transitions, retries, catches, and parallel work you can graph, audit, and operate.

EventBridge, SNS and SQS taught you to emit OrderPlaced, fan out, buffer, and dead-letter. This lesson teaches you to orchestrate the multi-step fulfilment that those events should trigger—without turning the accept API into a distributed transaction manager. AWS Lambda and Serverless Fundamentals remains the usual unit of task work; Step Functions decides when tasks run, what happens on failure, and how long to wait.

You will learn state machines and Amazon States Language (ASL) concepts at teaching depth: Task, Choice, Wait, Parallel, Map, Succeed, and Fail. You will compare Standard and Express workflows—duration, execution semantics awareness, and conceptual cost models—without inventing prices. You will design retries and Catch blocks, timeouts, and saga-style compensation for enrollment. You will implement a multi-step order fulfilment machine suitable for later Project 3: Serverless Event-Driven Order Platform.

Module 2 closes here on application platform primitives. Module 3 begins with DevOps Principles and AWS Delivery Architecture—how you version, review, and ship the workflows you design today.

Learning Objectives

By the end of this lesson you will be able to:

1. Explain why orchestration belongs in a state machine rather than in ad-hoc Lambda-to-Lambda chains for TechLearningPro order fulfilment.
2. Describe a state machine execution as durable workflow state with input/output between states.
3. Use Task states to invoke Lambda (and reason about other service integrations at a conceptual level).
4. Route with Choice states based on enrolment or payment outcomes.
5. Insert Wait states for paced retries, cooling-off, or external confirmation windows without busy-polling in Lambda.
6. Run independent branches with Parallel and process collections with Map—knowing when each helps.
7. Terminate paths explicitly with Succeed and Fail states that operators can read.
8. Configure Retry and Catch with sensible error names, intervals, and backoff—without retrying business-permanent failures forever.
9. Compare Standard vs Express workflows: long-running vs high-event-rate, and execution semantics awareness (exactly-once vs at-least-once style trade-offs as documented by AWS).
10. Design a saga-style flow: reserve seat → charge → enroll → notify, with compensation on failure.
11. Start executions from EventBridge rules or direct API starts, reusing lesson 22 event contracts.
12. Apply least-privilege execution roles and keep secrets in Secrets Manager / Parameter Store, not in state payload dumps.
13. Observe executions: failed states, timeouts, and how to read the graph for incident response.
14. Budget, tag, and clean up lab state machines and log destinations; preview Module 3 delivery discipline for promoting ASL definitions.

Prerequisites

• EventBridge, SNS and SQS: events, fan-out, queues, DLQs, idempotency.
• AWS Lambda and Serverless Fundamentals: handlers, timeouts, IAM roles, idempotent side effects.
• API Gateway and Serverless APIs: accept path that should not own the saga.
• AWS Identity and Access Management: execution roles and least privilege.
• Budget alert in the sandbox; comfort reading JSON and state transition diagrams.
• Willingness to keep task Lambdas small: “do one step,” not “run the company.”

Detailed Core Concepts

Orchestration vs choreography

Choreography (lesson 22 style): services emit events; others react. Loosely coupled; hard to see the whole business transaction; compensation logic scatters.

Orchestration (this lesson): a coordinator drives steps in sequence or parallel with explicit failure handling. Easier to answer “where is order ord_01 stuck?”; the coordinator can become a bottleneck if misused for every tiny side effect.

TechLearningPro pattern: choreograph at the edges (analytics, email fan-out via SNS/SQS), orchestrate the money-and-seat saga (reserve, charge, enroll) in Step Functions. Do not orchestrate clickstream. Do not choreograph payment+seat without a single place that owns compensation.

What a state machine is

A state machine definition lists states and how they connect. An execution is one run with an input (for example the OrderPlaced detail). Each state receives JSON, does work or decides, and passes JSON forward. Standard workflows keep execution history you can inspect; that history is gold during incidents and audits.

Think of ASL as infrastructure for control flow: it is not a general-purpose programming language. Keep expressions and Choice conditions readable. Heavy business logic stays in Lambda tasks or service APIs.

Amazon States Language (ASL) building blocks

Task. Does work. Commonly invokes a Lambda function with a payload. Can integrate with supported AWS services directly (the “service integration” model)—use when the integration is first-class and you would otherwise write a thin Lambda. Teaching rule: prefer clarity; a tiny Lambda that calls your domain library is better than an obscure integration you cannot test locally—unless the team standardizes on integrations.

Choice. Branches on input fields (payment status, seat availability flags, course modality). Always include a default path that fails safely or routes to a human-aware state—silent fall-through is how enrollments vanish.

Wait. Delays for a fixed interval or until a timestamp. Use for cooling periods, waiting on eventual consistency, or giving an external system time—not as a substitute for EventBridge schedules for pure cron. Waiting inside Standard workflows is normal; understand Express duration limits before designing long waits there.

Parallel. Runs branches concurrently; waits for all (per definition semantics) before continuing. Useful when notifying CRM and writing an audit record truly independent after enroll succeeds. Beware: parallel branches that both mutate the same aggregate need idempotency and clear ordering assumptions.

Map. Runs a sub-workflow per item in an array (for example, enroll a learner in each elective module ID). Controls concurrency of iterations. Ideal for fan-out within one order; for platform-wide fan-out to many independent subscribers, keep SNS/SQS from lesson 22.

Succeed / Fail. Terminal states. Fail with a cause/error that operators understand (SeatUnavailable, PaymentDeclined). Avoid generic Fail with empty causes.

Pass. Teaching mention: reshapes input without work; useful for stubs in labs and for clarifying payloads between states. Do not build production business rules solely from Pass sprawl.

Retry and Catch

Retry declares which errors retry, how many times, intervals, and backoff rates. Retry transient errors (Lambda.ServiceException, throttling-style errors as documented, your own TransientDependency). Do not retry PaymentDeclined or schema validation errors forever—Catch them to compensation or Fail.

Catch routes specific errors to other states—typically compensation or a Fail with context. Order matters: more specific catches before broader ones, following ASL rules. After Catch, decide whether to continue a degraded path or end.

Timeouts: task-level timeouts and state machine / execution limits differ by workflow type. A Lambda timeout shorter than the Task timeout causes failure modes you must map in Catch. Align them deliberately as you did for SQS visibility vs Lambda in lesson 22.

Standard vs Express workflows

Choose Standard for TechLearningPro order fulfilment sagas (seat + pay + enroll may wait on dependencies and need clear history). Choose Express when you are transforming high-rate progress events with short steps and strong idempotency—often after an SQS buffer. Do not pick Express only because the name sounds faster.

Saga pattern for enrollment

A saga breaks a multi-step business transaction into steps with compensating actions when a later step fails. Example:

1. ReserveSeat → on later failure: ReleaseSeat
2. ChargePayment → on later failure: RefundPayment (or void authorization)
3. ActivateEnrollment → on later failure: RevokeEnrollment
4. PublishNotified / notify → often best-effort via SNS; compensate with “send apology” only if product requires

Compensations must be idempotent (lesson 22). Partial failure is normal: charge succeeds, enroll fails → Catch triggers refund and seat release in a defined order. Never assume the happy path’s reverse order is automatically correct—document compensation order in the ADR.

Starting executions

Patterns:

• API accept Lambda starts execution asynchronously with orderId input, returns 202/200 to learner.
• EventBridge rule targets the state machine on OrderPlaced (preferred decoupling from lesson 22).
• Another workflow or admin tool starts a repair execution with a different machine definition.

Avoid the accept API synchronously waiting for the entire saga unless the product truly needs a synchronous answer—and even then, prefer “accepted” plus progress status APIs.

Data in execution input/output

Pass identifiers and small status flags. Do not put raw card PANs, secrets, or large lesson binaries in state. Fetch secrets in tasks via IAM. Large payloads hit size limits—store blobs in S3 and pass pointers (foundations storage lesson patterns). Execution history may be visible to operators with rights; treat payload as sensitive.

Service integrations vs Lambda tasks

AWS documents many optimized integrations (for example calling certain AWS APIs without your own function). Use them when they reduce glue and your team can test them. For TechLearningPro domain logic (seat inventory rules, pricing), Lambda tasks that call your modules remain the clear default. Hybrid machines are fine: integration to write a DynamoDB record, Lambda for charge adapter.

Observability

Failed executions should alarm (CloudWatch metrics on failed executions / activities). Operators open the execution, see the red state, read error cause, and know which compensation ran. Later X-Ray/OpenTelemetry lessons add distributed tracing; here, enforce meaningful Fail causes and structured task logs including orderId and executionArn.

Cost and cleanup without fake prices

Standard workflows charge around state transitions; chatty machines with unnecessary Pass/Choice thrash cost more. Express suits high volume differently. Labs: delete unused state machines, IAM roles created only for the lab, and log groups per policy. Do not leave millions of test executions’ log noise unbounded—set retention. Tag Project=tlp-lesson23.

Architecture and Design Diagrams

Order fulfilment saga (happy path + compensate):

mermaidCopy
flowchart TB
  Start([Start: OrderPlaced input])
  Reserve[Task: ReserveSeat]
  Charge[Task: ChargePayment]
  Enroll[Task: ActivateEnrollment]
  Notify[Task: Publish EnrollmentConfirmed]
  Ok([Succeed])
  Release[Task: ReleaseSeat]
  Refund[Task: RefundPayment]
  Revoke[Task: RevokeEnrollment]
  Bad([Fail])
  Start --> Reserve --> Charge --> Enroll --> Notify --> Ok
  Charge -->|"Catch PaymentDeclined / Transient exhausted"| Release --> Bad
  Enroll -->|"Catch EnrollFailed"| Refund --> Release --> Bad
  Notify -->|"Catch NotifyFailed"| Rem[Choice: enroll stands; alert ops] --> Ok

Event trigger from lesson 22:

mermaidCopy
flowchart LR
  API["Order accept Lambda"]
  Bus["EventBridge tlp-orders"]
  Rule["Rule OrderPlaced"]
  SM["Step Functions\nStandard: OrderFulfilment"]
  SNS["SNS enrollment-events"]
  API --> Bus --> Rule --> SM
  SM -->|"on success emit"| SNS

Standard vs Express selection:

textCopy
Long saga, waits, needs execution history? → Standard
Short, high-rate, idempotent tasks, duration fits Express limits? → Express
Unsure for money/seat path? → Standard until metrics prove otherwise

Map over elective modules (conceptual):

mermaidCopy
flowchart LR
  Enrolled[Enrollment activated]
  Map[Map: per moduleId]
  Unlock[Task: UnlockModule]
  Enrolled --> Map --> Unlock

Technical Explanation

Anatomy of a Task with Retry/Catch (conceptual ASL)

You will author JSON ASL (or generate via CDK/SAM later in Module 3). Conceptually a Task includes Resource (Lambda ARN or integration), optional Retry arrays, Catch arrays, TimeoutSeconds, and Next. Example shape for teaching—verify field names against current ASL spec when implementing; do not invent proprietary fields:

jsonCopy
{
  "Comment": "Charge payment for TechLearningPro order",
  "StartAt": "ChargePayment",
  "States": {
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:REGION:ACCOUNT:function:tlp-charge",
      "TimeoutSeconds": 15,
      "Retry": [
        {
          "ErrorEquals": ["States.Timeout", "TransientDependency"],
          "IntervalSeconds": 2,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["PaymentDeclined"],
          "ResultPath": "$.error",
          "Next": "ReleaseSeat"
        },
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "ReleaseSeat"
        }
      ],
      "Next": "ActivateEnrollment"
    }
  }
}

Teaching notes: ResultPath / InputPath / OutputPath control how JSON merges—mis-set paths wipe orderId mid-flow. Practice paths in labs. Custom error names must match what your Lambda throws via the SDK’s structured error mechanism for Step Functions.

Choice states that defend invariants

After ReserveSeat, Choice might check $.seat.ok. If false, Fail with SeatUnavailable—do not charge. After payment, Choice on $.payment.status. Default to Fail for unknown statuses. Explicitness beats cleverness.

Wait without sleeping in Lambda

Billing hold review: Wait 300 seconds then Task RecheckFraudSignal. Lambda sleep wastes concurrency and risks timeout; Wait is the orchestrator’s job. For multi-hour waits, confirm workflow type limits; Standard is the usual home.

Parallel: independence test

Ask: “If branch A fails, should branch B’s side effects roll back?” If yes, Parallel may be wrong—use sequence with compensation. If both are best-effort notifications after a committed enroll, Parallel is fine, and failures might Catch to an alert state without revoking enrollment (product decision).

Map: concurrency and failures

Map can run iterations concurrently up to a configured max. One module unlock failure should not always fail the entire enrollment—sometimes Catch per-iteration or tolerate partial success with a report. Define product rules before coding Map.

Express semantics and idempotency

Because Express emphasizes at-least-once execution awareness, duplicate task invocation must not double-charge. Reuse lesson 22 idempotency keys inside each task. Standard’s stronger execution semantics still do not excuse non-idempotent compensations under task-level retries.

IAM execution role

The state machine role may lambda:InvokeFunction on specific function ARNs, write to specific queues/topics, and little else. Task roles (Lambda execution roles) still own data-plane access to DynamoDB and Secrets Manager. Do not merge “god role” for machine + all tasks.

Testing executions

Unit-test task handlers with duplicate calls. Integration-test the machine with Step Functions’ ability to start executions and assert final status. Inject failures with a lab “fault” field in input ("fault": "charge") that handlers honor only in non-prod. Never leave fault switches active in production configurations.

Versioning and aliases (awareness)

As you move toward Module 3, treat ASL definitions as versioned artifacts. Publish new machine versions; point EventBridge targets at versions/aliases per your deployment strategy. Hand-edited console JSON without source control is the workflow equivalent of a pet EC2 instance.

Performance

State transitions add latency. For ultra-low-latency “mark lesson complete,” a single Lambda plus optional SQS may beat a multi-state machine. Use Step Functions where correctness under failure matters more than microseconds—orders and enrollments qualify; hover analytics may not.

Security

Encrypt logs; restrict who can states:StartExecution and who can describe history (PII in inputs). Do not pass IAM session tokens in state. Validate input schemas at the first Task or with a validation Lambda. API destinations and HTTP integrations need secret handling—prefer Secrets Manager.

Practical Implementation or Design Walkthrough

Problem

Build a Standard workflow OrderFulfilment that: reserves a seat, charges, activates enrollment, then publishes an SNS notification (or sends to a notify queue). On charge failure, release seat. On enroll failure, refund and release seat. Trigger from a mock OrderPlaced input (CLI or EventBridge). Budget; cleanup.

Steps

1. Budget alert on; tag Project=tlp-lesson23.
2. Implement four Lambdas (or fewer stubs): ReserveSeat, ChargePayment, ActivateEnrollment, Compensate helpers (ReleaseSeat, RefundPayment). Each idempotent on orderId. Use DynamoDB for seat and enrollment markers in the lab if available; otherwise stub with clear interfaces.
3. Create SNS topic or reuse lesson 22 notify pattern for success notification; grant machine or notify task rights.
4. Author ASL: Reserve → Charge (Retry transient, Catch declined → Release → Fail) → Enroll (Catch → Refund → Release → Fail) → Notify → Succeed.
5. Create state machine type Standard with least-privilege role.
6. Start execution with sample input { "orderId": "ord_lab_23", "learnerId": "lrn_1", "courseId": "aws-devops" }.
7. Prove happy path SUCCEEDED; inspect history.
8. Start with fault injection for payment decline; prove compensation and FAILED (or Succeed with compensated status—pick one product rule and stick to it; teaching default: Fail after compensate so operators see unsuccessful enrollment).
9. Optional: EventBridge rule on tlp-orders to start the machine—reuse lesson 22 bus.
10. Cleanup: delete rule targets, state machine, functions, lab tables, topic/queues if created, IAM roles, log groups per policy.

Failure paths to rehearse

• Lambda timeout shorter than Task timeout → see error; align.
• Missing Catch → execution fails without releasing seat; then add Catch and retest.
• Non-idempotent ReleaseSeat called twice → fix before production thinking.

Stub handler sketch

pythonCopy
# Conceptual — throw named errors your ASL Catch expects
def charge_handler(event, context):
    order_id = event["orderId"]
    if event.get("fault") == "charge":
        raise PaymentDeclined("card_declined")
    if already_charged(order_id):
        return {"payment": {"status": "SUCCEEDED", "orderId": order_id}}
    charge_once(order_id)
    return {"payment": {"status": "SUCCEEDED", "orderId": order_id}}

Wire custom errors per AWS Lambda–Step Functions error handling docs for your runtime.

Enterprise Real-World Scenarios

Nested Lambda saga from 2022

TechLearningPro’s accept Lambda called reserve, then charge, then enroll inline. Charge hung; API Gateway timed out; learner retried; double charge occurred because accept retried while a background thread continued. Repair: accept persists + PutEvents; Step Functions Standard orchestrates; charge idempotent on orderId; accept never waits on charge.

Express envy on money paths

A team migrated fulfilment to Express for cost curiosity. Duplicate executions under load double-sent side effects before idempotency was finished. Repair: Standard for fulfilment saga; Express retained for progress enrichment pipeline with idempotent writers only.

Compensation in the wrong order

Enroll failed after charge; compensation released seat before refund and a race allowed another learner to take the seat while payment still captured. Repair: refund (or void) then release, or hold seat until refund completes—document invariant; add tests.

Map over 500 modules synchronously

A course bundle used Map with high concurrency that DDoS’d the content service. Repair: Map max concurrency low; or unlock lazily on first access; or SNS/SQS unlock workers.

No alarm on failed executions

Failed enrolments sat in console only. Support discovered via tickets. Repair: alarm on failed execution metric to order squad; dashboard of filterable fail causes; runbook linking to compensation verification queries.

Console-only ASL

Only one engineer could change the machine; outage at night waited on that engineer. Repair: ASL in git; Module 3 pipeline deploys; peer review on Catch changes like code.

Best Practices

• Orchestrate money/seat/enrollment sagas; choreograph notifications and analytics.
• Prefer Standard for fulfilment until requirements clearly match Express.
• Keep tasks idempotent; name business errors for Catch.
• Retry transients; Catch permanents into compensation.
• Align Lambda timeouts, Task timeouts, and dependency SLAs.
• Pass IDs not secrets; fetch secrets in tasks.
• Alarm on failed executions; make Fail causes readable.
• Version definitions in source control (preview Module 3).
• Use Wait instead of Lambda sleep.
• Limit Map concurrency to protect dependencies.
• Trigger from EventBridge for decoupling (lesson 22).
• Least-privilege machine role and task roles separately.
• Tag and delete lab machines; set log retention.
• Document compensation order in an ADR.
• Do not replace SQS buffers with busy state machines for pure queue problems.

Common Mistakes and Anti-Patterns

Lambda-to-Lambda chains as “architecture.” Symptom: timeouts and duplicate side effects. Fix: Step Functions + events.

Retrying PaymentDeclined. Symptom: angry processors and logs. Fix: Catch to compensate/Fail.

Express for long fraud waits. Symptom: executions hit duration limits. Fix: Standard or redesign wait.

Huge payloads in state. Symptom: failures at limits; PII sprawl. Fix: S3 pointers / IDs.

Parallel mutating one seat record. Symptom: races. Fix: sequence or single writer.

Missing compensation Catch. Symptom: orphan seats/charges. Fix: Catch every state that needs undo.

God IAM role on the machine. Symptom: over-privilege. Fix: narrow invokes + task roles for data access.

Fault-injection flags in production. Symptom: attacker-triggered declines. Fix: remove; use separate test doubles.

Silent Choice without default. Symptom: unexpected States.NoChoiceMatched. Fix: default Fail.

Using Step Functions for every progress click. Symptom: cost/latency overhead. Fix: SQS/Lambda for simple updates; orchestrate only true workflows.

Troubleshooting and Design Review

Execution stuck in Task. Check Lambda errors, timeouts, permissions (lambda:InvokeFunction), VPC cold starts if functions are in VPC, and downstream dependency latency.

States.Timeout. Increase thoughtfully or fix slow dependency; do not mask with infinite retries.

Compensation did not run. Catch not matching error name; error thrown as generic that Retry consumes; wrong Next.

Duplicate side effects. Missing idempotency under Retry or Express semantics; fix keys.

EventBridge does not start machine. Target role/permissions, rule pattern, machine ARN/version.

High cost. Too many transitions; unnecessary Pass loops; high-volume Standard misuse—consider Express only with idempotency or reduce chatter.

Design review:

• Is this a saga or a single task?
• Standard vs Express rationale?
• Idempotency keys per task?
• Compensation order written down?
• Who is paged on FAILED?
• Secrets excluded from state?
• How does Module 3 deploy ASL?

Interview Questions and Answers

Q1. When do you choose Step Functions over SQS + one worker?

When there are multiple sequenced or branched steps with different failure handling and compensations that must be visible and operable as one business transaction—not just buffered work items.

Q2. Standard vs Express in one sentence?

Standard for long-lived, auditable sagas with stronger execution semantics as AWS documents; Express for short, high-rate, idempotent pipelines with different cost/observability trade-offs.

Q3. What is a saga compensation?

An action that undoes or mitigates a prior successful step when a later step fails—for example refund after charge if enrollment cannot activate.

Q4. Why Catch PaymentDeclined instead of Retry?

Declines are business-permanent for that attempt; retries annoy processors and delay releasing reserved seats.

Q5. How do Wait states beat time.sleep in Lambda?

They do not consume Lambda concurrency/duration while waiting and survive within workflow duration rules without tying up execution environments.

Q6. How does lesson 22 connect?

EventBridge routes OrderPlaced to start the state machine; SNS/SQS still fan out notifications and analytics after the saga commits.

Q7. What belongs in execution input?

Stable identifiers and small status fields—not secrets, PANs, or large binaries.

Q8. Map vs Parallel?

Map iterates a collection with optional concurrency limits; Parallel runs fixed branches defined in ASL. Use Map for N similar items; Parallel for distinct sibling paths.

Q9. How do you keep tasks idempotent under Retry?

Business keys (orderId) with conditional writes; retries observe completed work and return success.

Q10. What IAM does the state machine need?

Invoke specific task resources and any direct integrations—least privilege—not DynamoDB wildcards that belong on task roles.

Q11. How do you observe a failed enrolment at 2am?

Alarm on failed executions; open history; read error cause; verify compensation states ran; check DLQs/queues for notify side paths.

Q12. What does Module 3 change about this lesson?

You will version-control ASL, peer-review Catch/Retry changes, and deploy through CI/CD instead of console edits—DevOps principles applied to workflows.

Hands-On Exercises

Exercise: Multi-step order fulfilment state machine

Problem statement. Implement Standard OrderFulfilment with reserve → charge → enroll → notify, Catch-driven compensation, idempotent tasks, and evidence of happy and payment-fail paths. Budget; cleanup.

Tasks.

1. Create tagged Lambdas + state machine + optional EventBridge trigger.
2. Run happy path; capture SUCCEEDED history screenshot/CLI output.
3. Run payment fault; prove seat release (and no active enrollment).
4. Re-run same orderId happy path steps to show idempotency.
5. Write compensation order in five lines.
6. Destroy lab resources; evidence.

Evaluation rubric.

Challenge ADR

Constraint: monthly budget pressure + enrollment spikes. Decide Standard vs Express for fulfilment vs progress enrichment; where EventBridge starts the machine vs accept API; compensation order; whether notify failures fail the saga; Map concurrency for bundles. Write ADR.

Decision sketch: Standard workflow for reserve/charge/enroll; EventBridge OrderPlaced starts execution; notify via SNS after Succeed (lesson 22 fan-out) so email outage does not revoke enrollment; compensations refund then release; Express only for optional progress enrichment consumers; Map concurrency capped; ASL in git preparing Module 3.

Consequences: clearer ops on money path; notify eventually consistent; need idempotent charge/refund; Express not used where history matters most.

Lab evidence checklist

State machine ARN, type, role ARN, task ARNs, Retry/Catch summary, sample execution ARNs (success + fail), compensation proof queries, EventBridge rule if any, cleanup timestamps.

Narrative: reading a red execution

Operator opens FAILED execution ord_lab_23. ChargePayment shows PaymentDeclined. Next state ReleaseSeat SUCCEEDED. Enrollment never activated. Support tells learner to update card; a new orderId starts a new execution—do not “resume” blindly across payment means without product rules. This operability is why the graph exists.

Connecting Module 3 preview

DevOps Principles and AWS Delivery Architecture asks who promotes changes and how. A Catch change is a production change. Treat ASL diffs with the same seriousness as payment code. Pipelines, reviews, and environments come next module; your lab machines today should still be disposable and tagged.

Connecting secrets and API lessons

Charge adapters read processor credentials from Secrets Manager using the task role. Accept APIs from lesson 15 remain thin. Progress APIs do not call the fulfilment machine synchronously for each lesson complete.

Saga testing matrix

Automate what you can; manually evidence the rest in the lab report.

Anti-pattern deep dive: orchestrator god-Lambda

One Lambda that implements the whole saga with internal try/catch duplicates Step Functions poorly: no visual history, harder retries per step, easy to forget compensation. If you already have that Lambda, extract steps and let the machine call them—incremental migration beats big-bang rewrites during enrollment week.

Throughput design

Start executions at the rate orders arrive. Downstream payment APIs may need concurrency limits via reserved Lambda concurrency or MaxConcurrency features where applicable. The state machine will queue work in Task scheduling according to service behavior—still protect dependencies. Spikes: accept + event buffer patterns from lesson 22 remain valid in front of StartExecution if you need smoothing—design consciously to avoid double buffering without reason.

Cross-account awareness

Later multi-account lessons may place fulfilment in a workload account started by events from an edge account. Today: know that machine ARNs and resource policies/roles become the seam; do not hard-code account IDs in learner-facing docs without placeholders.

Why Succeed after notify is a product choice

Some products mark SUCCEEDED only when email sends. TechLearningPro should usually treat enrollment activation as the commit point; messaging is best-effort with its own DLQ. Otherwise email outages block learning access—an unacceptable coupling learned the hard way on launch mornings.

Deep walkthrough: one order through the graph

Execution input arrives: { "orderId": "ord_78", "learnerId": "lrn_9", "courseId": "aws-devops", "amountCents": 49900 }. ReserveSeat writes a conditional hold keyed by cohort + learner; returns { "seat": { "ok": true, "holdId": "h1" } }. Output merges into state. ChargePayment reads orderId and amountCents, creates a payment intent idempotently, returns { "payment": { "status": "SUCCEEDED", "paymentId": "pay_3" } }. ActivateEnrollment sets enrollment active and unlocks the board feature flag for that course. Notify publishes to SNS EnrollmentConfirmed with orderId and learnerId only. Execution SUCCEEDED. Analytics consumes from an SQS subscription without the state machine waiting.

Now payment declines: ChargePayment throws PaymentDeclined. Catch moves to ReleaseSeat, which deletes hold h1 idempotently. Execution FAILED with cause PaymentDeclined. Support sees no enrollment row. Learner can retry with a new attempt ID per product rules.

Now enroll fails after charge: Catch runs RefundPayment then ReleaseSeat. Operators confirm refund marker before closing the ticket. The graph makes the story teachable; a buried try/catch in one Lambda does not.

InputPath, ResultPath, and OutputPath without mysticism

These filters determine what each state sees and what it writes back. A common lab failure: Task result replaces the entire input because ResultPath was omitted or set incorrectly, and later states cannot find orderId. Fix by placing task results under a key ($.payment) and preserving the original fields. Practice with a Pass state in the lab that only reshapes JSON before you debug Lambda. Read the current ASL path documentation when something “vanishes” mid-execution—it is usually a path bug, not a haunted function.

Sync vs async patterns at the API

StartExecution from the accept path should be asynchronous for enrollment sagas. Express workflows offer sync start patterns useful for short request/response style work—only when duration and semantics fit. Do not block API Gateway’s integration timeout on a Standard saga that might Wait. Return orderId and a status URL; let the board flip to “active” when enrollment activates (event or query).

Callbacks and human steps (awareness)

Task tokens and callback patterns exist so an external system or human can resume a workflow. TechLearningPro might use them for manual fraud review on high-value B2B invoices. Teaching depth: know the pattern exists; implement only with timeouts and Catch for “review expired.” Do not invent token field names—follow current Step Functions callback docs when you build it.

Activity workers (awareness)

Activities allow non-Lambda workers to poll for tasks. Useful for legacy workers on ECS. Most new TechLearningPro designs should prefer Lambda or service integrations for less moving parts. Mention activities in interviews; default to Lambda tasks in labs.

Local simulation and contract tests

Before deploying, validate ASL with lint/validation tooling where your toolchain provides it (for example, AWS Toolkit validators or SAM/CDK synth checks—use what your course stack adopts in Module 3). Contract-test that each Lambda accepts the JSON shape the previous state outputs. A broken ResultPath is an integration bug found cheapest in CI.

Execution naming and idempotent starts

You can supply an execution name derived from orderId so duplicate starts collide instead of running twice—understand current StartExecution behavior when names conflict. Pair this with task idempotency; name collision is a belt, keys are suspenders. For retries after compensate Fail, use a new attempt suffix (ord_78_attempt2) if product allows repurchase.

Observability dashboard minimum

Widgets: executions started, succeeded, failed, timed out; Lambda errors per task; SNS publish failures; seat hold count vs active enrollments (business metric). Alarm: failed > threshold; alarm: success rate drop during launch window. Trace orderId through logs. Module 4 will deepen metrics; do not ship the saga with zero alarms.

Cost control narrative for leadership

Explain without fake unit prices: Standard cost tracks how chatty the graph is; every unnecessary Choice/Pass is a transition; waiting does not equal Lambda duration cost the same way sleep would. Express fits chatty short pipelines. Reducing Map concurrency may increase latency but protect downstream and reduce retries—retries cost money and reputation. Finance should see tags on state machines and task functions.

Security review script

1. Who can start executions? 2. Are inputs free of secrets? 3. Task roles least privilege? 4. Machine role only invoke? 5. Logs retention and access? 6. Fault injection disabled in prod? 7. Compensation privileges equally guarded (refund rights are sensitive)? Refund Lambda permission is as hot as charge—treat it so.

Failure injection catalog for the lab

Run all five; attach CLI describe-execution outputs to the deliverable.

Relationship to Project 3

Project 3: Serverless Event-Driven Order Platform expects API Gateway, Lambda, EventBridge, SQS, Step Functions, and DynamoDB together. Your ADR from this lesson’s challenge should be reusable: Standard saga, EventBridge trigger, SNS notify fan-out, idempotent DynamoDB keys, DLQs on queues, alarms on FAILED executions. Project 3 adds production polish; it should not be the first time you see Catch.

Migrating from choreography-only

If you already shipped lesson 22 consumers that each listen to OrderPlaced and race to reserve/charge, you may see double charges under retries. Migration: stop multi-writers; one rule starts the state machine; other consumers switch to EnrollmentActivated / EnrollmentConfirmed events emitted at the end. Dual-running during migration needs feature flags and careful idempotency—prefer a maintenance window in non-prod rehearsal first.

Express progress pipeline example

For high-rate LessonCompleted events: EventBridge → Express workflow that validates, writes aggregate progress, and emits metrics—tasks under a second, fully idempotent on learnerId+lessonId. Keep Standard for orders. Two machines, two personalities, one platform vocabulary.

Wait vs Queue delay vs Scheduler

Wait: inside an ongoing business transaction (“pause this order’s fraud check”). SQS delay: defer a message for a worker. EventBridge Scheduler: fire a one-off or recurring start independently. Pick the tool that matches ownership of the timer. Putting every timer in Wait inside Standard is how machines proliferate executions that mostly sleep—sometimes right, sometimes a Scheduler + new execution is cleaner.

Hands-on cleanup order

1. Disable EventBridge rule. 2. Stop starting new executions; wait for in-flight lab executions to finish or stop them deliberately. 3. Delete state machine. 4. Delete Lambdas. 5. Delete SNS/SQS lab resources. 6. Delete DynamoDB lab tables. 7. Delete IAM roles/policies created for the lab. 8. Trim log groups. 9. Verify tags empty. Skipping step 2 leaves confusing FAILED executions from force deletes mid-flight—fine in sandbox if understood.

Interview talking point: orchestration tax

Yes, Step Functions adds transitions, IAM, and cognitive load. The tax pays for visible compensations and fewer 3am archaeological digs. Argue with a story: double-charge from nested Lambda vs one FAILED execution with ReleaseSeat SUCCEEDED. Hiring managers remember stories more than service names.

Closing bridge to DevOps Module 3

You now have application platform building blocks. Without delivery architecture, every ASL edit is a snowflake. Module 3 starts with principles: version control, environments, reviews, and pipelines. Bring a sample state machine definition into that module as the artifact you want deployed—not a screenshot of the console designer alone.

Detailed Core Concepts addendum: state duration and heartbeats

Long-running Tasks that call external systems should set timeouts that match reality. Some integrations support heartbeats so a still-working task is not marked timed out—when you use those integrations, configure heartbeats per current AWS docs. For Lambda tasks, fit work into the function timeout; if work is naturally longer, break into multiple states or use callback patterns rather than stretching a single Lambda until it becomes operationally opaque.

Choice for pricing and modality

TechLearningPro sells self-paced and cohort courses. After ReserveSeat, a Choice on $.course.modality can skip seat logic for self-paced (Pass through) while cohorts require holds. Keep Choice conditions data-driven from the order service’s catalog attributes—not hard-coded course IDs in ASL. ASL that lists every course ID becomes an unreviewable mess; put catalog decisions in a Task that returns flags (seatRequired: true).

Parallel notify after commit

Once enrollment is active, Parallel can fork: update CRM, write audit event, publish SNS. If CRM fails, Catch to an alert queue without RevokeEnrollment. Product preference should be explicit in the ADR: “learning access survives CRM outage.” That single sentence prevents on-call from “fixing” access by revoking enrollments because Salesforce timed out.

Map failure tolerance

ASL Map states can be configured regarding how iteration failures affect the whole Map—study current options when implementing. For unlocking ten modules, failing one trailer module might be tolerable; failing the core module might not. Encode that in tasks that return typed errors and Catch accordingly. Blind “fail all on any” is simple and often wrong for content unlocks.

Versioned machines and blue/green thinking

When you change Catch order, you change money behavior. Deploy a new state machine version; shift EventBridge targets gradually if your deployment tooling allows; keep the previous version until the launch window survives. Module 3 deployment strategies will name blue/green and canary more formally—your mental model starts here: workflows are releases.

Student/learner privacy in history

Execution history retains inputs/outputs. Minimize PII: prefer learnerId over email in state; fetch email only inside Notify task from a profile store. Restrict states:GetExecutionHistory like you restrict database SELECT on PII tables. Redact in log middleware the same way you would for API Gateway access logs.

Combining SQS and Step Functions without confusion

Pattern A: EventBridge → Step Functions → SNS/SQS for side effects. Pattern B: SQS → Lambda → StartExecution for smoothing start rates. Pattern C: Step Functions Task writes to SQS for a legacy worker. All valid. Anti-pattern: SQS consumer runs the entire saga inline and also starts a duplicate state machine for the same orderId. Pick one orchestrator per saga.

Lab DynamoDB table sketch

Tables (lab names): tlp_seat_holds, tlp_payments, tlp_enrollments with orderId as idempotency key or condition attribute. Each Task writes its own table. Compensation deletes or marks RELEASED / REFUNDED. Do not invent exotic transaction APIs—use conditional puts/updates documented for DynamoDB, and keep the saga’s consistency story at the business level.

Operator runbook snippet

On FAILED payment decline: verify ReleaseSeat; no enrollment; notify support template “payment declined.” On FAILED after charge: verify Refund and Release; escalate if refund marker missing. On SUCCEEDED but learner cannot access: check notify path and board entitlement read model lag—not necessarily a workflow bug. On stuck RUNNING: check Task timeout and Lambda concurrency throttles.

Performance budget for enrollment UX

Accept API: tens to low hundreds of milliseconds to persist+publish/start. Saga: seconds to tens of seconds depending on payment provider—not user-blocking. Board entitlement: visible when ActivateEnrollment completes, via push event or short poll. If product insists on synchronous “you are enrolled” in one HTTP response, you accept API Gateway timeout risk—document the exception and keep the saga short.

Anti-pattern: infinite Retry on States.ALL

A Retry on States.ALL with huge MaxAttempts turns permanent bugs into long, expensive executions and delayed compensations. Prefer narrow ErrorEquals lists. Let Catch handle the rest quickly. Review Retry blocks in PR the same way you review catch-all exception handlers in application code.

Cross-link recap for learners

• Events and queues: EventBridge, SNS and SQS
• Functions: AWS Lambda and Serverless Fundamentals
• Edge: API Gateway and Serverless APIs
• Secrets: Secrets Manager and Parameter Store
• Capstone shape: Project 3
• Next module: DevOps Principles and AWS Delivery Architecture

What “exactly-once” does not mean for your ledger

Even with Standard workflow execution semantics as AWS describes them, your payment provider might see two API calls if you mishandle timeouts (request succeeded, response lost, Retry fires). Idempotency keys to the provider and in your DynamoDB markers remain mandatory. Step Functions reduces duplicate workflow starts and clarifies control flow; it does not erase distributed-systems physics at the HTTP edge of a processor.

Enrollment week game day

Before a major launch, run a game day: inject charge faults, fill a DLQ on notify, throttle the charge Lambda, and practice reading execution history under time pressure. Step Functions value shows up in drills, not only in architecture diagrams. Record gaps in runbooks as Module 3 and Module 4 work items.

Lesson Summary

Step Functions turns TechLearningPro’s order and enrollment saga into an operable graph: Tasks for work, Choice for invariants, Wait for time, Parallel and Map for structured concurrency, Succeed/Fail for terminals, Retry/Catch for transient versus permanent failure, and compensations for partial success. Prefer Standard workflows for fulfilment history and saga clarity; use Express only with eyes open on duration and at-least-once awareness. Trigger from EventBridge; fan out notifications with SNS/SQS; keep tasks idempotent and secret-free in state.

Module 2’s application platform stack—edge, compute, containers, secrets, events, workflows—is now in place for serverless orders. Next: DevOps Principles and AWS Delivery Architecture begins Module 3 so you can ship these definitions safely. Project 3 will demand you combine this lesson with lesson 22 under production pressure—practice compensation and cleanup now.

Further Reading

• AWS Step Functions Developer Guide
• Amazon States Language
• Standard vs Express workflows
• Error handling in Step Functions
• Retrying after errors
• Service integrations
• Starting a state machine execution from EventBridge
• Lambda invoke from Step Functions
• Step Functions quotas
• Well-Architected serverless lens (orchestration context)