Secrets Manager and Systems Manager Parameter Store

AWS Cloud & DevOps Engineering · Lesson 21 · AWS Application Platform

Lesson Overview

The progress API is only as trustworthy as the credentials it uses to reach the database, the signing keys that protect learner sessions, and the third-party tokens that upload content artifacts. When those values live in git, in AMI user data, in everlasting container environment variables, or in a shared slide deck titled prod-passwords, Monday’s incident is not a compute failure—it is a secret sprawl failure. Fleet operations from AWS Systems Manager and Fleet Operations taught you to reach private hosts without inbound SSH. This lesson teaches you where configuration and secrets live so those hosts—and Lambda, ECS, and EKS consumers—never need a human to paste a password into a shell.

You will compare AWS Secrets Manager and Systems Manager Parameter Store: when each fits, how SecureString and hierarchical parameters work, what automatic rotation means for RDS-style secrets at a conceptual level, how version stages such as AWSCURRENT and AWSPENDING shape rotation-safe reads, and how KMS encryption awareness (customer managed versus AWS managed keys) influences control and audit. You will design application retrieval with least-privilege GetSecretValue / GetParameter (and batch variants where appropriate), caching that survives rotation, and a dual-write mental model during rotation windows. You will refuse plaintext secrets in git, user data, and AMIs as a matter of engineering identity—not as a slogan.

TechLearningPro’s shape: enrollment services, progress API, content workers, and build workers each retrieve only the secrets they need; rotation does not require redeploying every AMI; ECS tasks and Lambda functions use IAM roles, not baked env files; budget-conscious labs create secrets and parameters you will delete; cleanup is part of the grade.

Learning Objectives

By the end of this lesson you will be able to:

1. Decide when Parameter Store is sufficient and when Secrets Manager is the better home for a credential lifecycle.
2. Design hierarchical Parameter Store paths for TechLearningPro environments and services.
3. Explain SecureString parameters and why plaintext String parameters are wrong for passwords and API keys.
4. Distinguish standard versus advanced parameter concepts at a level sufficient for capacity and policy planning (verify current limits and pricing dimensions in docs).
5. Describe Secrets Manager secret versioning and staging labels such as AWSCURRENT and AWSPENDING during rotation.
6. Explain automatic rotation patterns conceptually, including Lambda-based rotation for RDS-style secrets, without inventing proprietary API shapes.
7. Apply KMS awareness: encryption at rest for secrets and SecureStrings, and when customer managed keys add control versus AWS managed keys.
8. Grant least-privilege IAM for applications to call GetSecretValue / GetParameter on specific resources or path prefixes.
9. Implement rotation-safe application retrieval with caching, refresh, and dual-write window reasoning.
10. Eliminate plaintext secrets from git, user data, AMIs, and long-lived process environments as standing practice.
11. Map secret consumers across EC2, Lambda, ECS/Fargate, and EKS patterns at Module 2 depth.
12. Troubleshoot AccessDenied, decryption failures, stale cache after rotation, and mis-staged versions.
13. Run a sandbox lab that integrates an app-style client with managed secrets and documents rotation-safe access.
14. Preview how event-driven services (EventBridge, SNS and SQS) will consume credentials the same IAM-first way.
15. Know that deeper KMS key policy design continues in the later KMS lesson while this lesson stays application-secret focused.

Prerequisites

• AWS Systems Manager and Fleet Operations for Parameter Store as ops configuration and for the habit of not echoing secrets through Run Command.
• AWS Identity and Access Management for role trust and least privilege.
• Compute Services: EC2, AMIs and Auto Scaling for why secrets must not live in user data or AMIs.
• Container and compute teasers from Module 2 context: Lambda, ECS, and EKS as secret consumers (read sibling lessons as you reach them; patterns here still apply).
• AWS CLI, SDKs and Infrastructure Access for SDK retrieval patterns.
• Sandbox account with budget alert; secrets and advanced parameters can incur charges—verify current pricing dimensions; delete lab secrets and parameters when finished.
• Willingness to rotate any credential that ever appeared in chat logs or screenshots during the lab.

Detailed Core Concepts

The problem: secret sprawl versus managed lifecycle

A secret is any value that grants access or signing power: database passwords, API tokens, private keys, OAuth client secrets, webhook HMAC keys. Configuration is everything else: log levels, feature flags, non-sensitive endpoints, page sizes. Teams blur them under “config,” then store both in the same .env committed to git. Managed stores exist so the lifecycle—create, distribute, rotate, revoke, audit—is intentional.

Parameter Store excels at hierarchical configuration, including SecureString values encrypted with KMS, with tight IAM path prefixes. It is often enough for application config and for many secrets that you rotate with your own process or that change rarely.

Secrets Manager adds first-class secret lifecycle features: built-in rotation integration patterns, staging labels for versions, richer secret-centric audit and replication features (verify current cross-Region replication capabilities in docs when you need DR), and pricing/operational characteristics aimed at secret materials. Prefer it when automatic rotation, multi-version staging, or secret-centric operations are primary requirements—especially database master/user credentials following AWS rotation recipes.

Neither service excuses logging the retrieved value. Retrieval success plus application log of the password is still a breach pattern.

When which: a practical decision guide

Use Parameter Store when:

• You need hierarchical config (/tlp/prod/progress-api/...) with path-based IAM.
• Values are non-secret or Soft-sensitive SecureStrings you rotate infrequently by pipeline.
• You want a single config tree that bootstrapping, Run Command, and apps can share carefully.
• Cost and simplicity dominate and rotation automation is not the headline requirement.

Use Secrets Manager when:

• You need managed rotation (for example, RDS-attached secret rotation patterns with a rotation Lambda).
• You need version stages (AWSCURRENT / AWSPENDING) during dual-credential windows.
• The artifact is clearly a credential with an owner, rotation period, and incident revoke story.
• Multiple services must fetch a rotated database secret without redeploying AMIs.

Use both deliberately: Parameter Store for config hierarchy; Secrets Manager for database and third-party credentials. Do not duplicate the same password in both “for convenience.”

Avoid: plaintext in git; SSM String parameters for passwords; Secrets Manager for every boolean feature flag (noise and cost without benefit).

Parameter Store hierarchies

Paths are design. TechLearningPro convention:

textCopy
/tlp/{env}/{service}/{property}
/tlp/staging/progress-api/log-level
/tlp/staging/progress-api/db-pool-size
/tlp/prod/content-worker/max-concurrency

IAM policies can allow ssm:GetParametersByPath on /tlp/staging/progress-api/* without granting production. Environments become prefixes, not informal filenames on laptops. Renaming environments later is painful—choose stable env names early (staging, prod, sandbox).

SecureString versus String

String parameters store plaintext values from the service’s perspective of parameter typing—appropriate for non-secrets. SecureString encrypts the parameter value with KMS. Always use SecureString for credentials if the credential lives in Parameter Store at all. Remember: encryption at rest does not stop a principal with GetParameter (and decrypt permission via KMS) from reading the secret. IAM remains the gate.

Standard versus advanced parameters (concepts)

Parameter Store offers standard and advanced parameter tiers with different size limits, throughput, and policy features. Advanced parameters unlock larger values and additional capabilities (such as parameter policies in current product form—verify specifics). Choose advanced when you need those capabilities; do not upgrade everything “just in case.” Confirm current limits, throughput quotas, and pricing dimensions in AWS documentation at implementation time—do not memorize numbers from older posts.

Secrets Manager mental model

A secret stores encrypted secret material (often JSON for database username/password/host) with metadata, rotation settings, and versions. Applications typically call GetSecretValue for the current stage. Rotation creates a new version, moves staging labels through a carefully ordered process, and only promotes AWSCURRENT when the new credential works. That ordering is why naive “cache forever” clients break after rotation, and why dual-write/dual-accept windows matter for databases.

Version stages: AWSCURRENT and AWSPENDING

During rotation, multiple versions can exist:

• AWSCURRENT — the version applications should use under normal operation.
• AWSPENDING — the candidate version being validated during rotation before promotion.

Exact rotation step labels and Lambda handler contracts are defined in current Secrets Manager rotation documentation—follow those recipes rather than inventing step names. The engineering takeaway: clients should request the current version (default behavior in typical SDK usage) and be prepared to refresh when authentication fails after a rotation event. During pending periods, database-side users may briefly need both old and new passwords to work, depending on the rotation strategy (single-user versus alternating users). Know the strategy your secret uses.

Automatic rotation patterns (conceptual)

For RDS-style secrets, AWS provides rotation patterns where a Lambda function performs rotation steps: create pending secret values, set database password, test, finish by labeling current. Your job as designer:

• Ensure the rotation Lambda’s IAM can talk to the secret and to the database admin path required.
• Ensure network path from Lambda to database (VPC configuration) if the database is private.
• Ensure applications retry with fresh secret fetches on auth failure.
• Schedule rotation intervals that match policy without rotating so often that operational noise hides real failures.
• Test rotation in staging before production; watch for connection pool clients that never refresh.

Do not hand-roll a cron that only changes the password in Secrets Manager without updating the database—or only updates the database without updating the secret. Split-brain credentials are outages.

KMS encryption awareness

Secrets Manager encrypts secrets with KMS. SecureString parameters likewise use KMS. AWS managed keys reduce setup friction. Customer managed keys (CMKs) give key policy control, clearer separation across accounts, and richer governance for regulated environments. This lesson does not invent key API call sequences; it requires awareness: if you use a CMK, the caller needs both secrets/parameters permission and KMS decrypt permission per key policy. Later AWS KMS and encryption architecture deepens key policies, grants, and multi-account patterns. Here: never disable encryption “to make the lab easier,” and never grant kms:* on all keys to an application role.

Application retrieval patterns

Preferred pattern:

1. Instance profile / task role / Lambda execution role granted least privilege to specific secret ARNs or parameter ARNs/paths.
2. Application retrieves at startup and periodically, or on auth failure.
3. In-memory cache with TTL and forced refresh path.
4. No secret written to disk, logs, or crash dumps.
5. No secret committed to container images.

Anti-patterns: long-lived env vars injected once at deploy from a CI plaintext variable that never rotates; baking secrets into task definition JSON in git; sharing one secret for all microservices.

ECS, EKS, and Lambda each have documentation for referencing Secrets Manager or Parameter Store from task definitions, operators, or environment mapping features. Prefer native integrations that pull at task start over custom bootstrap when they meet your rotation refresh needs—and still plan for mid-lifetime rotation if processes are long-lived.

Caching and the dual-write window

Rotation-safe mental model:

• Before rotation: clients use password A (AWSCURRENT).
• During rotation: systems may create password B (AWSPENDING); database may accept A and B depending on strategy; some clients still hold A in cache.
• After promotion: AWSCURRENT points at B; clients with stale A fail until refresh.

Design clients to: catch authentication failures, refresh secret from Secrets Manager, rebuild pools, retry with bounded backoff. Design pools not to pin a password for days without refresh. For Parameter Store SecureStrings you rotate manually, use the same refresh discipline.

Dual-write for application-level secrets (HMAC keys, for example) can mean accepting signatures from both previous and current keys during a TTL window. Document the window; expire the old key deliberately.

Consumers: EC2, Lambda, ECS, EKS

• EC2: retrieve via SDK using instance profile; never user data plaintext.
• Lambda: execution role gets GetSecretValue; consider extension or SDK caching patterns appropriate to invocation volume (verify current Lambda extensions guidance).
• ECS/Fargate: task execution role may be needed to inject secrets at start; task role for runtime fetch—know which role your pattern uses; do not confuse them.
• EKS: IRSA (IAM Roles for Service Accounts) to fetch secrets; prefer external secrets operators or CSI patterns your platform standardizes—without putting Kubernetes Secrets unencrypted in etcd as the long-term source of truth if policy forbids it.

Audit and least privilege

CloudTrail records API calls to Secrets Manager and Parameter Store. Restrict who may write secrets separately from who may read. Separate human admin roles from application roles. Use resource policies where appropriate for cross-account access patterns (verify current resource policy capabilities). Tag secrets with Owner, Env, and Service for inventory.

Cost awareness without invented prices

Secrets Manager prices per secret per month and per API call dimensions historically; Parameter Store has standard versus advanced pricing and API throughput considerations. Check current pricing. Labs: create few secrets; delete them; avoid thousands of empty “test” secrets. Advanced parameters and CMKs add dimensions—budget them consciously.

Architecture and Design Diagrams

Config versus secrets split for TechLearningPro:

mermaidCopy
flowchart TB
  App["Progress API process"]
  PS["Parameter Store\n/tlp/env/service/* config"]
  SM["Secrets Manager\nDB credential secret"]
  KMS["KMS encryption at rest"]
  DB["RDS / Aurora"]
  App -->|"GetParameter(s)"| PS
  App -->|"GetSecretValue"| SM
  PS --> KMS
  SM --> KMS
  SM -->|"rotated password"| DB
  App -->|"connect with refreshed secret"| DB

Rotation-safe client loop:

mermaidCopy
sequenceDiagram
  participant App
  participant Cache
  participant SM as Secrets Manager
  participant DB as Database
  App->>Cache: get password
  Cache-->>App: password A
  App->>DB: connect A
  Note over SM,DB: Rotation promotes B to AWSCURRENT
  App->>DB: connect A fails
  App->>SM: GetSecretValue AWSCURRENT
  SM-->>App: password B
  App->>Cache: store B
  App->>DB: connect B succeeds

Hierarchical IAM boundary:

textCopy
Human admin role → PutSecretValue / PutParameter (prod) via change control
App role progress-api → GetSecretValue on arn:...:secret:tlp/prod/progress-api/db-*
App role content-worker → GetParametersByPath /tlp/prod/content-worker/*
No role → Get* on another service’s prefix

Technical Explanation

IAM actions that matter

Applications typically need secretsmanager:GetSecretValue on specific secret ARNs (with wildcards only when naming conventions are strict) and kms:Decrypt when key policy requires it. For Parameter Store: ssm:GetParameter, ssm:GetParameters, ssm:GetParametersByPath, plus KMS decrypt for SecureStrings. Writers need put/delete actions separated from readers. Avoid Resource: "*" on GetSecretValue in production teaching accounts—it trains a bad default.

Secret naming and JSON shape

Database secrets often use a JSON structure with keys such as username, password, host, port, engine—aligned with rotation Lambdas that expect a schema. Keep application parsers strict; do not log the JSON. Name secrets with environment and service: tlp/prod/progress-api/db. Random suffixes may appear in ARNs—use ARN patterns carefully in IAM.

Partial ARNs and IAM gotchas

Secrets Manager ARNs include a hyphenated random suffix. IAM policies often allow a prefix with wildcard. Mis-copying the ARN causes AccessDenied that looks like KMS failure. Train operators to read the error code: AccessDenied versus ciphertext/KMS errors versus ResourceNotFoundException.

SDK retrieval and binary versus string

GetSecretValue may return string or binary secret material depending on what you stored. Prefer string JSON for DB credentials unless you have a binary key use case. Validate content-type assumptions in tests.

Parameter policies and tiers

Advanced parameters can carry policies such as expiration or notification-related behaviors in current Parameter Store features—verify before depending on them. Expiration for temporary credentials in Parameter Store is not a substitute for Secrets Manager rotation of database passwords, but it can help with temporary integration tokens if your design fits documented policies.

Replication and multi-Region awareness

Secrets Manager supports replication patterns for multi-Region DR designs in current product form. If TechLearningPro runs warm standby in a second Region, decide whether secrets replicate and how applications select Region. Do not assume replication is on by default. Parameter Store is Regional; design promotion of config across Regions deliberately (pipeline writes both, or read from AppConfig-style systems later—stay honest about complexity).

Rotation Lambda networking

Private RDS requires the rotation Lambda to reach the database. That usually means Lambda in VPC subnets with security group access to the DB SG, plus endpoints or NAT for Secrets Manager and KMS API calls. Missing any piece produces rotation failures that leave secrets stuck mid-stage. Staging rehearsals catch this; production first-attempts create outages.

Alternating users versus single-user rotation

Rotation strategies may use one database user or alternate between two users so that changing a password does not break all live connections simultaneously. Choose the strategy based on current AWS rotation templates and your connection behavior. Document which you use in the ADR. Interview answers should admit that the strategy matters, not merely that “rotation is enabled.”

Application config loaders

A good loader: resolve environment name; fetch parameters by path; fetch secrets by ARN list; merge; fail fast if required keys missing; never print values. A bad loader: silently defaults production DB host to localhost; catches all exceptions and starts anyway.

ECS execution role versus task role

If the platform injects secrets as environment variables at task start using the execution role, the process still holds them in env for its lifetime—rotation mid-task will not appear until restart unless the app also fetches at runtime. Prefer runtime fetch for long-lived tasks when rotation intervals are shorter than task lifetimes, or accept that task recycle is part of rotation completion.

Kubernetes caution

Syncing Secrets Manager into Kubernetes Secret objects can reintroduce etcd exposure if not carefully controlled with encryption and RBAC. Platform teams should pick a supported pattern and make it the only pattern. Developers should not each invent a sidecar that prints secrets to logs.

Testing without secret leakage

Integration tests should use sandbox secrets. CI should assume roles; CI logs must be scrubbed. Never paste GetSecretValue CLI output into tickets. For lessons, redacted evidence means “command succeeded, exit 0,” not the secret body.

CLI disclaimer

CLI flags and SDK method names evolve. Verify aws secretsmanager ... and aws ssm ... syntax against current AWS CLI documentation before scripting. Examples in labs are illustrative of workflow, not eternal flag contracts.

Practical Implementation or Design Walkthrough

Scenario

Wire a staging progress API style client to: Parameter Store for log-level and pool size; Secrets Manager for a simulated DB credential JSON; IAM least privilege; cache with refresh-on-auth-failure; document how rotation would proceed. Optional: enable rotation on a supported engine in sandbox only if your account and budget allow—and clean it up.

Walkthrough steps

1. Budget. Confirm alert. Tag resources Project=tlp-lesson21.

2. Create parameters.

   • String /tlp/staging/progress-api/log-level = INFO
   • String /tlp/staging/progress-api/db-pool-size = 5
     Optionally one SecureString for a dummy API token to practice decrypt permissions.

3. Create a secret. Store JSON shaped like {"username":"appuser","password":"...","host":"db.internal","port":5432}. Use a generated password; do not reuse any real credential.

4. IAM role. Create an application role (or instance profile) allowing get on those specific parameters/paths and that secret ARN pattern; allow required kms:Decrypt if using CMK; deny broad wildcards.

5. Client sketch. Implement or outline a small program/SDK snippet that: loads parameters by path; loads secret; caches; on simulated auth failure refreshes secret and retries once.

6. Negative tests. Prove AccessDenied with a role missing KMS or wrong ARN; prove missing parameter fails fast.

7. Rotation narrative. Write the dual-write steps for your chosen strategy; if you enable automatic rotation in sandbox, run a rotation and show the app recovering after refresh (or document blockers honestly).

8. Consumer matrix. One paragraph each: how EC2, Lambda, and ECS would attach the same IAM permissions.

9. Cleanup. Delete secret (schedule deletion per current waiting period rules—understand recovery window), delete parameters, detach roles if lab-only, note pending secret deletion charges/state until fully removed.

Illustrative application pseudocode

textCopy
config = GetParametersByPath("/tlp/staging/progress-api/")
secret = GetSecretValue("tlp/staging/progress-api/db")  # current stage
pool = connect(secret, pool_size=config["db-pool-size"])

on AuthFailure:
  secret = GetSecretValue(...)  # force refresh, bypass cache
  pool.rebuild(secret)
  retry_once()

Verify real SDK calls and exception types in the language you use. Do not log secret.

Break-glass and revoke

Document how to revoke a leaked database password: rotate immediately, invalidate sessions if tied to the same secret material, review CloudTrail for GetSecretValue from unexpected principals, rotate again if the leak source remains. Parameter Store SecureString “rotation” may be a pipeline put of a new value plus coordinated deploy/refresh.

Enterprise Real-World Scenarios

Scenario 1: The AMI that knew too much

A baker AMI for TechLearningPro baked a staging DB password into an application properties file “only for smoke tests.” The AMI was shared to another account. Weeks later staging was scraped. Fix: purge AMI and snapshots; rotate credentials; move retrieval to Secrets Manager; add a pipeline check that fails images containing secret-like patterns.

Scenario 2: Rotation success, application failure

Automatic rotation succeeded in Secrets Manager and on RDS. The progress API connection pool cached the old password for hours. Learners saw intermittent 500s. Fix: refresh-on-failure; shorter cache TTL; recycle tasks after rotation for pools that cannot refresh; alarm on auth failure spikes correlated with rotation events.

Scenario 3: One secret to rule all services

A single tlp/prod/shared-db secret was readable by every microservice role. A low-privilege content worker compromise dumped the password. Fix: per-service database users and per-secret IAM; network SG still required; principle of least privilege applied to secret ARNs.

Scenario 4: Parameter Store String for API keys

Teams stored third-party API keys as String parameters because SecureString “needed KMS knowledge.” A principal with broad ssm:GetParameter* exfiltrated them without triggering KMS-focused detections the security team watched. Fix: SecureString or Secrets Manager; reduce broad SSM read; detect unusual Get* volume.

Scenario 5: Lambda out of VPC cannot rotate

Rotation Lambda lived outside the VPC while RDS was private. Rotations failed; labels stuck; engineers disabled rotation. Fix: place Lambda correctly with endpoints for Secrets Manager/KMS and SG to DB; test in staging monthly.

Scenario 6: GitOps env sprawl

Helm values in git contained base64 “encrypted” secrets that were merely obfuscated. A public fork leaked them. Fix: externalize to Secrets Manager; seal only with real encryption systems your platform supports; rotate everything that lived in git history.

Best Practices

1. Split config (Parameter Store) from credential lifecycle (Secrets Manager) on purpose.
2. Use hierarchical paths and matching IAM prefixes per environment and service.
3. Prefer SecureString or Secrets Manager for any credential-shaped value—never plaintext String for passwords.
4. Grant Get* on specific ARNs/paths; separate writers from readers.
5. Encrypt with KMS; use CMK when key policy and audit demand it; grant decrypt narrowly.
6. Design clients for refresh after rotation; never cache forever.
7. Keep secrets out of git, AMIs, user data, and screenshot tickets.
8. Rehearse rotation in staging; monitor auth failures around rotation windows.
9. Prefer per-service credentials over shared super-secrets.
10. Tag secrets and parameters; inventory orphaned lab secrets; delete them.
11. Align ECS/Lambda/EKS roles carefully (execution versus task versus IRSA).
12. Redact logs; add tests that fail if loaders print secret material.
13. Document dual-accept windows for signing keys and DB strategies.
14. Verify CLI/SDK details against current docs at implementation time.
15. Plan cleanup including secret deletion recovery windows.

Common Mistakes and Anti-Patterns

1. Secrets in git or user data. Symptom: leaks via history or describe-instance. Fix: managed store + rotate + rewrite history policy for git if needed.

2. String parameters for passwords. Symptom: broad SSM read exfiltration. Fix: SecureString/Secrets Manager.

3. Cache forever. Symptom: outage after successful rotation. Fix: TTL + refresh-on-auth-failure.

4. Resource "*" on GetSecretValue. Symptom: lateral secret read after one role compromise. Fix: ARN-scoped policies.

5. Disabling rotation after one failure. Symptom: permanent static prod passwords. Fix: repair Lambda networking/IAM; keep rotation.

6. Duplicating the same password in Parameter Store and Secrets Manager. Symptom: rotated one side only. Fix: single source of truth.

7. Logging GetSecretValue responses in debug mode. Symptom: CloudWatch becomes a secret store. Fix: structured logging ban list; review logger config in prod.

8. Confusing execution role and task role in ECS. Symptom: “works in console test, fails in task” or over-privileged execution role. Fix: map which step needs which role per current ECS docs.

9. Using Secrets Manager for every non-secret flag. Symptom: cost and operational noise. Fix: Parameter Store Strings for non-secrets.

10. Forgetting KMS in CMK designs. Symptom: AccessDenied that confuses operators. Fix: include kms:Decrypt in app role and key policy.

Troubleshooting and Design Review

Troubleshooting map

Design review questions

• Which values are secrets versus config, and where does each live?
• Who can write production secrets? Is that change-controlled?
• Can each app role read only its own ARNs/paths?
• What happens sixty seconds after rotation?
• Are AMIs and task definitions free of plaintext secrets?
• Is CMK required by policy, and are decrypt grants correct?
• Are lab secrets scheduled for deletion?
• How will EventBridge-triggered workers in the next lessons obtain credentials?

Operator runbook snippet

Suspected leak: rotate now; review CloudTrail GetSecretValue; tighten IAM; recycle all tasks/instances that might have cached values; notify owners. Rotation failed: do not manually “fix” by putting a password only in the database; inspect rotation Lambda; finish or safely roll back per AWS rotation guidance. App cannot decrypt: check CMK policy before widening secrets IAM to *.

Interview Questions and Answers

Q1. When do you choose Secrets Manager over Parameter Store?

When you need managed secret lifecycle features such as automatic rotation and version staging for credentials—especially database secrets—whereas Parameter Store fits hierarchical config and simpler SecureString use cases.

Q2. Why are hierarchical parameter paths valuable?

They enable environment/service namespacing and path-based IAM so staging roles cannot read production prefixes.

Q3. What does SecureString provide that String does not?

KMS encryption at rest for the parameter value. Access still requires IAM (and KMS decrypt as applicable); it is not a substitute for least privilege.

Q4. What are AWSCURRENT and AWSPENDING?

Staging labels on secret versions. Applications normally use AWSCURRENT; AWSPENDING marks a version under rotation before promotion.

Q5. How should applications behave across rotation?

Cache briefly, refresh on authentication failure (and/or TTL), rebuild connection pools, and retry with bounded backoff—assuming the rotation strategy may briefly require dual acceptance on the data store side.

Q6. Why is putting secrets in user data or AMIs dangerous?

They persist in launch templates, describe APIs for privileged callers, image snapshots, and clones—expanding leak surface and resisting rotation.

Q7. How do customer managed keys change the design?

They add key policy control and governance; application roles must be allowed to decrypt with that key in addition to secret/parameter get permissions.

Q8. What IAM mistake creates broad secret exfiltration risk?

Granting GetSecretValue or GetParameter* on * to many roles, or storing secrets as plaintext Strings readable by broad SSM policies.

Q9. How do ECS task roles relate to secret retrieval?

Runtime SDK fetches typically use the task role; some injection-at-start patterns use the execution role—know which your platform uses and least-privilege both.

Q10. What is a dual-write or dual-accept window?

A period during key or password rotation when both old and new credentials may be valid so clients can refresh without a hard cut-outage.

Q11. How does this lesson connect to SSM fleet operations?

Fleet ops may consume parameters for config and must not echo secrets through Run Command; secret material belongs in managed stores with audited Get*.

Q12. What comes next for event-driven platforms?

Workers driven by EventBridge, SNS, and SQS still need IAM-based secret retrieval and rotation-safe clients—messaging does not justify plaintext env secrets.

Hands-On Exercises

Exercise: Managed secrets + rotation-safe access

Problem statement. In a sandbox, create Parameter Store config entries and a Secrets Manager secret for a fake DB credential JSON. Attach a least-privilege role to a runtime (EC2 with SSM Session Manager from lesson 20, or Lambda, or local assumed-role—pick one). Prove successful retrieval, AccessDenied negative paths, and a documented refresh-on-failure strategy. Budget first; delete parameters and schedule secret deletion.

Tasks.

1. Create tagged parameters (String config) and one Secrets Manager secret (JSON).
2. Create an app role with scoped Get* (+ KMS if needed).
3. Run a client/SDK script that loads config + secret without printing secret values.
4. Demonstrate AccessDenied with a deliberately wrong resource policy or role.
5. Simulate stale cache: call with old password mock, refresh from Secrets Manager, succeed.
6. Write an ADR paragraph choosing Parameter Store versus Secrets Manager for (a) log level (b) DB password (c) third-party API token.
7. Cleanup: delete parameters; schedule secret deletion; remove lab roles if dedicated; evidence tagged resources handling.

Evaluation rubric.

Challenge

Extend the lab for a production constraint: multi-service isolation (progress API versus content worker secrets), CMK encryption awareness, and a written dual-accept window for an HMAC signing key rotation that EventBridge workers will later use. Deliver a short ADR.

Challenge ADR shape

Decision: TechLearningPro stores non-secret config in Parameter Store under /tlp/{env}/{service}/*; stores DB and third-party credentials in Secrets Manager with rotation where supported; applications use scoped IAM Get*; clients refresh on auth failure; CMK used for production secrets; no secrets in git/AMI/user data; ECS/Lambda/EKS follow runtime fetch patterns compatible with rotation.

Consequences: More IAM and KMS design work; fewer leak blast radii; need staging rotation rehearsals; secret deletion and orphan inventory become operational tasks; messaging workers in subsequent lessons inherit the same rules.

Lab evidence checklist

Parameter names (not secret values), secret ARN (redacted suffix ok), role names, policy JSON redacted of sensitive sid noise if needed, CloudTrail event IDs for Get* and AccessDenied, cleanup timestamps, budget alert id.

Deep dive: naming conventions that survive multi-account

Secret and parameter names are APIs used by IAM policies. If production uses prod/progress-api/db in one account and tlp-prod-progress-db in another, every role template forks. Standardize early:

textCopy
Secrets Manager: tlp/{env}/{service}/{purpose}
Parameter Store: /tlp/{env}/{service}/{property}

Include purpose (db, redis-auth, stripe-webhook, jwt-hmac) so humans and policies stay precise. Avoid embedding rotation intervals or ticket numbers in names; metadata and tags carry that. When you promote a service from staging to production, you create new secrets in the production account or path—you do not “copy paste password once.” Pipelines should generate values and store them; humans should rarely view them.

Multi-account designs often keep secrets in a security or shared-services account and grant resource policies to application accounts. That pattern needs careful ARN and KMS key policy work—preview only here; implement with current AWS cross-account secrets guidance when your organization is ready. Until then, per-account secrets with identical naming still beat random names.

Deep dive: connection pools and secret refresh

Most progress API outages after “successful” rotation are pool problems. A pool created at process start holds TCP connections authenticated with password A. Secrets Manager now returns password B. Idle pooled connections may still work until the database severs them; new connections using cached password A fail. Symptoms look like intermittent errors, not a clean break—exactly the worst on-call puzzle.

Mitigations that work in combination:

• On auth failure, invalidate the pool, refresh the secret, rebuild.
• Periodically refresh the secret in the background before rotation is due if you know the schedule.
• Recycle ECS tasks or ASG instances as a coarse but reliable completion step after rotation in staging rehearsals until fine-grained refresh is proven.
• Prefer rotation strategies (such as alternating users) that reduce single-password cutovers when your database and AWS template support them—verify current template behavior rather than assuming.

Do not swallow auth errors as generic 500s without metrics. An db_auth_failure counter correlated with Secrets Manager rotation CloudTrail events shortens incidents.

Deep dive: feature flags versus secrets versus config

TechLearningPro will be tempted to put everything in Secrets Manager because it feels “more secure.” Feature flags that are not capabilities tokens belong in Parameter Store Strings or a dedicated feature-flag system. Database passwords belong in Secrets Manager. Signing keys belong in Secrets Manager (or KMS/HSM patterns later) with dual-accept windows. Third-party API tokens usually belong in Secrets Manager with manual or Lambda-assisted rotation depending on vendor APIs—many vendors have no AWS rotation template, so your pipeline must rotate and update the secret atomically with the vendor console or API.

Write a one-page classification guide for the platform team. Without it, every engineer invents a new home for the next token.

Deep dive: CI/CD injection without secret fossilization

Pipelines need to pass configuration into deploys without fossilizing secrets into artifact stores. Patterns:

• Deploy role assumes permission to write secret references into task definitions (ARN pointers), not secret values into git.
• Build jobs that need a secret to run integration tests assume a role and fetch at test time; they never echo values.
• AMI builds pull temporary tokens at bake time only if unavoidable, then discard—prefer baking clients that fetch at boot.
• Pull request environments get short-lived staging secrets, never production ARNs in the PR bot role.

If your CI system offers “secret variables,” treat them as break-glass for tooling, not as the system of record for application DB passwords. The system of record should remain Secrets Manager or Parameter Store so runtime and humans share one lifecycle.

Deep dive: AppConfig and Parameter Store boundary (awareness)

AWS AppConfig exists for validated configuration deployments with rollbacks and growth strategies. This lesson does not replace AppConfig teaching; it notes the boundary. Parameter Store is a hierarchical key-value configuration store. AppConfig is a deployment system for configuration. Secrets Manager is a secret lifecycle system. Using Secrets Manager as a feature-flag deploy mechanism is usually the wrong abstraction. Using AppConfig to ship database passwords is also wrong. Keep the boundaries so each control plane stays understandable.

Deep dive: evidence packs without leaking values

Enterprise homework and incident reviews need evidence. Acceptable evidence includes: secret ARN, parameter names, IAM policy documents, CloudTrail event IDs, screenshots of console pages with values masked, exit codes from CLI get commands redirected to /dev/null, and architecture diagrams. Unacceptable evidence includes: plaintext passwords, full SecretString JSON, and .env files. Teach reviewers to reject leaky submissions the same way they reject open SSH screenshots in lesson 20 labs.

Narrative: enrollment spike plus rotation night

Marketing schedules an enrollment email at 09:00. Platform schedules DB secret rotation at 09:05 because “off peak” was misread from another Region’s clock. Rotation succeeds. Half of progress API tasks refresh; half do not. Error rates climb as new learners arrive. On-call stops the email campaign, forces an ECS service redeploy to recycle tasks, confirms AWSCURRENT, and moves rotation to a maintenance window with product freeze awareness—the same calendar discipline Patch Manager required in lesson 20. The ADR gains an explicit rule: no credential rotation during enrollment launches without dual-accept proof and a recycle plan.

Narrative: the contractor and the broad SSM read

A contractor role inherited AmazonSSMReadOnlyAccess-style breadth “temporarily.” Parameter Store held SecureStrings for several integration tokens. The contractor exported parameters for “documentation.” Months later a token appeared in a public gist. Encryption at rest did not save the day; IAM breadth lost it. Remediation: rotate tokens; replace broad managed read with path-scoped policies; separate human read of secrets behind break-glass with MFA and reason codes. Lesson for TechLearningPro: managed policies that sound read-only can still be too wide for multi-tenant teaching platforms.

ECS, EKS, and Lambda checklist for reviewers

When reviewing a service design that claims “secrets are handled”:

1. Where is the secret ARN defined—git as reference only?
2. Which IAM role can GetSecretValue?
3. Is there a refresh story for process lifetime greater than rotation interval?
4. Are logs scrubbed?
5. Is staging rotation rehearsed?
6. For EKS, is IRSA bound to one service account with narrow ARN access?
7. For Lambda, is VPC/NAT/endpoint design compatible with both DB and Secrets Manager/KMS?
8. For ECS, are execution and task roles distinguished and least-privileged?

Any “no” becomes a ticket before production promotion.

What success looks like for TechLearningPro

A new engineer can deploy a staging service that reads /tlp/staging/{service}/* and a single Secrets Manager DB secret without ever seeing the password. Rotation runs monthly in staging without pages. Production rotation runs in a window with dual-accept confidence. CloudTrail shows only expected principals fetching secrets. Lesson 20 operators can Session Manager into hosts without discovering passwords in files on disk. Lesson 22’s event workers inherit the same retrieval library. That is the bar.

Parameter Store throughput and service control awareness

Under load, configuration fetches at every request are a self-inflicted outage pattern. Cache parameters in process with TTL; use hierarchical fetch at startup; watch API throttling errors. If you need higher throughput features, review current advanced parameter and throughput documentation—do not invent unlimited get rates. Control mapping packages and SCPs in organizations may deny Secrets Manager or KMS in sandbox OUs; know your guardrails before the lab night.

Signing keys and dual-accept in product terms

Learner session tokens signed with HMAC key K1 must continue validating for the TTL of outstanding tokens when you introduce K2. Deploy applications that accept K1 and K2, publish K2 as current for new tokens, wait TTL, then retire K1 from the accept list and from Secrets Manager versions you still expose. This is the dual-accept window in product language. Database passwords follow analogous overlap when alternating users or when the engine accepts both during pending stages. Interview answers that only say “use Secrets Manager” without overlap reasoning are incomplete for senior roles.

Practical SDK hygiene for progress API services

Language teams differ, but hygiene converges. Create a small internal library used by the progress API, content workers, and build workers that: resolves environment from a non-secret env var such as TLP_ENV; builds parameter paths and secret ARNs from that environment; fetches once at startup; exposes refreshSecrets() for auth-failure paths; redacts any accidental stringification; and emits metrics for fetch latency, cache hit, refresh count, and auth-failure-triggered refresh. Services should not each invent YAML parsers for secret JSON.

Timeouts and retries on the Secrets Manager and Parameter Store APIs should be bounded. An outage of the secrets API should not wedge every request thread in unbounded retry. Decide fail-fast versus serve-stale for non-secret config; for credentials, serving forever-stale after a known rotation is worse than failing closed. Document the choice.

Lab budget notes specific to secrets

Creating dozens of secrets while debugging naming costs money and clutters the account. Use one secret and a few parameters per learner lab. Schedule secret deletion as soon as evidence is captured; understand that recovery windows may keep the secret in a pending-deletion state—verify current deletion behavior so you do not think cleanup failed when it is waiting. CMKs created only for the lab should be scheduled for deletion per KMS rules after they are unused; key deletion has waiting periods by design. Prefer AWS managed keys in short sandbox exercises unless the learning objective is CMK policy.

Cross-link recap for the module path

Lesson 20 made private fleets operable. Lesson 21 makes their credentials manageable. Containers and Lambda earlier or later in Module 2 are consumers of the same patterns. EventBridge, SNS, and SQS come next as how asynchronous enrollment and grading events move—workers that subscribe still need this lesson’s retrieval library. KMS deepens later for key policies, envelope encryption, and multi-account key strategy. Do not wait for the KMS lesson to stop putting passwords in git; use managed secrets now with AWS managed keys if needed, then tighten to CMKs when policy demands.

Failure injection exercises worth running in staging

1. Deny GetSecretValue via a temporary SCP or permission boundary on a staging role and confirm the service fails loudly with actionable metrics.
2. Rotate a staging DB secret during a load test and confirm refresh paths keep error rates inside budget.
3. Put an incorrect JSON schema in the secret and confirm the loader fails fast at startup.
4. Grant a second service role access by mistake, then remove it and show CloudTrail evidence of the mistaken Gets.
5. Attempt Run Command to echo a parameter SecureString into output and show why that document should be denied by policy culture even if IAM allows GetParameter on the instance—prefer application retrieval, not shell catapults.

These drills build muscle memory that documentation alone cannot.

Security review questions for pull requests

• Does this PR add a plaintext secret or a reference ARN?
• Did IAM scope shrink or quietly grow to *?
• Are new parameters under the correct /tlp/{env}/{service}/ prefix?
• Is there a rotation owner listed in the service README?
• Do integration tests run with assumed roles rather than long-lived keys on the runner?
• Could logs from the new code path print configuration objects that embed secrets?

PR templates that include these checkboxes reduce lesson-21 failures more than another wiki page.

Lesson Summary

Secrets are not configuration with moodier names—they are credentials with a lifecycle. Parameter Store gives TechLearningPro hierarchical, IAM-scoped configuration and SecureString options; Secrets Manager gives rotation-aware secret versions and staging for credential material. KMS underpins encryption at rest; IAM underpins who may read; application design underpins survival across rotation. The progress API, content workers, and future EventBridge consumers retrieve with least privilege, cache briefly, refresh on failure, and never bake passwords into AMIs, git, or everlasting env files.

You already operate the fleet with AWS Systems Manager and Fleet Operations. Now the fleet can fetch what it needs without storing it forever. Next: EventBridge, SNS and SQS connects enrollment events and asynchronous workers—still IAM-first, still secret-safe. Deeper key policy craft continues later in the KMS lesson.

Further Reading

• AWS Secrets Manager User Guide
• Rotate AWS Secrets Manager secrets
• AWS Secrets Manager rotation function templates
• Get a secret value
• AWS Systems Manager Parameter Store
• Restricting access to Parameter Store parameters
• Managing parameter tiers
• AWS KMS concepts
• IAM: Accessing Secrets Manager
• Amazon ECS: secrets from Secrets Manager / Parameter Store