AWS Systems Manager and Fleet Operations

AWS Cloud & DevOps Engineering · Lesson 20 · AWS Application Platform

Lesson Overview

Monday’s progress board stays green only when the fleet behind it is operable without improvisation. When a content worker needs a package update, when a build worker’s agent drifts, or when an operator must inspect a private progress API host at 05:40, the question is not “can we SSH from the internet?” The question is whether identity, agent health, and a controlled operations plane already exist. Pets with long-lived SSH keys fail that test. Cattle managed through AWS Systems Manager pass it when the design is intentional.

Security Groups, NACLs and VPC Connectivity foreshadowed Session Manager as the preferred path into private subnets: no inbound SSH from 0.0.0.0/0, IAM as the gate, and VPC endpoints so private instances never need a public IP solely for administration. Compute Services: EC2, AMIs and Auto Scaling made the fleet replaceable. This lesson makes the fleet operable: Session Manager for interactive access, Run Command for remote scripts, Patch Manager for baseline-driven updates, Automation documents for repeatable runbooks, and light awareness of Inventory, State Manager, and Parameter Store as configuration—not the secrets deep dive, which belongs to Secrets Manager and Systems Manager Parameter Store.

TechLearningPro’s teaching shape stays fixed: progress API hosts, content workers, and build workers in private subnets; instance profiles that can talk to Systems Manager; Session Manager preferred over bastions; patch and command workflows that leave an audit trail; budget alerts before lab instances; cleanup before the bill becomes the lesson.

Learning Objectives

By the end of this lesson you will be able to:

1. Explain why Session Manager is the default administrative path for TechLearningPro’s private EC2 fleet when agent, IAM, and network reachability are in place.
2. List the prerequisites for SSM-managed instances: SSM Agent, instance profile permissions, and connectivity to Systems Manager endpoints.
3. Design VPC interface endpoints for Systems Manager so private subnets remain private without opening inbound SSH.
4. Start a Session Manager session, describe session logging options, and contrast that model with bastion hosts and key distribution.
5. Use Run Command to execute a controlled script or document against tagged fleet members and interpret command invocation results.
6. Describe Patch Manager baselines and why maintenance windows matter for Monday reliability.
7. Explain Automation documents at a conceptual level as versioned, auditable operational workflows—not ad-hoc console clicking.
8. State what Inventory and State Manager contribute to fleet awareness and desired-state enforcement without memorizing every capability.
9. Use Parameter Store lightly for non-secret or lightly sensitive configuration paths that Run Command and bootstrapping may read (full secrets comparison in lesson 21).
10. Apply least privilege to SSM actions: who may start sessions, who may run commands, which instances are in scope via tags or resource policies.
11. Diagnose common “instance not appearing in Fleet Manager / Session Manager” failures across agent, IAM, and network planes.
12. Run a sandbox lab that proves remote command and patching awareness with budget controls and full cleanup.
13. Decide when a bastion is still justified and how to avoid running bastion and open SSH as false defense in depth.
14. Connect SSM operations to Auto Scaling: new instances must inherit agent, profile, and endpoints—not only the baker host.
15. Preview how secrets and SecureString parameters will separate from fleet ops in the next lesson.

Prerequisites

• Security Groups, NACLs and VPC Connectivity for private subnets, security-group references, and the Session Manager connectivity tease.
• Amazon VPC Architecture for route tables, NAT versus endpoints, and private placement of the progress API.
• AWS Identity and Access Management for instance profiles and operator IAM policies that gate ssm:StartSession and Run Command.
• Compute Services: EC2, AMIs and Auto Scaling for launch templates that must bake or install the SSM Agent and attach the instance profile.
• AWS CLI, SDKs and Infrastructure Access for describing instance information and invoking commands from a controlled identity.
• A sandbox account with a budget alert before launching lab instances or creating interface endpoints (endpoints have hourly and data-processing charges—treat them as billable lab resources and delete when done).
• Comfort with tags: TechLearningPro labs should tag Project=tlp-lesson20 (or equivalent) so cleanup and Run Command targeting stay precise.

Detailed Core Concepts

What Systems Manager is for in this course

AWS Systems Manager (SSM) is the operations plane for managing EC2 instances (and, in broader AWS use, hybrid nodes and other managed compute) without treating SSH keys and bastions as the primary control path. For TechLearningPro, SSM answers: How do operators reach private progress API hosts? How do we run the same diagnostic script across content workers? How do we patch build workers against a baseline without inventing a snowflake process per host? How do we leave an audit trail when someone changes a production configuration file at 02:00?

SSM is not a replacement for CI/CD, not a substitute for Auto Scaling health, and not a place to hide long-lived application secrets (lesson 21). It is the fleet operations toolkit that sits beside IAM, VPC, and EC2. When those three are designed for SSM, Monday operations become a session or a command invocation. When they are not, Monday operations become a scramble for a forgotten key pair and an emergency security-group edit that opens port 22.

Session Manager over bastions

A bastion concentrates administrative ingress: patch it, harden it, distribute keys or certificates, and allow SSH or RDP from somewhere—ideally a corporate VPN CIDR, too often the open internet. It is still a host to operate, still a hop to document, and still a single place where a compromised key becomes lateral movement.

Session Manager starts an interactive shell (or port-forwarding style workflows where configured) through the Systems Manager service. Typical production shape for TechLearningPro:

• No inbound TCP 22 or 3389 from the internet on app and data tiers.
• SSM Agent running on the instance (included on many current Amazon Linux and Windows AMIs—verify your AMI).
• Instance profile with permissions to register with and communicate with Systems Manager (commonly via AWS-managed policies designed for SSM instance roles—prefer least privilege overlays in production rather than forever-admin).
• Network path from the instance to Systems Manager: NAT gateway outbound HTTPS, or preferably VPC interface endpoints for the Systems Manager related services so private subnets need neither public IP nor NAT solely for admin access.
• Operator IAM permissions to ssm:StartSession (and related actions) scoped to tagged instances, plus Session Manager preferences for CloudWatch Logs or S3 session logging where policy requires audit.

Choose a bastion when a third-party tool, regulated workflow, or non-agent environment truly requires SSH. Prefer Session Manager for TechLearningPro’s private fleets when agent and endpoints exist. Running both with open SSH is not layered defense; it is two ways to fail the same review.

The three planes that make an instance “managed”

Operators often say “SSM is broken” when one of three planes failed:

1. Agent plane. The SSM Agent must be installed, running, and able to update or maintain connectivity. Ancient custom AMIs sometimes omit it. User data that disables agents “for hardening” without a replacement path creates silent orphans.
2. Identity plane. The instance profile must allow the agent to call Systems Manager APIs. Missing role, wrong role, or overly denylisted outbound identity errors show up as instances that never appear as managed.
3. Network plane. The agent must reach Systems Manager endpoints. Security groups that block outbound 443 to the endpoints, missing interface endpoints without NAT, or custom NACLs that drop ephemeral return paths all produce “not managed” symptoms. IAM can be perfect and the agent still cannot register.

Diagnose in that order: is the agent alive, does the role exist, can HTTPS to the endpoints succeed? Do not open SSH as the first fix; that trains the wrong muscle.

Run Command for fleet scripts

Run Command executes SSM documents (AWS-provided or custom) on managed instances you select by instance ID, tags, or resource groups. TechLearningPro uses it for: collecting progress API version strings across the ASG; restarting a systemd unit after a config push; running a read-only diagnostic; applying a one-time remediation that is not yet worth a full Automation document.

Design rules:

• Prefer documents over opaque one-liners pasted into the console when the action will repeat.
• Target by tags (Role=progress-api, Env=staging) so Auto Scaling replacements stay in scope without rewriting instance ID lists.
• Capture output and status; failed invocations are first-class signals, not noise.
• Separate who may run commands from who may start sessions. An on-call engineer might need session access; a CI role might need only a narrow Run Command document on staging tags.
• Never use Run Command as a secret distribution channel that echoes passwords into command history and logs. Parameter Store and Secrets Manager exist for a reason (lesson 21).

Patch Manager concepts

Patch Manager helps you define what “patched” means and drive instances toward that state. Core ideas you must own:

• Patch baselines define approved patches (operating system family, classification, severity, auto-approval delays, approved/rejected lists). A baseline for Amazon Linux content workers may differ from a baseline for Windows build helpers if you still run them.
• Patch groups (via tags) associate instances with a baseline strategy so the progress API fleet is not accidentally held to an experimental baseline meant for a lab ASG.
• Scan versus install. Scanning reports compliance without changing the host. Installing applies patches. Monday reliability prefers scheduled install windows, not surprise reboots during peak enrollment traffic.
• Maintenance windows schedule when disruptive work may run. Awareness level in this lesson: patching without a window is how you reboot half the API tier during a live cohort launch. Module-level operations culture expects windows, freeze periods, and rollback notes.

Patch Manager does not replace golden AMI pipelines. Many enterprises patch and bake: AMI pipelines absorb routine updates; Patch Manager covers interim CVEs and fleets that cannot immediately recycle. TechLearningPro should prefer immutable recycle from a new AMI when possible, and use Patch Manager where long-lived workers or compliance scanning demand it. Do not invent a fake “zero reboot forever” promise—read current OS and Patch Manager behavior for reboot handling.

Automation documents (conceptual)

Automation runs multi-step operational workflows defined as documents: create AMI, stop instance, run checks, branch on success/failure, notify. Conceptual uses for TechLearningPro: controlled AMI bake runbook; pre-patch drain and post-patch health verification; standardized “collect logs and upload to S3” incident pack.

Treat Automation documents like code: version them, review them, restrict who may execute them, and prefer them over tribal knowledge in a wiki that disagrees with reality. This lesson does not dump every Automation action type. You need the mental model: repeatable ops as documents with IAM gates and execution history.

Inventory and State Manager (awareness)

Inventory collects metadata about managed instances—applications, network config, and other inventoried data types you enable—so you can answer “which hosts still run library X?” without SSH archaeology. Use it for awareness and compliance inputs; do not confuse inventory presence with vulnerability scanning depth from dedicated security products.

State Manager associates a desired state document with a schedule or association so configurations converge (agent settings, package presence, file contents at a controlled level). Awareness here prevents two anti-patterns: using State Manager as an ad-hoc CI/CD for application deploys, and ignoring drift entirely. Application releases belong in deployment pipelines; State Manager belongs to configuration that must hold between deploys.

Parameter Store as light configuration (preview)

Systems Manager Parameter Store stores configuration strings and SecureStrings under hierarchical names such as /tlp/staging/progress-api/log-level. Run Command documents and bootstrapping often read parameters for non-secret flags. SecureString parameters encrypt with KMS. Lesson 21 compares Parameter Store to Secrets Manager for rotation, staging labels, and application retrieval patterns. Here: know that fleet ops may consume parameters; do not bake database passwords into AMI user data or shell history because “SSM can fix it later.”

IAM least privilege for operators and instances

Two IAM stories must stay distinct:

• Instance role: allows the agent to communicate with SSM, write session logs if configured through instance-side paths, and read only the parameters or artifacts the host needs.
• Operator / automation role: allows StartSession, SendCommand, patch operations, and Automation execution on selected resources. Condition keys and tag-based authorization keep a contractor from sessioning into production payment-adjacent hosts if your account still co-hosts them.

SSO permission sets should map to these jobs. “AdministratorAccess for whoever is on-call” is how Session Manager’s audit beauty is wasted.

Logging, audit, and forensics posture

Enable Session Manager logging to CloudWatch Logs or S3 per current Session Manager documentation when policy requires command audit. Combine with AWS CloudTrail for API-level who-started-what. Run Command outputs and Automation executions create operational history. Together they answer: who patched, who shelled in, what document ran. Without logging, SSM is convenient; with logging, SSM is governable.

Cost awareness without invented prices

Billable shapes you must plan for in labs: EC2 instance hours, EBS, NAT data processing if you use NAT instead of endpoints, VPC interface endpoint hourly charges per AZ and data processing, CloudWatch Logs ingestion for session logs, and S3 storage for logs. Prefer the smallest burstable instances for labs; limit endpoint AZs if your lab design allows while still proving private connectivity; delete endpoints and instances in cleanup. Never quote a memorized dollar amount—check current pricing pages for your Region when you estimate.

Hybrid and non-EC2 awareness (boundary)

Systems Manager can manage on-premises and multi-cloud servers with the hybrid activation model in broader AWS practice. This course’s Module 2 focus is EC2 fleet operations for TechLearningPro. Know hybrid exists so interview answers are not blank; do not derail the lesson into activation codes unless your sandbox exercise explicitly needs it.

Architecture and Design Diagrams

Session Manager path without inbound SSH:

mermaidCopy
flowchart TB
  Ops["Operator IAM principal"]
  SSM["Systems Manager Session Manager"]
  EP["VPC interface endpoints\nfor SSM-related services"]
  Agent["SSM Agent on progress API EC2"]
  API["Private subnet instance\nno inbound :22"]
  Ops -->|"StartSession"| SSM
  SSM --> EP
  EP --> Agent
  Agent --> API

Run Command and Patch Manager relative to the ASG:

mermaidCopy
flowchart LR
  Tags["Tags: Role=progress-api Env=staging"]
  RC["Run Command document"]
  PM["Patch baseline + window"]
  ASG["ASG instances with SSM Agent + instance profile"]
  Tags --> RC
  Tags --> PM
  RC --> ASG
  PM --> ASG

Three-plane managed-instance checklist:

textCopy
Agent running? → Instance profile attached & sufficient?
  → HTTPS to SSM endpoints (endpoints or NAT) allowed by SG/NACL/routes?
    → Instance appears as managed → Session / Run Command / Patch allowed by operator IAM?

Technical Explanation

Managed instance registration flow

When an EC2 instance boots with a valid instance profile and network path, the SSM Agent establishes communication with Systems Manager in the Region. The instance becomes visible to Fleet Manager / managed instance views used for Session Manager and Run Command targeting. If registration fails, the console simply omits the instance from “online” managed lists—there is often no dramatic error on the EC2 page itself. That asymmetry is why operators open SSH: they looked in the wrong place. Train yourself to check SSM agent logs on the instance once you have any temporary access path, or to validate IAM and endpoints from outside first.

Launch templates must carry the instance profile. An ASG that launches without the profile produces a fleet that looks healthy to the load balancer and invisible to SSM. Lesson 8’s baker host that had SSM does not help if the template forgot the profile.

VPC endpoints for Systems Manager

Private subnets often reach AWS APIs through NAT. For administration-only reachability, many designs add interface VPC endpoints so instances can reach Systems Manager (and related endpoint services required for Session Manager and advanced features) privately. Exact endpoint service names and which combination you need can evolve—verify against current AWS documentation for Session Manager VPC endpoint requirements before you paste IaC. Security groups on interface endpoints must allow inbound 443 from instance security groups (or subnet CIDRs, less ideal). Instance security groups need outbound 443 to the endpoint security group or prefix.

Gateway endpoints for S3 may also matter when Session Manager or Patch Manager workflows pull objects from S3. Design the dependency list; do not assume “one SSM endpoint” is always sufficient—confirm the current list for your feature set (Session Manager, Run Command output to S3, patch source buckets, and so on).

Session Manager preferences and access control

Account or Region-level Session Manager preferences can enforce encrypted session logs, shell profiles, and idle timeouts per current capabilities. Operators still need IAM permission to start sessions. You can require MFA on the human side through your identity center policies. Prefer tag-scoped policies: ssm:StartSession allowed when ssm:resourceTag/Env matches the caller’s permitted environments.

Port forwarding and SSH-over-Session-Manager patterns exist for tooling that still speaks SSH locally while the instance has no inbound 22 from the internet. Use them deliberately; document them; do not confuse “SSH protocol locally” with “security group open to the world.”

Documents: AWS-provided versus custom

SSM documents define actions: AWS-RunShellScript, patch documents, Automation runbooks, Session Manager documents. Custom documents should live in version control and promote through environments. Parameters to documents should avoid secret literals; pass Parameter Store names or Secrets Manager ARNs when values are sensitive (lesson 21). Review document parameters the way you review user data: anything that lands in logs is in scope for redaction policy.

Command invocations and concurrency

Run Command invocations have status lifecycles (pending, in progress, success, failed, timed out, cancelled—confirm current status names in docs). Rate and concurrency limits exist; blasting ten thousand instances with a heavy script is an availability risk to the guests and a throttle risk to the API. For TechLearningPro’s fleets, tag-scoped batches and maintenance windows beat heroic global fan-out. Idempotent scripts matter: Run Command retries and human re-runs happen.

Patch compliance as a signal

Compliance summaries tell you which instances diverge from baseline. Wire awareness into operations: a staging ASG at 0% compliance after a baseline change is expected until the window runs; production at 40% noncompliant for thirty days is a process failure. Do not equate “scanned” with “safe.” Scanning without installation is measurement. Installation without soak testing is gambling. AMI recycle plus scanning is often the sober enterprise pattern.

Maintenance windows and freeze culture

Maintenance windows associate tasks (Run Command, Automation, patching) with schedules and targets. Even if your first lab only schedules a simple shell document, practice the idea that disruptive work has a clock. TechLearningPro enrollment launches and live cohort hours are freeze candidates. Document freezes next to the window. An Automation that ignores freeze culture will eventually patch during a keynote demo.

Inventory collection intervals and privacy

Inventory is powerful and can collect software lists that include unexpected detail. Align collection with privacy and security policy. Limit which data types you enable. Treat inventory output as sensitive operational data in multi-tenant teaching platforms when host packages reveal unpublished features.

State Manager versus deploy pipelines

If State Manager installs the progress API version from a mutable S3 object every thirty minutes while CI also deploys, you will fight yourself. Pick a source of truth. Recommended split for this course: CI/CD deploys application versions; State Manager enforces agent configuration, required packages for operations, and guardrail files; Patch Manager handles OS patches or feeds compliance into AMI rebuild triggers.

Parameter hierarchies for ops config

Use paths like /tlp/{env}/{service}/{key} for log levels, feature flags that are not capabilities secrets, and endpoint hostnames that are not credentials. Access policies can mirror hierarchy. SecureString for anything that approaches credential shape. Full decision tree with Secrets Manager is lesson 21—here, stop putting DB_PASSWORD in plain String parameters “temporarily.”

Interaction with Auto Scaling lifecycle

On scale-out, new instances must self-register with SSM quickly enough that on-call can session during an incident. On scale-in, sessions end; do not store unique state only in a live session. Warm pools and hibernation edge cases can confuse agent state—prefer simple ASG recycle in labs. Instance refresh (lesson 8) plus SSM means every refreshed host must still meet the three planes.

Security groups for endpoints and instances

Endpoint security groups are often misconfigured: operators allow outbound from instances to 0.0.0.0/0 and forget that interface endpoints are ENIs that need inbound 443 from the instance SG. Symptom: NAT-based works, endpoint cutover breaks SSM. Test Session Manager after cutover, not only curl to public HTTPS.

CLI and console verification mindset

Use current AWS CLI commands such as describing instance information and listing command invocations—verify exact subcommands and flags against today’s CLI reference before scripting exams or pipelines. Console paths move; the concepts do not: managed status, document name, target tags, output, errors.

Disclaimer for all CLI examples in exercises: flags and parameter names change. Confirm against AWS documentation current at implementation time. Do not memorize flags from older blog posts as eternal truth.

Practical Implementation or Design Walkthrough

Scenario

You operate TechLearningPro staging: two progress API instances in private subnets across two AZs, plus a tagged content worker. Goal: remove bastion dependency, prove Session Manager, run a fleet diagnostic via Run Command, and outline a patch baseline + maintenance window approach without surprising Monday traffic.

Step-by-step design walkthrough

1. Budget and tags. Confirm a budget alert. Decide tag set: Project=tlp-lesson20, Env=staging, Role=progress-api or Role=content-worker.

2. Launch template check. AMI with SSM Agent; instance profile for SSM; IMDSv2 required; no SSH key required for the operational story (keys optional for break-glass only if policy demands); security group without inbound 22 from the internet.

3. Network path. Either NAT already exists from Module 1 labs, or create interface endpoints required for Session Manager per current docs, with endpoint SG allowing 443 from instance SG. Record endpoint IDs for cleanup.

4. Prove managed status. Wait for instances to show as managed. If not, walk the three planes before any SSH temptation.

5. Session Manager. From console or CLI identity with least privilege, start a session to one progress API host. Echo a non-secret hostname; exit. Confirm session logging destination if you enabled it.

6. Run Command. Target Role=progress-api. Run a shell document that prints app version file or systemctl is-active for your service. Save invocation ID and status as evidence. Run a deliberate failure (wrong tag) to capture a negative path.

7. Patch awareness. Create or select a patch baseline appropriate to the OS (follow current console/API guidance). Tag a patch group. Run a scan first; capture compliance summary. Do not install reboot-heavy patches on a shared sandbox without a window and owner approval. Document what an install window would look like.

8. Automation sketch. Write a one-page outline of an Automation document that: scans; if noncompliant count exceeds threshold, notifies; optionally installs in staging only. You need not implement every step in the first lab pass, but the outline must be executable later.

9. Parameter light touch. Create a String parameter /tlp/staging/progress-api/log-level = INFO. Show how a Run Command script could read it with instance role permissions. Do not store passwords here in the lab.

10. Cleanup. Terminate or delete ASG/instances as appropriate; delete lab endpoints if created solely for this exercise; delete lab parameters; confirm tagged resources are gone; leave budget alert in place for the account.

Example: thin Run Command shell (illustrative)

bashCopy
#!/bin/bash
set -euo pipefail
# Illustrative only — verify document schema and Run Command parameters in current docs
echo "host=$(hostname)"
echo "role_tag_expected=progress-api"
if command -v systemctl >/dev/null 2>&1; then
  systemctl is-active tlp-progress-api || true
fi

Wrap scripts in SSM documents for reuse. Prefer reading configuration from Parameter Store over embedding environment-specific values in the document text.

Example: operator policy shape (conceptual)

Describe, do not paste a fake full production policy as if it were certified: allow ssm:StartSession on instances where Env=staging; allow ssm:SendCommand for specific document ARNs; allow ssm:GetCommandInvocation; deny session to Env=prod for this role. Attach via SSO permission set. Test AccessDenied on production tags as a required negative case.

Break-glass procedure

Document a break-glass path that still avoids permanent 0.0.0.0/0 SSH: temporary security group change from a VPN CIDR with ticket and expiry, or a short-lived bastion with MFA and session recording, plus mandatory teardown checklist. Break-glass that never expires becomes the real architecture.

Enterprise Real-World Scenarios

Scenario 1: The silent unmanaged ASG

TechLearningPro recycles the progress API AMI. The new launch template omits the instance profile. The board stays healthy; deployments succeed; at 03:00 a memory leak needs a heap dump. On-call cannot start Session Manager. Someone opens SSH from a wide CIDR “for fifteen minutes.” Three days later the rule remains. Root cause was not “SSM is unreliable”; it was an incomplete template. Fix: pipeline test that a fresh ASG instance becomes managed within N minutes; alarm on unmanaged count for critical tags.

Scenario 2: Patch Tuesday versus live cohort

A well-meaning engineer enables automatic patch install without a maintenance window on content workers that also serve live lab terminals. Mid-cohort reboots drop in-progress exercises. The incident review adopts scan-by-default, install only in windows, and AMI recycle for the API tier. Patch Manager remains; the calendar becomes part of the control.

Scenario 3: Run Command as undeclared CD

A team uses Run Command to pull latest jar from a mutable S3 key whenever someone feels like it. Staging and production drift; CloudTrail shows fragmented history; rollback is “run the other command.” They move application releases to the CI/CD pipeline and restrict Run Command to diagnostics and approved remediations. Change failure rate drops because the deploy path is singular.

Scenario 4: Endpoint cutover outage

Network team removes NAT from the data-adjacent ops subnet after adding only a subset of required interface endpoints. SSM Agent fails registration; Patch scans stop; only one jump host with old NAT still works, so people conclude “private network is fine.” Restoring the missing endpoints and adding a canary managed instance check per subnet fixes visibility. Moral: treat SSM reachability as a first-class network dependency with synthetic checks.

Scenario 5: Shared session credentials culture

Operators share a single IAM user access key to run SSM from laptops. Session logs show one identity for everyone. After a contentious change, nobody can prove who ran the destructive document. Moving to SSO with per-human permission sets restores accountability Session Manager was designed to provide.

Scenario 6: Inventory as unexpected disclosure

Inventory collection includes application lists that reveal an unreleased grading engine package name on a shared host used by contractors. Policy narrows inventory data types and restricts who may read inventory for production accounts. Capability without data governance is still risk.

Best Practices

1. Prefer Session Manager over bastions for private TechLearningPro fleets when agent, IAM, and endpoints are ready.
2. Bake SSM Agent and instance profile into launch templates; verify managed status on ASG soak tests.
3. Use VPC endpoints for Systems Manager paths when private-only administration is a design goal; verify the current required endpoint set.
4. Scope operator SSM permissions by tags, environments, and document ARNs.
5. Enable session and command audit logging aligned with organizational policy.
6. Target Run Command by tags, not stale instance ID spreadsheets.
7. Scan before install; install inside maintenance windows; respect freezes around enrollment events.
8. Keep Automation and documents in version control; review like application code.
9. Separate application deploy pipelines from State Manager and Run Command remediations.
10. Use Parameter Store for configuration hierarchy; reserve secret rotation patterns for lesson 21 services design.
11. Tag everything; clean up lab endpoints and instances; never leave break-glass SSH open.
12. Alarm on unmanaged instance counts for critical roles.
13. Test negative IAM paths (AccessDenied) as part of lab evidence.
14. Prefer AMI recycle for API tiers when speed allows; use Patch Manager as complement, not as a substitute for image pipelines.
15. Document break-glass with expiry and owner—not tribal SSH keys in a password manager titled final-final.

Common Mistakes and Anti-Patterns

1. Opening SSH to fix SSM. Symptom: security findings and forgotten rules. Cause: treating network allow as easier than diagnosing agent/IAM/endpoints. Fix: three-plane checklist first; temporary break-glass with ticket if truly blocked.

2. Instance profile only on the baker. Symptom: ASG instances unmanaged. Cause: launch template omission. Fix: template + automated managed-status test.

3. Run Command distributing secrets. Symptom: passwords in output logs and CloudTrail-adjacent artifacts. Cause: convenience. Fix: Secrets Manager / SecureString retrieval patterns (lesson 21); redact; deny documents that echo secrets.

4. Patch install without windows. Symptom: reboots during peak. Cause: “critical means now” without product calendar. Fix: maintenance windows, soak in staging, AMI strategy.

5. AdministratorAccess for all SSM operators. Symptom: untraceable blast radius. Cause: urgency. Fix: tag-conditioned policies and SSO sets.

6. Assuming one VPC endpoint is enough. Symptom: intermittent Session Manager failures after NAT removal. Cause: incomplete endpoint set or SG on endpoint ENIs. Fix: verify current Session Manager endpoint guidance; test sessions after cutover.

7. Using State Manager as the application CD. Symptom: drift wars with pipelines. Cause: tool confusion. Fix: ownership boundaries written in the ADR.

8. Untargeted command fan-out. Symptom: mass failures and throttling. Cause: targeting entire account. Fix: tags, rate awareness, staged rollout.

9. No session logging in regulated accounts. Symptom: audit gaps after an incident. Cause: defaults left on. Fix: preferences + log retention + access controls on log destinations.

10. Ignoring hybrid/agent versions until something breaks. Symptom: odd document failures on old agents. Cause: ancient AMIs. Fix: agent version inventory and AMI refresh cadence.

Troubleshooting and Design Review

Troubleshooting map

Design review questions

• Do private instances require inbound SSH for normal operations? If yes, why has Session Manager not cleared the three planes?
• Is managed status part of ASG definition-of-done?
• Who can run which documents against production tags?
• Are patch installs bound to windows and freezes?
• Are session logs retained and restricted?
• Will endpoint charges and lab instances be cleaned up?
• Is Parameter Store used for config without dumping secrets into String parameters?
• Is there a tested break-glass that expires?

Operator runbook snippet

When on-call cannot session: verify instance managed; verify caller IAM; verify endpoints; escalate to network if subnet-wide; avoid permanent SSH. When patch compliance tanks overnight: check for baseline change, failed windows, or new ASG launched into wrong patch group tag. When Run Command fails on half the fleet: sort by AMI age and agent version before rewriting the script.

Interview Questions and Answers

Q1. Why prefer Session Manager over a bastion for private EC2?

It removes inbound administrative ports from the typical path, uses IAM for authorization, works with private subnets via endpoints, and can log sessions for audit—reducing key distribution and bastion patch burden.

Q2. What three prerequisites must exist for an EC2 instance to be SSM-managed?

A running SSM Agent, an instance profile with Systems Manager permissions, and network connectivity to Systems Manager endpoints (NAT or VPC interface endpoints), plus Regional service availability.

Q3. How do security groups change when you adopt Session Manager?

Administrative inbound SSH/RDP from the internet can close; outbound (or endpoint-oriented) HTTPS paths for the agent become critical; endpoint ENI security groups must allow instance traffic on 443.

Q4. What is Run Command used for?

Executing SSM documents on fleets selected by IDs, tags, or resource groups for diagnostics, remediations, and controlled operational tasks with status and output history.

Q5. What is a patch baseline?

A definition of which patches are approved for an OS ecosystem, including classification/severity rules and approval settings, used by Patch Manager to evaluate and optionally install updates.

Q6. Why do maintenance windows matter?

They schedule disruptive operations so patching and reboots do not collide with peak business traffic such as TechLearningPro live cohorts or enrollment launches.

Q7. How is Automation different from a one-off Run Command?

Automation documents orchestrate multi-step runbooks with execution tracking—suited to repeatable operational procedures—whereas a single Run Command invocation is typically one document execution on targets.

Q8. What do Inventory and State Manager contribute?

Inventory reports installed software and related metadata for visibility; State Manager applies desired-state associations over time. Neither replaces a proper application deployment pipeline.

Q9. How should IAM differ between instance roles and operator roles for SSM?

Instance roles let agents register and perform host-side duties; operator roles authorize StartSession, SendCommand, and patch/automation actions, ideally scoped by tags and document ARNs.

Q10. An instance is healthy in the target group but missing from Session Manager targets—what do you inspect first?

Managed-instance prerequisites: agent, instance profile, and path to SSM endpoints—not the load balancer health check path.

Q11. How does Parameter Store relate to this lesson versus the next?

Here it is a configuration source for ops and bootstrap. Lesson 21 compares it to Secrets Manager for secret lifecycle, rotation, and application retrieval patterns.

Q12. When is a bastion still justified?

When tooling or policy mandates SSH workflows, when agents cannot run, or as a tightly controlled break-glass—never as permanent open SSH alongside an unused Session Manager design.

Hands-On Exercises

Exercise: Session Manager + Run Command + patch scan

Problem statement. In a sandbox, launch or reuse two tagged private instances (or an ASG min=2) with SSM Agent and instance profiles. Prove Session Manager without inbound SSH from the internet. Run a tag-targeted command. Perform a patch scan and capture compliance evidence. Budget first; cleanup instances and any lab-only VPC endpoints.

Tasks.

1. Confirm budget alert; apply Project and Role tags.
2. Validate managed status for both instances; record evidence.
3. Start a Session Manager session; run a harmless command; exit.
4. Send a Run Command document to Role=progress-api (or your tag); save invocation results.
5. Demonstrate AccessDenied or empty targets for a wrong tag or denied identity (negative test).
6. Associate or apply a patch baseline strategy; run scan; capture compliance summary.
7. Write five lines on when you would install versus recycle AMI.
8. Delete lab compute and lab endpoints; verify tagged resources are gone.

Evaluation rubric.

Challenge

Add a maintenance window that runs a non-disruptive Run Command (for example, a disk usage diagnostic) on staging tags only, outline an Automation-style ADR for staging patch install with rollback via ASG instance refresh to the previous AMI, and state how production freezes would block the window during a live cohort.

Challenge ADR shape

Decision: TechLearningPro EC2 fleets use Session Manager as primary admin access; Run Command for tagged diagnostics; Patch Manager scan continuous with install only in maintenance windows for workers; progress API prefers AMI recycle; VPC interface endpoints for SSM in private subnets; session logs to a restricted log group.

Consequences: Endpoint cost to manage; IAM discipline required; bastion optional break-glass only; patch windows must align with product calendar; launch templates become part of security control evidence.

Lab evidence checklist

Record instance IDs, AMI IDs, instance profile names, endpoint IDs if any, tag keys, Session Manager session IDs or log evidence, Run Command invocation IDs, patch scan summary export, budget alert identifier, and cleanup timestamps.

Deep dive: Session Manager logging that auditors accept

Session logging is only useful if someone can retrieve it during an incident without discovering that logs were never enabled, were written to an open bucket, or were retained for three days against a one-year policy. For TechLearningPro production accounts, decide the destination first: CloudWatch Logs for searchable operational review, S3 for long-term archive, or both when policy requires. Encrypt log destinations with keys your security account understands. Restrict who may read session logs—operators who can start sessions should not automatically gain unbounded log replay across every environment if separation of duties matters.

Prefer enforceable Session Manager preferences at the account or Region level so a single unconfigured laptop profile cannot silently disable logging. Test that a staging session produces a log object or log stream within minutes. Include that test in the same definition-of-done checklist that proves managed status after ASG launch. If logging fails open—sessions work but logs do not—treat it as a control failure equal to missing CloudTrail in an account that claims auditability.

Idle session timeouts reduce the blast of abandoned shells on shared jump culture that Session Manager was meant to end. Shell profiles can present a legal banner and set safer umask defaults; keep them thin so they do not become another configuration management system. Document whether port-forwarding sessions are allowed in production; many teams allow them in staging only until tooling catches up.

Deep dive: Run Command output, S3, and evidence packs

Large command outputs may be truncated in the API response path depending on size. Designs that need durable evidence—compliance packs, incident forensics, weekly fleet diagnostics—should send output to an S3 bucket with encryption, lifecycle rules, and deny-public policies. Instance roles then need narrow s3:PutObject on a prefix such as ssm-output/staging/. Do not grant read on that prefix to every instance role; writers and readers differ. When the output itself might contain sensitive host data, treat the bucket as sensitive and align retention with privacy policy.

Command invocation IDs belong in tickets the way deployment IDs do. “We ran something last week” is not an audit trail. Teach on-call to paste invocation IDs into the incident timeline. Failed invocations with partial output are still evidence: they show where a script died and on which instance AMI.

Deep dive: Patch Manager and immutable progressive delivery

Enterprises argue endlessly between “patch in place” and “recycle from AMI.” TechLearningPro’s progress API should bias immutable: weekly or on-CVE AMI builds, instance refresh, previous template retained for rollback. Content workers and build workers that hold longer-lived local caches may still need interim Patch Manager installs when AMI rebuild lag exceeds risk tolerance. Write that bias in the ADR so two teams do not invent opposite religions on the same account.

Baselines should be environment-aware. Staging may auto-approve faster to soak; production may delay auto-approval to watch staging compliance and application soak. Rejected patch lists exist for a reason: a bad kernel update that broke SSM Agent last quarter stays rejected until a fixed version arrives. Track rejects as code review artifacts, not tribal memory.

Reboot behavior differs by OS and patch type. Maintenance windows must assume some fraction of hosts reboot. For ASG-backed APIs, prefer rolling recycle over simultaneous reboot of the entire min capacity. For single workers without peers, schedule true downtime or stand up temporary capacity first. Patch Manager is a tool; capacity design remains yours.

Deep dive: Automation as executable runbooks

Wiki runbooks rot. Automation documents that create AMIs, drain connections via a documented hook, snapshot before risky change, or gather a standard incident bundle age better when versioned in git and promoted like infrastructure. Gate execution with IAM so only the platform team can run production Automations that stop instances. Require approvals outside AWS if your change system demands them—AWS execution history complements, not replaces, change tickets.

Start small: one Automation that takes instance IDs tagged Env=staging, runs a health document, and publishes a success/fail summary to an SNS topic you already own. Grow into patch orchestration only after the simple document is trusted. Resist importing a hundred shared Automations from the internet without reading their steps; they are code running with your privileges.

Connecting VPC endpoints to FinOps and reliability

Interface endpoints charge while they exist. A forgotten lab endpoint in three AZs becomes quiet spend. Reliability wants endpoints in each AZ where instances run so you do not hairpin through another AZ unnecessarily; labs may start with fewer endpoints to save cost if you accept the trade-off and document it. Production TechLearningPro private subnets that removed NAT should include synthetic canaries: a tiny scheduled Run Command that only echoes ok and alarms on failure. That canary detects endpoint SG regressions before Monday’s human session attempt.

DNS settings on the VPC must resolve endpoint private DNS names when you rely on private connectivity. Teams that disable private DNS “temporarily” while debugging and forget to re-enable create mysterious public-path dependency again. Record DNS enablement in the network ADR beside the endpoint list.

Game-day script for unmanaged fleets

Quarterly, terminate one staging progress API instance during a game day and require the responder to session into the replacement within a timebox without opening SSH. Separately, revoke the instance profile from a non-critical worker and confirm alarms fire when managed count drops. Separately, block endpoint SG inbound briefly in staging and confirm detection. These drills hurt less than learning during a production memory leak at 05:40.

Content worker and build worker specifics

Content workers may process learner uploads and need Run Command diagnostics for stuck jobs without granting developers Session Manager into production. Build workers may use Spot (lesson 8) and churn faster; SSM still matters for the On-Demand floor hosts that debug pipeline failures. Tag Role=content-worker and Role=build-worker distinctly so patch baselines and command documents do not collide. A build worker baseline that reboots mid-pipeline is an avoidable CI outage—windows belong outside peak build schedules if your org has them.

What this lesson deliberately leaves out

Distributor package catalogs, every Compliance pack, OpsCenter deep workflows, and Explorer rollups across organizations exist in the Systems Manager family. Knowing they exist prevents false confidence that you have “finished SSM” after Session Manager works. This course’s next priority for secrets is lesson 21; organization-wide compliance aggregation belongs with later governance lessons when multi-account is in scope. Curiosity is good; unbounded console tourism in a paid sandbox is how budgets die.

Narrative: Monday 05:40 without SSH

The progress board latency page fires. On-call assumes the role for production Session Manager, lists managed instances for Role=progress-api, starts a session on one host in AZ-a, checks systemd and recent deploy version via a read-only command document already approved, compares against AZ-b, and finds a bad config flag pushed through an out-of-band Run Command the previous evening. They roll the ASG to the last known launch template version, confirm managed status on replacements, and open a problem record that bans out-of-band config pushes—the same moral as undeclared CD. No security group changed. No key pair was found in a vault. The audit log shows who did what. That is the operating model this lesson is selling.

Lesson Summary

Fleet operations are a design you prove on every launch template, not a console feature you remember during an outage. Systems Manager makes TechLearningPro’s private progress API and worker fleets operable without inbound SSH when the agent, IAM, and endpoint planes align. Session Manager replaces bastion-first habits; Run Command gives auditable fleet actions; Patch Manager and maintenance windows keep updates from ambushing Monday; Automation turns tribal fixes into documents; Inventory and State Manager add awareness and drift control without stealing CD’s job. Parameter Store appears as configuration hierarchy—secret lifecycle waits for the next lesson.

Next: Secrets Manager and Systems Manager Parameter Store decides where credentials live, how rotation works, and how applications retrieve secrets without baking them into user data, AMIs, or everlasting environment variables. Later messaging lessons (EventBridge, SNS and SQS) will consume the same private, IAM-first posture for event-driven workers you can still operate through this fleet tooling.

Further Reading

• AWS Systems Manager User Guide
• AWS Systems Manager Session Manager
• Setting up Session Manager
• VPC endpoints for Systems Manager
• AWS Systems Manager Run Command
• AWS Systems Manager Patch Manager
• AWS Systems Manager Automation
• AWS Systems Manager Inventory
• AWS Systems Manager State Manager
• AWS Systems Manager Parameter Store
• Amazon EC2 launch templates