AWS Cloud & DevOps Tutorial 0/70 lessons ~6 min read Lesson 60“IAM, VPC, EC2, Lambda, ECS, EKS, Terraform, CI/CD, and AWS operations”

Bash and Python Automation for AWS Operations

Manual Session Manager clicks do not scale across TechLearningPro’s accounts, ASGs, and Monday incident pressure. Operators need **Bash and Python automation** that inventories res

Course progress0%

Focus

15 guided sections

Practice signal

Examples included

Career prep

Interview Q&A included

Bash and Python Automation for AWS Operations

AWS Cloud & DevOps Engineering · Lesson 60 · Production Projects and Interview

Lesson Overview

Manual Session Manager clicks do not scale across TechLearningPro’s accounts, ASGs, and Monday incident pressure. Operators need Bash and Python automation that inventories resources, checks compliance tags, pages through AWS APIs safely, retries throttling, logs structured evidence, and runs under IAM roles—never under hardcoded access keys pasted into scripts. This lesson turns lesson 59’s host checks and earlier SSM skills into auditable utilities: shell for glue and SSM documents, boto3 for AWS API work, pagination and retries done correctly, and execution via instance roles, CloudShell, or CI OIDC—not long-lived keys in git.

You will design an inventory and compliance reporting utility for TechLearningPro sandbox resources tagged for Module 6 labs. Cross-link /learn/unix for shell depth; do not clone it. Tag artifacts Project=tlp-lesson60. Budgets on before any billable API chatter that creates resources; prefer read-only inventory first.

Learning Objectives

Explain when Bash, Python/boto3, SSM Run Command, or IaC should own an operational task.
Authenticate with roles/profiles/OIDC—never commit access keys.
Use boto3 paginators and NextToken loops correctly for inventory APIs.
Apply retries/backoff for throttling without inventing non-existent client parameters.
Emit structured logs (JSON lines) suitable for CloudWatch Logs Insights.
Write Bash wrappers that fail closed (set -euo pipefail patterns) and quote variables.
Invoke SSM Run Command for fleet scripts instead of SSH for-loops.
Scope IAM least privilege for inventory versus mutate operations.
Build a tag compliance report (missing Project/Owner/Env tags).
Handle partial failures and exit codes that CI can gate on.
Separate read-only reporting roles from break-glass mutate roles.
Store config in Parameter Store / files—not secrets in source.
Document dry-run flags for any destructive automation.
Estimate API call volume and cost awareness for chatty scripts.
Prepare interview answers on automation safety and key hygiene.
Connect outputs to FinOps and Security Hub style evidence packs.
Avoid scraping console HTML; use supported APIs/CLI/SDK only.
Test negative paths: AccessDenied, empty pages, throttling simulation.
Hand off cleanly to DevSecOps pipeline gates in lesson 61.
Cleanup: delete lab buckets/functions/roles created for the utility.

Prerequisites

AWS CLI, SDKs and Infrastructure Access.
AWS Systems Manager and Fleet Operations.
Lesson 59 Linux administration habits for commands you automate.
IAM for roles and least privilege.
Optional: /learn/unix for Bash discipline.
Sandbox + budget; Python 3 with boto3 in a venv or CloudShell.

Detailed Core Concepts

Automation as an operations product

Scripts are production software. They need owners, version control, code review, least privilege, logging, and rollback/dry-run. TechLearningPro treats inventory/compliance utilities as platform products: a wrong EC2 terminate script is an outage generator.

Bash versus Python versus SSM versus IaC

Bash: glue, jq over CLI JSON, packaging SSM document payloads, CI steps.
Python/boto3: multi-service inventory, complex filtering, CSV/JSON reports, retries.
SSM Run Command / Automation: execute on fleets with audit; good for host-local checks from lesson 59.
Terraform/CloudFormation: desired state of infrastructure—not a substitute for daily compliance reports, but complementary.

Pick the thinnest safe tool. Do not write a Python framework to echo one CLI describe.

Credential rules (non-negotiable)

No AWS_ACCESS_KEY_ID hardcoded in repos, AMI user-data, or Slack.
Prefer instance/task/IRSA roles, IAM Identity Center profiles, or CI OIDC assume-role.
Locally: named profiles via SSO or short-lived keys in the OS keychain/credentials file excluded from git.
Rotate and delete lab keys immediately; treat leaked keys as incident.

boto3 clients, paginators, errors

Use service clients (boto3.client("ec2")) and paginators where AWS provides them (get_paginator("describe_instances")). For APIs with NextToken/Marker, loop until empty—do not assume one page. Catch ClientError; inspect codes like AccessDeniedException, Throttling, RequestLimitExceeded (verify exact codes per service in current docs). Use documented retry configuration / botocore standard retry modes rather than inventing sleep magic without understanding.

Structured logging and exit codes

Log event, account, region, resource_id, result as JSON lines. Exit non-zero when compliance fails so pipelines can gate. Distinguish “script error” from “found violations.”

Safe execution patterns

--dry-run for any mutate path (and AWS DryRun where supported—verify per API).
Allowlists of accounts/regions.
Rate awareness; avoid hammering Describe* across all regions every minute without need.
Write outputs to S3 or local artifacts tagged Project=tlp-lesson60 for evidence.

Inventory and compliance utility shape

Input: account/region list, required tag keys. Process: list EC2/ASG/RDS/S3 (as permitted), evaluate tags, emit CSV/JSON, summary counts. Output: report + non-zero exit if critical resources missing Project tag. Extend later with security group openness checks—without inventing Security Hub APIs that do not exist; use supported describe APIs and document limitations.

Architecture and Design Diagrams

mermaid
flowchart TB
  Op[Operator / CI OIDC]
  Role[IAM role least privilege]
  Py[Python boto3 utility]
  API[AWS Describe APIs]
  Out[JSON/CSV report]
  SSM[SSM Run Command optional]
  Host[Fleet host checks]

  Op --> Role --> Py --> API --> Out
  Op --> Role --> SSM --> Host

mermaid
sequenceDiagram
  participant U as Utility
  participant AWS as AWS API
  U->>AWS: Describe (page 1)
  AWS-->>U: Resources + NextToken
  U->>AWS: Describe (page N)
  AWS-->>U: Resources
  U->>U: Evaluate tags / write report

Technical Explanation

Example control flow (illustrative)

A progress-API compliance job assumes tlp-ops-inventory-readonly, iterates allowed regions, paginates describe_instances, filters by TechLearningPro VPCs or tags, records missing Project/Owner, writes s3://tlp-ops-reports/... (bucket optional in lab), exits 1 if production-tagged resources miss tags. Mutating remediation is a separate change-controlled job.

Bash companion

bash
#!/usr/bin/env bash
set -euo pipefail
PROFILE="${AWS_PROFILE:?set AWS_PROFILE}"
aws ec2 describe-instances --profile "$PROFILE" --region "$REGION" \
  --query 'Reservations[].Instances[].InstanceId' --output text

Quote variables; prefer aws CLI with profiles over embedding secrets. For complex logic, call Python.

Pagination pitfall

One-page scripts look fine in sandbox (few instances) and miss resources in production. Always paginate. Test with enough data or mocked multi-page fixtures.

Throttling

Large Organizations inventory can throttle. Use exponential backoff via botocore retries; shard by account; cache read-mostly data. Do not tight-loop on failure.

SSM for host facts

Python gathers cloud inventory; SSM document gathers df, unit status (lesson 59). Merge reports by instance id. Still no SSH for-loops with keys on a jump box.

Cost note

Describe APIs are usually low cost but not free at extreme chatty scale; S3 report storage and any Lambda schedule are billable. Label estimates illustrative; verify pricing pages. Delete lab report buckets.

Practical Implementation or Design Walkthrough

Problem

Build an inventory and tag-compliance reporting utility for TechLearningPro sandbox resources.

Steps

Budget alert on; decide read-only IAM policy (ec2:Describe*, optionally s3:ListAllMyBuckets / rds:Describe*—narrow further in production).
Create IAM role/user profile for lab—no long-lived keys in git.
Implement Python script with paginator for EC2 instances; print JSON lines of instance id + tags.
Add compliance evaluation for required tags Project, Owner, Env.
Add Bash entrypoint that sets profile/region and invokes Python.
Negative test: run with a role missing Describe—expect AccessDenied handling.
Optional: SSM Run Command to collect df -h from a tagged instance; attach output snippet.
Write ADR: why Python+boto3 over pure Bash for this report.
Tag any lab resources Project=tlp-lesson60; produce sample report artifact.
Cleanup roles/buckets/instances; save evidence.

Enterprise Real-World Scenarios

Keys in git

Access key committed; bot scans exploit. Fix: purge keys, rotate, secret scanning, OIDC.

One-page inventory

Missed untagged prod ASG. Fix: paginators + multi-account loop.

Mutate without dry-run

Script terminated wrong ASG. Fix: dry-run default; allowlisted accounts; dual control.

SSH for-each

Key distribution nightmare. Fix: SSM Run Command documents.

Silent success on failure

CI green despite AccessDenied. Fix: check exit codes; fail closed.

Best Practices

Roles/OIDC only; never hardcoded keys.
Paginators always for list/describe collections.
Structured logs and meaningful exit codes.
Dry-run defaults on destructive tools.
Least privilege separate from break-glass.
Code review automation like application code.
Prefer SSM over SSH fleets.
Allowlist accounts/regions.
Test AccessDenied paths.
Tag reports and lab resources.
Document API permissions in README.
Backoff on throttle; don't hammer.
Keep Bash thin; Python for logic.
Budget before scheduling chatty jobs.
Cleanup lab artifacts same day.

Common Mistakes and Anti-Patterns

Hardcoded secrets in scripts.
Ignoring pagination.
Scraping the AWS console.
Catching all exceptions and printing 'ok'.
World-writable script dirs on shared hosts.
Using root account access keys.
Unbounded fan-out across all regions every minute.
Embedding passwords in SSM command strings.
Inventing boto3 methods that do not exist.
Leaving lab keys active overnight.

Troubleshooting and Design Review

Symptom	Cause	Action
AccessDenied	Role too narrow or wrong account	Decode error code; adjust policy deliberately
Incomplete inventory	No pagination	Add paginator / NextToken loop
Throttling errors	Too chatty	Retries, shard, reduce frequency
Empty report	Wrong region/profile	Print caller identity; verify region
SSM command pending	Agent/IAM/network	Lesson 59/20 diagnostics
CI leaked key	Secret in logs	Rotate; redact; use OIDC

Interview Questions and Answers

Q1: Why not long-lived keys? A: Leak blast radius; prefer roles/OIDC short-lived creds. Q2: Paginator purpose? A: Complete inventory beyond first page. Q3: Bash vs Python? A: Glue vs complex multi-API logic. Q4: Dry-run? A: Safe default for destructive automation. Q5: Structured logs? A: Queryable evidence in incidents and CI. Q6: SSM vs SSH loops? A: IAM auditability and no inbound SSH. Q7: Fail closed? A: Non-zero exit on compliance breach. Q8: Least privilege? A: Read-only inventory role separate from mutate. Q9: Throttling response? A: Backoff/retry; reduce call rate. Q10: Secret scanning? A: Prevent keys in git; rotate if found. Q11: FinOps link? A: Chatty jobs and leftover lab stacks cost money. Q12: Negative test? A: Prove AccessDenied handling. Q13: Parameter Store? A: Config without baking secrets into code. Q14: Interview red flag? A: Keys in repo README 'for ease'. Q15: Next lesson? A: Supply chain gates for what automation deploys.

Hands-On Exercises

Exercise: Inventory and compliance reporting utility

Deliverable: Python+Bash utility, sample JSON/CSV report, IAM policy sketch, AccessDenied evidence, Project=tlp-lesson60 proof, cleanup.

Challenge ADR

Require OIDC for CI inventory jobs within 30 days; temporary lab keys allowed only on workstations with expiry ≤8 hours.

Lab evidence checklist

Caller identity output; paginated run; compliance fail exit code demo; no keys in git (git grep clean); destroy proof.

Narrative: pagination saves the audit

A one-page script cleared an audit; a second page held untagged RDS. Rewriting with paginators caught it before production promotion.

Continuity

Automation without key hygiene is liability. Next: Secure Software Supply Chain and DevSecOps.

Extended practice scenario 1 — 60 bash and python automation for aws operations

Confirm AWS Budgets alerts exist for the sandbox; tag every resource Project=tlp-lesson60. Do not proceed without a budget subscriber.

In a timed drill for Bash and Python automation for AWS operations at TechLearningPro, implement or design the core lab with a partner observing. The observer's job is to stop any step that invents API flags, hardcodes secrets, invents pricing numbers presented as current quotes, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: utility repo path, sample report, IAM policy sketch, AccessDenied log, tagging proof, and destroy confirmation for any lab resources.

Discuss how Bash and Python automation for AWS operations connects prior AWS lessons (IAM, VPC, EC2/SSM, CI/CD, observability, FinOps) and upcoming Module 6 projects. List two automation risks before Project 1, with owners and due dates.

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps become flashcards. Premium courses finish when you can operate under time pressure without treating Bash and Python automation for AWS operations as a mystery.

Extended practice scenario 2 — 60 bash and python automation for aws operations

Document the lesson's primary architecture decision in a ten-line ADR: context, decision, status, consequences.

Extended practice scenario 3 — 60 bash and python automation for aws operations

Map identity paths: who can create/modify the resources in this lesson; which IAM roles are least privilege.

Extended practice scenario 4 — 60 bash and python automation for aws operations

Design observability: CloudWatch metrics/alarms or equivalent signals that prove the control works.

Extended practice scenario 5 — 60 bash and python automation for aws operations

Run or design a negative test: denied access, misconfiguration, or failed dependency—capture evidence.

Extended practice scenario 6 — 60 bash and python automation for aws operations

Estimate cost dimensions for this lesson's activities. Label figures illustrative; verify pricing pages with dates.

Extended practice scenario 7 — 60 bash and python automation for aws operations

Produce a cleanup checklist and execute destroy for any billable lab resources created for this lesson.

Extended practice scenario 8 — 60 bash and python automation for aws operations

Cross-link prior course lessons that this design depends on; list two failure modes inherited from those layers.

Extended practice scenario 9 — 60 bash and python automation for aws operations

Security review: encryption at rest/in transit, public exposure checks, CloudTrail visibility for management events.

Extended practice scenario 10 — 60 bash and python automation for aws operations

Reliability review: failure isolation, rollback, how detection and recovery work under Monday traffic.

Extended practice scenario 11 — 60 bash and python automation for aws operations

Performance review: bottlenecks, scaling limits, or saturation signals relevant to this lesson's topic.

Extended practice scenario 12 — 60 bash and python automation for aws operations

Write a runbook section for on-call: symptoms, first three diagnostic steps, escalation, rollback.

Extended practice scenario 13 — 60 bash and python automation for aws operations

Config/compliance angle: which controls must be true before production promotion.

Extended practice scenario 14 — 60 bash and python automation for aws operations

Multi-account consideration: how this control behaves across sandbox, staging, and production accounts.

Extended practice scenario 15 — 60 bash and python automation for aws operations

Interview drill: explain the lesson topic to a hiring manager in three minutes without inventing APIs.

Extended practice scenario 16 — 60 bash and python automation for aws operations

Pair review: partner stops any invented CLI flags, hardcoded secrets, or fake current prices.

Extended practice scenario 17 — 60 bash and python automation for aws operations

FinOps continuity from lesson 47: idle resources, data transfer, and when to decommission lab stacks.

Extended practice scenario 18 — 60 bash and python automation for aws operations

DR continuity from lesson 45: ensure this design does not destroy recoverability during change.

Extended practice scenario 19 — 60 bash and python automation for aws operations

Produce evidence pack: diagrams, configs, test results, cleanup proof.

Extended practice scenario 20 — 60 bash and python automation for aws operations

Challenge constraint: tighter monthly budget—what do you sequence differently?

Extended practice scenario 21 — 60 bash and python automation for aws operations

Challenge constraint: stricter compliance—what evidence gates release?

Extended practice scenario 22 — 60 bash and python automation for aws operations

Challenge constraint: shorter maintenance window—what pre-staging changes?

Extended practice scenario 23 — 60 bash and python automation for aws operations

Teach-back: recite the Lesson Overview paragraph, then check against the text; flashcard gaps.

Extended practice scenario 24 — 60 bash and python automation for aws operations

Dress rehearsal: tabletop under a timer with observer notes.

Extended practice scenario 25 — 60 bash and python automation for aws operations

Final synthesis note: one paragraph connecting this lesson's topic to TechLearningPro production ownership.

Compliance evidence pack for auditors / platform leads

Budget configs, tagging proof (Project=tlp-lesson60), architecture diagram, ADR, negative-test evidence, encryption settings notes, CloudTrail/Config/SSM references where applicable, pricing-page retrieval dates for any dollar claims, cleanup confirmation.

Glossary drill

Define boto3, paginator, OIDC assume-role, dry-run, structured logging, SSM Run Command from memory.

Budget and cleanup sworn statement

I verified budgets remain enabled with alert subscribers. I deleted or scheduled deletion for orphan lab resources from lesson 60 named ____. I labeled every savings or cost figure with pricing-page retrieval dates or marked it illustrative. Signed ________ date ________.

Interview story to memorize

Craft a 60-second story: you built a boto3 tag compliance report with paginators and a read-only role—no keys in git—and caught untagged ASGs before exam week.

Self-assessment rubric

Area	Weight
Conceptual accuracy and service selection	20%
Working design/lab and reproducibility	25%
Security and least-privilege	20%
Reliability, observability, troubleshooting	20%
Documentation, ADR, cleanup	15%

Final synthesis before summary

Mastery of AWS ops automation is safe credentials, complete pagination, structured evidence, and dry-run discipline—not clever one-liners.

Extended facilitation script (print for guild leads)

Open with a leaked-key incident story (hypothetical). Ask: Where do credentials come from? How do we paginate? What exits non-zero? Who may mutate? Assign owners for secret scanning and for deleting leftover lab keys tagged Project=tlp-lesson60.

Lesson Summary

Bash and Python automation for TechLearningPro must inventory and enforce tag compliance with boto3 pagination, retries, structured logs, SSM for host checks, and zero hardcoded keys. Tag Project=tlp-lesson60; budget and cleanup. Next: Secure Software Supply Chain and DevSecOps.

Bash and Python Automation for AWS Operations

Bash and Python Automation for AWS Operations

Lesson Overview

Learning Objectives

Prerequisites

Detailed Core Concepts

Automation as an operations product

Bash versus Python versus SSM versus IaC

Credential rules (non-negotiable)

boto3 clients, paginators, errors

Structured logging and exit codes

Safe execution patterns

Inventory and compliance utility shape

Architecture and Design Diagrams

Technical Explanation

Example control flow (illustrative)

Bash companion

Pagination pitfall

Throttling

SSM for host facts

Cost note

Practical Implementation or Design Walkthrough

Problem

Steps

Enterprise Real-World Scenarios

Keys in git

One-page inventory

Mutate without dry-run

SSH for-each

Silent success on failure

Best Practices

Common Mistakes and Anti-Patterns

Troubleshooting and Design Review

Interview Questions and Answers

Hands-On Exercises

Exercise: Inventory and compliance reporting utility

Challenge ADR

Lab evidence checklist

Narrative: pagination saves the audit

Continuity

Extended practice scenario 1 — 60 bash and python automation for aws operations

Extended practice scenario 2 — 60 bash and python automation for aws operations

Extended practice scenario 3 — 60 bash and python automation for aws operations

Extended practice scenario 4 — 60 bash and python automation for aws operations

Extended practice scenario 5 — 60 bash and python automation for aws operations

Extended practice scenario 6 — 60 bash and python automation for aws operations

Extended practice scenario 7 — 60 bash and python automation for aws operations

Extended practice scenario 8 — 60 bash and python automation for aws operations

Extended practice scenario 9 — 60 bash and python automation for aws operations

Extended practice scenario 10 — 60 bash and python automation for aws operations

Extended practice scenario 11 — 60 bash and python automation for aws operations

Extended practice scenario 12 — 60 bash and python automation for aws operations

Extended practice scenario 13 — 60 bash and python automation for aws operations

Extended practice scenario 14 — 60 bash and python automation for aws operations

Extended practice scenario 15 — 60 bash and python automation for aws operations

Extended practice scenario 16 — 60 bash and python automation for aws operations

Extended practice scenario 17 — 60 bash and python automation for aws operations

Extended practice scenario 18 — 60 bash and python automation for aws operations

Extended practice scenario 19 — 60 bash and python automation for aws operations

Extended practice scenario 20 — 60 bash and python automation for aws operations

Extended practice scenario 21 — 60 bash and python automation for aws operations

Extended practice scenario 22 — 60 bash and python automation for aws operations

Extended practice scenario 23 — 60 bash and python automation for aws operations

Extended practice scenario 24 — 60 bash and python automation for aws operations

Extended practice scenario 25 — 60 bash and python automation for aws operations

Compliance evidence pack for auditors / platform leads

Glossary drill

Budget and cleanup sworn statement

Interview story to memorize

Self-assessment rubric

Final synthesis before summary

Extended facilitation script (print for guild leads)

Lesson Summary

Further Reading