AWS Cloud & DevOps Tutorial 0/70 lessons ~6 min read Lesson 37“IAM, VPC, EC2, Lambda, ECS, EKS, Terraform, CI/CD, and AWS operations”

AWS CloudTrail and Audit Logging

CloudWatch told TechLearningPro whether the progress API was failing. **AWS CloudTrail** answers a different operational and compliance question: **who did what, where, and when**

Course progress0%

Focus

15 guided sections

Practice signal

Examples included

Career prep

Interview Q&A included

AWS CloudTrail and Audit Logging

AWS Cloud & DevOps Engineering · Lesson 37 · Observability, Security and Reliability

Lesson Overview

CloudWatch told TechLearningPro whether the progress API was failing. AWS CloudTrail answers a different operational and compliance question: who did what, where, and when against the AWS APIs—and increasingly, selected data-plane events you deliberately enable. Audit logging is how you reconstruct a 2 a.m. IAM policy change, prove encryption settings were not silently weakened, and feed detective controls for regulated education workloads.

This lesson covers management versus data events, trail design (including organization trails), event history, log file integrity validation concepts, S3 destinations with encryption, and CloudTrail Lake query concepts for investigators—without inventing undocumented Lake SQL dialects. Pair with Amazon CloudWatch Metrics, Logs and Alarms for operational signals and with AWS Account Setup, IAM Identity Center and Organizations for multi-account thinking. Policy as Code and Infrastructure Compliance teased denying trail disablement via SCPs; here you understand what those trails must capture.

Budget: multi-region trails, data events, and Lake have cost dimensions—verify current pricing. Tag Project=tlp-lesson37. Prefer a dedicated audit bucket design even in labs. Clean up lab trails and Lake event data stores you create only for practice when the exercise ends (without deleting production org trails).

Learning Objectives

Explain why CloudTrail exists separately from application logs and CloudWatch metrics.
Distinguish management events from data events and when to enable each.
Describe single-account trails versus organization trails and who should own them.
Design an immutable-minded S3 destination: encryption, access control, lifecycle awareness.
Use Event History for recent investigations and trails for durable retention.
Explain log file integrity validation at a conceptual level and why it matters for evidence.
Outline CloudTrail Lake concepts for querying audit data without inventing SQL features.
Map detective controls: unauthorized API calls, trail stops, sensitive API usage.
Write an audit evidence plan for a regulated TechLearningPro account.
Avoid logging gaps from wrong regions, missing data events, or console-only assumptions.
Connect CloudTrail to IAM investigations and break-glass reviews.
Budget data events carefully (S3 object-level, Lambda invoke, etc., as documented).
Coordinate with SCPs that prevent disabling logging in production OUs.
Troubleshoot “empty trail” and delivery failures using documented status signals.
Deliver lab evidence without harming shared org logging.

Prerequisites

AWS Account Setup, IAM Identity Center and Organizations.
AWS Identity and Access Management.
Amazon CloudWatch Metrics, Logs and Alarms.
Storage Services: S3, EBS and EFS for bucket policy basics.
Budget alert; do not experiment by deleting organization trails in shared classrooms.

Detailed Core Concepts

CloudTrail versus application logs

Application logs say “progress save failed with ValidationError.” CloudTrail says “AssumedRole X called PutRolePolicy in account Y from source IP Z.” Both are necessary. Confusing them causes security teams to search CloudWatch Logs for IAM mutations that never appear there.

Event types

Management events record control-plane operations (CreateTrail, CreateFunction, AttachRolePolicy, PutBucketPolicy, and many others as documented). By default, trails commonly focus here.

Data events record resource data-plane operations you opt into (for example S3 object-level APIs, Lambda Invoke, DynamoDB item-level APIs—enable only the selectors you need). Data events can dominate cost at high QPS; TechLearningPro should enable them for sensitive buckets (identity photos, graded submissions) deliberately—not for every public marketing asset bucket.

Insight events (when enabled per current docs) highlight unusual API activity patterns—treat as an optional detective layer; verify behavior and pricing before relying on them in a regulated design.

Trails and Event History

Event history in the console provides a recent window of management events for quick lookups—useful, not a multi-year archive.

A trail delivers events to S3 (and optionally CloudWatch Logs) for durable retention and analytics. Trails can be region-scoped or apply to all regions supported by the feature set you configure—verify multi-region settings against current documentation so you do not miss activity in a secondary region used for DR drills.

Organization trails

In AWS Organizations, an organization trail can collect events from member accounts into a central audit account’s bucket. Ownership usually sits with a security/platform team—not each microservice squad. Workload engineers still need read paths for investigating their own account’s events. Lesson 48 deepens landing zones; today design the evidence plan assuming org trails exist or should exist.

Integrity and immutability mindset

CloudTrail can publish digest files for log file integrity validation, letting you detect tampering with delivered objects. Combine with S3 practices: deny public access, encrypt with KMS (lesson 41), restrict who can delete objects, consider Object Lock where your compliance program requires it and where you have verified the feature fit. “Immutable” is a design goal across IAM + bucket policy + org SCPs—not a single checkbox myth.

CloudTrail Lake (concepts)

CloudTrail Lake provides an auditable data store and query experience for CloudTrail events. Use it conceptually for investigator questions: “Which principals called Delete* on production KMS keys last month?” Do not memorize or invent proprietary SQL extensions here—practice against current AWS Lake query documentation when you lab it. Lake has its own cost model; sandbox carefully.

Detective controls narrative

Trail existence is not detection. Detection means: metric filters or EventBridge rules on critical management events, Security Hub/GuardDuty later lessons, periodic Lake or Athena-style queries (Athena over S3 trail logs is a common pattern—verify Glue/Athena setup against docs), and alerts when logging stops. Pair preventive SCPs (deny cloudtrail:StopLogging / DeleteTrail in prod OUs as an intent) with detective alarms on logging delivery failures.

Architecture and Design Diagrams

mermaid
flowchart TB
  AccA[Workload account A]
  AccB[Workload account B]
  OrgTrail[Organization CloudTrail]
  Bucket[Central audit S3 bucket]
  Lake[CloudTrail Lake optional]
  Detect[EventBridge / metric filters / queries]
  Sec[Security operations]

  AccA --> OrgTrail
  AccB --> OrgTrail
  OrgTrail --> Bucket
  OrgTrail --> Lake
  Bucket --> Detect --> Sec
  Lake --> Detect

Investigation path:

mermaid
sequenceDiagram
  participant Eng as Engineer
  participant IAM as IAM API
  participant CT as CloudTrail
  participant S3 as Audit bucket
  participant Inv as Investigator
  Eng->>IAM: PutRolePolicy
  IAM->>CT: management event
  CT->>S3: deliver JSON gzip
  Inv->>S3: retrieve / query
  Inv->>CT: Event History for recent

Technical Explanation

Anatomy of an event (conceptual fields)

Events typically include time, event name, event source, principal identity, source IP, user agent, region, request parameters, and response elements as recorded. Treat request/response contents carefully—they may include sensitive resource names. Redaction and access control on the audit store matter.

Selective data events for TechLearningPro

Example intents (configure with current advanced event selectors syntax from docs):

S3 data events on tlp-prod-submissions-* buckets.
Lambda Invoke data events only on functions that process grades or PII—if your threat model requires invoke attribution beyond management events.

Do not enable all S3 data events account-wide “just in case” without a budget conversation.

Delivery to CloudWatch Logs

Optional: deliver a subset to CloudWatch Logs for near-real-time metric filters (for example unauthorized API calls patterns using documented filter examples). Watch log ingestion cost. Prefer EventBridge integration patterns documented for CloudTrail when that fits your detection stack better.

Multi-account read patterns

Security account owns the bucket; use IAM roles for auditors. Avoid making the trail bucket writable by workload accounts beyond what CloudTrail service delivery requires. Bucket policies for CloudTrail delivery are documented—copy from current AWS guidance rather than inventing principals.

Separating duties

Engineers who can change IAM should not be the only people who can delete audit objects. Break-glass accounts must still be trailed. If someone can stop logging without detection, your evidence plan is theater.

Practical Implementation or Design Walkthrough

Problem

Create an audit evidence plan for a regulated TechLearningPro account: what is logged, where it lands, who can read/delete, how integrity is assured, how disablement is prevented/detected, and how investigators query the last 90 days of sensitive API use.

Steps

Budget alert; decide sandbox vs org-shared constraints.
Inventory: Event History available? Existing trail? Org trail?
Draft evidence plan document covering management events (all regions as applicable), data event selectors for sensitive resources, destination encryption, retention, integrity validation, access roles, detection of StopLogging, and query approach (Lake and/or S3 analytics).
In an isolated sandbox account only: create a trail to a dedicated bucket; enable integrity validation; encrypt with a lab CMK or AWS-managed key per your standard; tag resources; generate a few management events; locate them in Event History and in S3.
Optional: create a Lake event data store per current docs; run one documented-style query; note cost; tear down if lab-only.
Design (do not blindly apply in shared orgs) SCP intents denying trail deletion/stop in prod OUs.
ADR + cleanup of lab-only resources.

Acceptance evidence

Evidence plan Markdown, screenshots of trail/Event History, bucket encryption settings, integrity setting, query note or Athena/Lake screenshot if used, cleanup proof, tags.

Enterprise Real-World Scenarios

Trail only in one region

Attacker or mistaken automation operates elsewhere. Fix: multi-region / all-regions configuration as documented for your trail type.

Data events everywhere

Bill shock. Fix: selectors on sensitive ARNs only.

Engineers can empty the audit bucket

Evidence destroyed. Fix: restrictive bucket policy, separate admin role, Object Lock where required, SCP.

Org trail ignored by app teams

Nobody knows how to investigate. Fix: documented read role + runbook + training.

Relying only on Event History

Past retention window needed for audit. Fix: durable trail storage + lifecycle plan.

Best Practices

Prefer organization trails for enterprises.
Encrypt trail buckets; block public access.
Enable log file integrity validation for evidence-grade stores.
Scope data events to sensitive resources.
Detect StopLogging / DeleteTrail.
Separate duties on audit storage.
Document investigator query paths.
Tag audit infrastructure.
Budget before enabling high-volume data events.
Pair with SCPs in prod OUs.
Test that expected events appear after a known API call.
Retain per legal hold guidance—not vibes.
Do not dual-write conflicting trail designs without ownership.
Include break-glass activity in review rituals.
Clean up lab trails so they are not mistaken for production evidence.

Common Mistakes and Anti-Patterns

Assuming CloudWatch Logs replaces CloudTrail.
No trail—only Event History.
Public or weakly protected audit buckets.
Unscoped data events.
No detection when logging stops.
Same people admin IAM and delete audit objects with no oversight.
Inventing Lake SQL and teaching it as fact.
Deleting org trails during “cleanup.”
Ignoring management events from temporary credentials / roles.
Treating trail presence as compliance complete without reviews.

Troubleshooting and Design Review

Symptom	Cause	Action
Events missing	Wrong region; delay; data event not enabled	Check selectors; wait briefly; verify region
S3 delivery failing	Bucket policy / KMS permissions	Follow CloudTrail delivery troubleshooting docs
Huge bill	Broad data events	Narrow selectors; review pricing
Cannot query old events	No trail / lifecycle expired	Adjust retention design
Integrity validation fails	Object altered or incomplete digests	Investigate tampering or delivery gaps per docs
Org trail not visible to member	Expected centralization	Use delegated read path

Interview Questions and Answers

Q1: CloudTrail purpose? A: Record AWS API activity for audit, security, and ops investigation.
Q2: Management vs data events? A: Control plane vs resource data plane (opt-in).
Q3: Organization trail? A: Central trail across member accounts.
Q4: Event History vs trail? A: Recent console lookup vs durable delivery.
Q5: Integrity validation? A: Digests help detect tampering of delivered logs.
Q6: Why encrypt trail buckets? A: Audit data is sensitive; defense in depth.
Q7: CloudTrail Lake? A: Managed querying of audit events—verify current capabilities.
Q8: Cost surprise? A: Data events and Lake/storage—budget selectors.
Q9: Detective control example? A: Alert on trail stop or sensitive API calls.
Q10: Relation to SCPs? A: Prevent disablement; CloudTrail records attempts/related activity as applicable.
Q11: Who owns trails? A: Usually security/platform centrally.
Q12: Break-glass? A: Must still be audited and reviewed.
Q13: Athena pattern? A: Query S3 trail logs with a documented schema approach.
Q14: Multi-region why? A: Activity can occur outside your “primary” region.
Q15: Evidence plan? A: What/where/who/how long/how query/how protect.

Hands-On Exercises

Exercise: Audit evidence plan for a regulated account

Produce the plan plus sandbox trail evidence. Do not disrupt shared organization logging.

Challenge ADR

Finance asks to turn off CloudTrail in staging to save money. Argue for retaining management event trails in all accounts that can reach production data or IAM, with cheaper data-event choices.

Lab evidence checklist

Plan doc, trail config, bucket encryption, integrity flag, sample event, optional Lake note, cleanup, tags Project=tlp-lesson37.

Narrative: 2 a.m. policy change

A widened IAM policy shipped. CloudWatch showed errors later; CloudTrail showed which role changed the policy and from where. Without trails, you only have rumors.

Continuity

Next: AWS X-Ray and Distributed Tracing—from who changed AWS, to why a single learner request was slow across services.

Building a sensitive API watchlist

List management APIs that deserve near-real-time attention for TechLearningPro: IAM policy changes, KMS schedule key deletion, trail modifications, security group ingress changes, S3 public access block removals. Map each to a detection mechanism you can actually implement with documented services.

Retention and legal hold sketch

Define hot retention on S3, transition classes if used, and who can place holds. Align with school/enterprise records policy. Verify Glacier/lifecycle behavior against current S3 docs before promising auditors a retrieval SLA.

Cross-account investigation drill

Role-play: security account analyst assumes a read-only role into audit storage, finds PutBucketPolicy on submissions bucket, traces principal to a broken automation role, opens incident with lesson 40’s lifecycle.

SCP intent appendix

Write deny statements intents (not necessarily final JSON) for StopLogging, DeleteTrail, and DeleteBucket on audit buckets for prod OUs; note sandbox exceptions. Full SCP design returns in advanced modules.

CloudTrail and CI

Pipeline roles create resources—expect dense management event volume during deploys. Do not alert on every Create*; alert on anomalous identity or sensitive mutations. Baseline before paging.

Extended practice scenario 1 — lesson 37 CloudTrail audit logging

In a timed drill, implement or design the core lab for lesson 37 (CloudTrail audit logging) with a partner observing. The observer’s job is to stop any step that invents API flags, hardcodes secrets, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: commands run, screenshots of tagged resources, policy or plan output, and destroy confirmation.

Discuss how this lesson’s tooling interacts with IAM least privilege, VPC private placement, and CI promotion rules from earlier modules. List two failure modes you personally witnessed or simulated, including detection and fix. Estimate cost drivers you touched and how long resources remained before teardown.

Focus this iteration on: organization trail ownership and member-account read path design

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps in the teach-back become your flashcards. Premium courses are not finished when the Markdown ends—they are finished when you can operate the ideas under time pressure without opening the console as a first resort.

Extended practice scenario 2 — lesson 37 CloudTrail audit logging

Focus this iteration on: data event selector scoping for submissions buckets only

Extended practice scenario 3 — lesson 37 CloudTrail audit logging

Focus this iteration on: log file integrity validation and bucket policy review

Extended practice scenario 4 — lesson 37 CloudTrail audit logging

Focus this iteration on: StopLogging detection design with EventBridge or metric filters

Extended practice scenario 5 — lesson 37 CloudTrail audit logging

Focus this iteration on: CloudTrail Lake conceptual query for sensitive API usage

Extended practice scenario 6 — lesson 37 CloudTrail audit logging

Focus this iteration on: audit evidence plan for a regulated education account

Extended practice scenario 7 — lesson 37 CloudTrail audit logging

Focus this iteration on: multi-region trail coverage verification checklist

Extended practice scenario 8 — lesson 37 CloudTrail audit logging

Focus this iteration on: break-glass activity review using Event History and trail files

Extended practice scenario 9 — lesson 37 CloudTrail audit logging

Focus this iteration on: organization trail ownership and member-account read path design

Extended practice scenario 10 — lesson 37 CloudTrail audit logging

Focus this iteration on: data event selector scoping for submissions buckets only

Extended practice scenario 11 — lesson 37 CloudTrail audit logging

Focus this iteration on: log file integrity validation and bucket policy review

Extended practice scenario 12 — lesson 37 CloudTrail audit logging

Focus this iteration on: StopLogging detection design with EventBridge or metric filters

Extended practice scenario 13 — lesson 37 CloudTrail audit logging

Focus this iteration on: CloudTrail Lake conceptual query for sensitive API usage

Extended practice scenario 14 — lesson 37 CloudTrail audit logging

Focus this iteration on: audit evidence plan for a regulated education account

Extended practice scenario 15 — lesson 37 CloudTrail audit logging

Focus this iteration on: multi-region trail coverage verification checklist

Extended practice scenario 16 — lesson 37 CloudTrail audit logging

Focus this iteration on: break-glass activity review using Event History and trail files

Extended practice scenario 17 — lesson 37 CloudTrail audit logging

Focus this iteration on: organization trail ownership and member-account read path design

Extended practice scenario 18 — lesson 37 CloudTrail audit logging

Focus this iteration on: data event selector scoping for submissions buckets only

Extended practice scenario 19 — lesson 37 CloudTrail audit logging

Focus this iteration on: log file integrity validation and bucket policy review

Extended practice scenario 20 — lesson 37 CloudTrail audit logging

Focus this iteration on: StopLogging detection design with EventBridge or metric filters

Extended practice scenario 21 — lesson 37 CloudTrail audit logging

Focus this iteration on: CloudTrail Lake conceptual query for sensitive API usage

Extended practice scenario 22 — lesson 37 CloudTrail audit logging

Focus this iteration on: audit evidence plan for a regulated education account

Evidence plan template deep dive

Expand your regulated-account plan with explicit sections: scope of accounts and regions; management event coverage statement; data event ARN patterns; destination bucket ARN and encryption key ARN ownership; integrity validation on/off with rationale; retention days and legal hold process owner; investigator IAM role names; detection for trail modification APIs; quarterly access review ritual; and a table mapping TechLearningPro threat stories (grade tampering, PII exfiltration attempts via API, break-glass misuse) to the CloudTrail fields you would inspect first.

Include a “non-goals” list: CloudTrail will not replace application audit logs for quiz answer changes inside DynamoDB items unless you enable appropriate data events and application-level auditing. Saying non-goals prevents false auditor confidence.

Delivery failure runbook sketch

When CloudTrail shows insufficient delivery or operators notice gaps: check trail status in console, inspect destination bucket policy and KMS key policy for the CloudTrail service principal patterns documented by AWS, verify the bucket still exists and is in the expected region/account, confirm no SCP unexpectedly denies required actions, and open an incident if production evidence is at risk. Record times of gaps for compliance disclosure if required.

Classroom safety note

In shared Organization classrooms, never delete the organization trail or empty the central audit bucket as “cleanup.” Lab cleanup means removing only resources you created in an isolated sandbox account listed on your evidence sheet.

Lesson Summary

CloudTrail is TechLearningPro’s AWS-layer audit tape: management events everywhere they matter, data events where sensitivity justifies cost, organization trails into protected storage, integrity and access controls for evidence grade, Lake/query paths for investigators, and detective controls when logging stops. Write the evidence plan before the auditor arrives.

Next: AWS X-Ray and Distributed Tracing.

AWS CloudTrail and Audit Logging

AWS CloudTrail and Audit Logging

Lesson Overview

Learning Objectives

Prerequisites

Detailed Core Concepts

CloudTrail versus application logs

Event types

Trails and Event History

Organization trails

Integrity and immutability mindset

CloudTrail Lake (concepts)

Detective controls narrative

Architecture and Design Diagrams

Technical Explanation

Anatomy of an event (conceptual fields)

Selective data events for TechLearningPro

Delivery to CloudWatch Logs

Multi-account read patterns

Separating duties

Practical Implementation or Design Walkthrough

Problem

Steps

Acceptance evidence

Enterprise Real-World Scenarios

Trail only in one region

Data events everywhere

Engineers can empty the audit bucket

Org trail ignored by app teams

Relying only on Event History

Best Practices

Common Mistakes and Anti-Patterns

Troubleshooting and Design Review

Interview Questions and Answers

Hands-On Exercises

Exercise: Audit evidence plan for a regulated account

Challenge ADR

Lab evidence checklist

Narrative: 2 a.m. policy change

Continuity

Building a sensitive API watchlist

Retention and legal hold sketch

Cross-account investigation drill

SCP intent appendix

CloudTrail and CI

Extended practice scenario 1 — lesson 37 CloudTrail audit logging

Extended practice scenario 2 — lesson 37 CloudTrail audit logging

Extended practice scenario 3 — lesson 37 CloudTrail audit logging

Extended practice scenario 4 — lesson 37 CloudTrail audit logging

Extended practice scenario 5 — lesson 37 CloudTrail audit logging

Extended practice scenario 6 — lesson 37 CloudTrail audit logging

Extended practice scenario 7 — lesson 37 CloudTrail audit logging

Extended practice scenario 8 — lesson 37 CloudTrail audit logging

Extended practice scenario 9 — lesson 37 CloudTrail audit logging

Extended practice scenario 10 — lesson 37 CloudTrail audit logging

Extended practice scenario 11 — lesson 37 CloudTrail audit logging

Extended practice scenario 12 — lesson 37 CloudTrail audit logging

Extended practice scenario 13 — lesson 37 CloudTrail audit logging

Extended practice scenario 14 — lesson 37 CloudTrail audit logging

Extended practice scenario 15 — lesson 37 CloudTrail audit logging

Extended practice scenario 16 — lesson 37 CloudTrail audit logging

Extended practice scenario 17 — lesson 37 CloudTrail audit logging

Extended practice scenario 18 — lesson 37 CloudTrail audit logging

Extended practice scenario 19 — lesson 37 CloudTrail audit logging

Extended practice scenario 20 — lesson 37 CloudTrail audit logging

Extended practice scenario 21 — lesson 37 CloudTrail audit logging

Extended practice scenario 22 — lesson 37 CloudTrail audit logging

Evidence plan template deep dive

Delivery failure runbook sketch

Classroom safety note

Lesson Summary

Further Reading