AWS Cloud & DevOps Tutorial 0/70 lessons ~6 min read Lesson 55“IAM, VPC, EC2, Lambda, ECS, EKS, Terraform, CI/CD, and AWS operations”

AWS Glue, Athena and Lake Formation

TechLearningPro’s product and finance leaders ask the same questions every Monday: completion rates by course, exam pass rates by cohort, and content engagement by lesson. The answ

Course progress0%

Focus

15 guided sections

Practice signal

Examples included

Career prep

Interview Q&A included

AWS Glue, Athena and Lake Formation

AWS Cloud & DevOps Engineering · Lesson 55 · Advanced Platforms and Networking

Lesson Overview

TechLearningPro’s product and finance leaders ask the same questions every Monday: completion rates by course, exam pass rates by cohort, and content engagement by lesson. The answers live as files on the hardened lake from Amazon S3 Advanced Design and Data Protection—but files alone are not a governed analytics platform. AWS Glue provides cataloging and ETL job primitives; Amazon Athena provides serverless SQL over S3; AWS Lake Formation provides governed permissions on Data Catalog resources. This lesson teaches the platform and operations framing: how Cloud/DevOps engineers stand up catalog boundaries, job IAM roles, query cost controls, and permission models so data engineers can ship pipelines safely.

Deep Spark job design, dimensional modeling, and lakehouse table formats belong in the Data Engineering course—especially AWS Data Engineering Services, Data Lake Architecture, Data Lakehouse Architecture, and Data Platform Security and Governance. Here you own the shared AWS surface those teams consume. Tag Project=tlp-lesson55. Budget before crawlers, Glue jobs (DPU-hours), and Athena scans; partition projection and columnar formats are cost controls, not optional polish. Clean up lab databases, jobs, and sample data when finished.

Learning Objectives

Explain Glue Data Catalog as the metastore for tables over S3.
Describe crawlers versus explicit table definitions and when each fits.
Outline Glue ETL jobs / Spark jobs at an ops level (roles, networking, bookmarks).
Run Athena queries with workgroups and result locations.
Control Athena cost via partitions, file formats, and workgroup bytes-scanned limits.
Explain Lake Formation permission model versus raw IAM-on-S3 for analytics users.
Design LF-TBAC / named resource grants concepts without inventing grant APIs.
Separate platform account duties from domain data-producer duties.
Integrate encryption (S3 + KMS) with catalog and query paths.
Observe jobs and queries with CloudWatch and job run history.
Troubleshoot AccessDenied across S3, KMS, Glue, Athena, and Lake Formation.
Compare Athena vs Redshift for TechLearningPro serving patterns (preview lesson 56).
Tag and showback analytics spend (FinOps lesson 47 continuity).
Write an ADR for ingestion-and-query architecture for analytical datasets.
Avoid turning this course into a full DE curriculum—link out instead.
Secure Athena results buckets (often forgotten attack surface).
Plan VPC connectivity for Glue when jobs need private resources.
Prepare interview answers on catalog, governance, and scan cost.
Map controls to Well-Architected Security and Cost pillars.
Clean up crawlers/jobs/databases after labs.

Prerequisites

S3 advanced lesson 54 concepts; IAM fluency; budgets enabled.
Optional familiarity with SQL SELECT and Parquet/ORC ideas.
Sandbox permissions for Glue, Athena, Lake Formation (admin may need to enable LF).
Access to /learn/data-engineering for pipeline depth.

Detailed Core Concepts

Glue Data Catalog

Central metastore: databases, tables, partitions, schemas. Athena, Glue jobs, Redshift Spectrum, and EMR can share catalog definitions. Treat catalog changes as production changes—breaking schema without contracts harms Monday boards. Platform owns backup/export strategy for catalog where required; producers own table correctness.

Crawlers vs explicit DDL

Crawlers infer schema from S3 data; useful for exploration, dangerous as sole production source of truth when inference is wrong. Prefer explicit table definitions or IaC-managed schemas for TechLearningPro curated metrics. Schedule crawlers narrowly on raw zones if used.

Glue ETL / jobs (ops view)

Jobs run with IAM roles, may need VPC ENIs for private access, consume DPU capacity (or Flex/worker types per current docs), support job bookmarks for incremental processing concepts, and emit CloudWatch logs/metrics. Platform engineers size concurrency, set timeouts, route logs, and prevent orphaned connections. Pipeline code quality is a DE ownership boundary.

Athena

Serverless Presto/Trino-based query engine (engine versions evolve—check docs) that scans data in S3 per table metadata. You pay primarily by data scanned (verify pricing). Workgroups isolate users, enforce result locations, and can apply data usage controls. Always set a results bucket with encryption and lifecycle.

Lake Formation

Lake Formation centralizes permissions on Data Catalog databases/tables/columns (and cell-level concepts where offered) and can mediate access so analysts do not need broad s3:* on the lake. Hybrid access modes and IAMAllowedPrincipals behavior require careful migration—read current LF documentation before flipping production. Column filters protect PII in learner tables.

Governance split

Concern	Platform (this course)	Data Engineering course
Bucket encryption, Block Public Access	Yes	Consumes
Catalog IAM roles, LF admin setup	Yes	Uses grants
Job IAM, VPC, observability	Yes	Job logic
Spark transforms, SCD, quality tests	Link out	Deep coverage
Metric definitions / contracts	Collaborate	Own with product

Cost meters

Glue: DPU-hours, storage for temporary data, crawler runs. Athena: bytes scanned, optionally provisioned capacity where offered. Lake Formation: generally permission plane (confirm current pricing). S3: underlying storage/requests. FinOps tags on workgroups and jobs matter.

Security

Least privilege job roles; separate read-only analyst roles; encrypt results; CloudTrail for Glue/Lake Formation APIs; deny public Athena results; careful with IAMAllowedPrincipals during LF migration. Continuity with lessons 41 and 54.

Architecture and Design Diagrams

mermaid
flowchart TB
  Prod[App / event producers]
  S3Raw[S3 raw/]
  S3Cur[S3 curated/]
  GlueJob[Glue jobs]
  Cat[Glue Data Catalog]
  LF[Lake Formation permissions]
  Athena[Athena workgroups]
  Analysts[Analysts / BI]
  DE[Data engineers]

  Prod --> S3Raw
  S3Raw --> GlueJob --> S3Cur
  GlueJob --> Cat
  S3Cur --> Cat
  LF --> Cat
  LF --> Analysts
  Cat --> Athena
  S3Cur --> Athena
  Athena --> Analysts
  DE --> GlueJob

mermaid
flowchart LR
  WG[Athena workgroup]
  Limit[Bytes scanned controls]
  Results[Encrypted results bucket]
  WG --> Limit
  WG --> Results

Technical Explanation

Query path

Analyst submits SQL in Athena workgroup → engine plans scan using Catalog partitions → reads Parquet/ORC from S3 with credentials mediated by Lake Formation or IAM → writes results to workgroup results location → BI tool reads results. Missing partitions cause full-prefix scans and bill shock.

Why partitions and columnar formats matter

WHERE event_date = '2026-11-11' only helps if tables are partitioned by date and files are columnar. Platform can enforce “no CSV in curated” via guidelines and Config-like reviews; DE implements compaction. Link Data Platform Cost and Performance Optimization.

Glue job failure modes (ops)

OOM, permission errors to S3/KMS, VPC route/DNS failures to JDBC sources, bookmark stuck after bad deploy, concurrent runs stomping outputs. Alarms on failed job runs; DLQ/event patterns for orchestration with Step Functions (lesson 23 continuity) without rewriting the DE DAG course.

Lake Formation mental model

Grant SELECT on analytics.completion_daily to role tlp-analyst instead of bucket-wide list/get. Column security hides email. When LF is not yet enabled, teams over-share S3 prefixes—migration is a project, not a checkbox.

Multi-account

Shared services account hosts Catalog centralization patterns (or per-account catalogs with cross-account shares—verify Resource Links / LF cross-account docs). Producers in workload accounts write to their prefixes; consumers query via grants. Align with Organizations lessons 48–49.

Athena vs ad-hoc S3 Select

S3 Select is not a warehouse. Athena + Catalog is the interactive SQL plane for lake exploration; Redshift (lesson 56) serves repeated low-latency BI metrics.

Practical Implementation or Design Walkthrough

Problem

Design ingestion and query architecture for TechLearningPro analytical datasets: raw progress events → curated daily completion table → Athena workgroup for analysts with Lake Formation-style grants (design even if LF lab is limited).

Steps

Budget + tags Project=tlp-lesson55.
Reuse or design S3 raw/curated prefixes from lesson 54.
Define Catalog database tlp_analytics and a sample table (explicit DDL preferred).
Design Glue job role policies (S3 + KMS + CloudWatch logs) and VPC needs.
Create Athena workgroup with enforced results bucket + optional bytes limit.
Run a sample query; show partition pruning intent.
Negative test: deny unpartitioned SELECT * on large table via process/workgroup policy narrative.
Sketch Lake Formation grants for analyst vs engineer roles.
ADR separating platform vs DE ownership; link DE lessons.
Cleanup jobs, databases (if lab), and sample objects.

Enterprise Real-World Scenarios

Monday board bill shock

Analyst runs SELECT * on two years of JSON. Athena scans terabytes. Fix: workgroup limit, Parquet curated table, enforced partition filter guideline, showback to the analyst’s cost center.

Crawler overwrites curated schema

Nightly crawler “helps” and widens types; dashboards break. Fix: stop crawler on curated; schema PRs; contracts from DE course.

Glue job in public subnet

Job needs JDBC to Aurora; security group wide open. Fix: private subnets, SG least privilege, Secrets Manager for creds (lesson 21).

LF hybrid confusion

Some users still IAM-allowed; others LF-only; AccessDenied chaos. Fix: staged migration runbook; test matrix before cutting IAMAllowedPrincipals.

Results bucket public-ish

Misconfigured results ACL exposes query output with PII. Fix: Block Public Access, bucket policy, encryption, lifecycle expire results in 7–30 days (example—set per policy).

Best Practices

Explicit curated schemas over blind crawlers.
Athena workgroups per team with results isolation.
Bytes-scanned limits in non-prod and exploratory WG.
Columnar formats in curated zones.
Partition by high-cardinality-aware keys (often date).
Least-privilege Glue job roles.
Encrypted results buckets with lifecycle.
Lake Formation for analyst access at scale.
CloudWatch alarms on Glue job failures.
Tag jobs/workgroups for FinOps.
Document platform vs DE RACI.
Test AccessDenied paths intentionally.
Avoid SELECT * in shared examples.
Version job scripts in Git; no console-only production.
Link DE course for pipeline depth—do not fork curricula.

Common Mistakes and Anti-Patterns

Teaching full Spark optimization inside this CloudOps lesson.
Shared admin workgroup for everyone.
Unbounded Athena in production accounts.
Catalog as unowned global dump.
Granting s3:* to analysts “temporarily.”
Ignoring Glue VPC ENI leftover costs after bad deletes.
Storing results indefinitely.
Inventing Lake Formation grant syntax in ADRs without docs.
Mixing raw and curated in one table definition.
No showback → surprise finance meetings.

Troubleshooting and Design Review

Symptom	Cause	Action
Athena AccessDenied	IAM/LF/S3/KMS	Trace each layer; check LF grants
Huge Athena bill	Unpartitioned scan / JSON	Curate Parquet; WG limits
Glue fails on start	Role/VPC/subnet	Check ENI, SG, route tables
Empty partitions	Writer path mismatch	Align S3 keys with partition keys
Schema drift	Crawler or bad deploy	Lock curated DDL; CI checks
Query slow	Small files / skew	Compaction (DE) + platform guidance

Interview Questions and Answers

Q1: Glue Data Catalog? A: Metastore for databases/tables/partitions used by analytics services.
Q2: Athena pricing driver? A: Primarily data scanned—verify current pricing.
Q3: Why workgroups? A: Isolation, results config, cost controls.
Q4: Lake Formation purpose? A: Governed permissions on catalog data for lake access.
Q5: Crawler risk? A: Incorrect inferred schemas in production.
Q6: Platform vs DE? A: Platform owns shared AWS ops/governance; DE owns pipeline logic and models.
Q7: Partition pruning? A: Restricting scanned data via partition predicates.
Q8: Glue job IAM? A: Role needing S3/KMS/logs—least privilege.
Q9: Results bucket risk? A: Query output may contain sensitive data.
Q10: When Redshift instead? A: Repeated low-latency BI / warehouse serving (lesson 56).
Q11: How control Athena cost? A: Formats, partitions, WG limits, avoid SELECT *.
Q12: Cross-account catalog? A: Resource links / LF sharing—confirm current docs.
Q13: Job bookmarks? A: Incremental processing concept—verify behavior in docs.
Q14: PII in lake? A: Column filters, encryption, contracts, minimize raw copies.
Q15: First FinOps step? A: Tag + WG showback + bytes alarms.

Hands-On Exercises

Exercise: Ingestion and query architecture

Deliverable: diagram, Catalog table design, Glue role sketch, Athena workgroup config, LF grant matrix, ADR with DE cross-links, cleanup.

Challenge ADR

Multi-account: producers in app accounts, shared analytics account with LF. Justify Catalog placement.

Lab evidence checklist

Budget; workgroup; sample query evidence; deny/limit test; cleanup.

Narrative: SELECT * postmortem

Finance questions a spike; Athena query history shows unpartitioned scan; WG limit added same day; curated Parquet backlog prioritized with DE.

Continuity

Prior: lesson 54 S3. Next: Amazon Redshift and Analytics Architecture. Depth: /learn/data-engineering/aws-data-engineering-services.

Deep dive: Glue Data Quality / crawlers scope

Mention quality checks as available features to explore in docs; do not invent rule DSLs—send deep testing to Data Quality Engineering.

Deep dive: Iceberg/Hudi/Delta on AWS

Table formats may integrate with Glue Catalog—confirm current support matrices; deep ops in DE lakehouse lessons.

Deep dive: Cost anomaly for analytics

Wire Cost Anomaly Detection (lesson 47) to Glue/Athena service filters for TechLearningPro analytics accounts.

Extended enterprise scenario: exam season concurrency

Hundreds of analysts refresh dashboards; Athena queues; platform adds WG capacity strategy or offloads hot aggregates to Redshift (lesson 56).

Extended enterprise scenario: GDPR delete

Learner erasure requires coordinated S3 delete + catalog partitions + downstream warehouses—platform runbook + DE ownership; LF permissions alone are insufficient.

Extended practice scenario 1 — 55 aws glue athena and lake formation

Confirm AWS Budgets alerts exist for the sandbox; tag every resource Project=tlp-lesson55. Do not proceed without a budget subscriber.

In a timed drill for Glue, Athena, and Lake Formation platform ops, implement or design the core lab with a partner observing. The observer's job is to stop any step that invents API flags, hardcodes secrets, invents pricing numbers presented as current quotes, or skips budget and cleanup. After the drill, rewrite your ADR in ten lines: context, decision, consequences. Attach evidence: workgroup settings, IAM role sketches, Catalog DDL, LF grant matrix, query screenshots (redact data), and destroy confirmation for any lab resources.

Discuss how this platform layer interacts with S3 (lesson 54), FinOps (lesson 47), KMS (lesson 41), and deep pipelines at /learn/data-engineering/aws-data-engineering-services and /learn/data-engineering/data-lake-architecture. List two failure modes you simulated or observed, including detection and fix.

Finally, teach back the lesson overview paragraph from memory, then check against the text. Gaps become flashcards. Premium courses finish when you can operate under time pressure without treating Athena scan cost and Lake Formation grants as a mystery.

Extended practice scenario 2 — 55 aws glue athena and lake formation

Document the lesson's primary architecture decision in a ten-line ADR: context, decision, status, consequences.

Extended practice scenario 3 — 55 aws glue athena and lake formation

Map identity paths: who can create/modify the resources in this lesson; which IAM roles are least privilege.

Extended practice scenario 4 — 55 aws glue athena and lake formation

Design observability: CloudWatch metrics/alarms or equivalent signals that prove the control works.

Extended practice scenario 5 — 55 aws glue athena and lake formation

Run or design a negative test: denied access, misconfiguration, or failed dependency—capture evidence.

Extended practice scenario 6 — 55 aws glue athena and lake formation

Estimate cost dimensions for this lesson's lab (storage, requests, compute hours, data transfer, DPU/query bytes as applicable). Label figures illustrative; verify pricing pages with dates.

Extended practice scenario 7 — 55 aws glue athena and lake formation

Produce a cleanup checklist and execute destroy for any billable lab resources created for lesson 55.

Extended practice scenario 8 — 55 aws glue athena and lake formation

Cross-link prior course lessons that this design depends on; list two failure modes inherited from those layers.

Extended practice scenario 9 — 55 aws glue athena and lake formation

Security review: encryption at rest/in transit, public exposure checks, CloudTrail visibility for management events.

Extended practice scenario 10 — 55 aws glue athena and lake formation

Reliability review: what breaks if one AZ/Region/service fails; how detection and recovery work.

Extended practice scenario 11 — 55 aws glue athena and lake formation

Performance review: hot partitions, concurrency limits, throttling, or query/scan patterns relevant to Glue, Athena, and Lake Formation platform ops.

Extended practice scenario 12 — 55 aws glue athena and lake formation

Write a runbook section for on-call: symptoms, first three diagnostic steps, escalation, rollback.

Extended practice scenario 13 — 55 aws glue athena and lake formation

Config/compliance angle: which AWS Config rules or preventive controls would catch misconfiguration.

Extended practice scenario 14 — 55 aws glue athena and lake formation

Multi-account consideration: how this pattern changes across workload accounts vs shared platform accounts.

Extended practice scenario 15 — 55 aws glue athena and lake formation

Interview drill: explain Glue, Athena, and Lake Formation platform ops to a hiring manager in three minutes without inventing APIs.

Extended practice scenario 16 — 55 aws glue athena and lake formation

Pair review: partner stops any invented CLI flags, hardcoded secrets, or fake current prices.

Extended practice scenario 17 — 55 aws glue athena and lake formation

FinOps continuity from lesson 47: allocate costs with tags; note idle risk if labs are left running.

Extended practice scenario 18 — 55 aws glue athena and lake formation

Disaster-recovery continuity from lesson 45: how backups/replication affect RPO/RTO for this stack.

Extended practice scenario 19 — 55 aws glue athena and lake formation

Produce evidence pack: diagrams, config excerpts (redact secrets), test results, cleanup proof.

Extended practice scenario 20 — 55 aws glue athena and lake formation

Challenge constraint: tighter monthly budget—what do you remove first without breaking the security bar?

Extended practice scenario 21 — 55 aws glue athena and lake formation

Challenge constraint: stricter compliance (governed access / audit)—what changes in the design?

Extended practice scenario 22 — 55 aws glue athena and lake formation

Challenge constraint: higher traffic or larger datasets—which performance pattern do you apply first?

Extended practice scenario 23 — 55 aws glue athena and lake formation

Teach-back: recite the Lesson Overview paragraph, then check against the text; flashcard gaps.

Extended practice scenario 24 — 55 aws glue athena and lake formation

Dress rehearsal: end-to-end lab or design walkthrough under a timer with observer notes.

Extended practice scenario 25 — 55 aws glue athena and lake formation

Final synthesis note: one paragraph connecting Glue, Athena, and Lake Formation platform ops to TechLearningPro production ownership.

Compliance evidence pack for auditors / platform leads

Budget configs, tagging proof (Project=tlp-lesson55), architecture diagram, ADR, negative-test evidence, encryption settings notes, CloudTrail/Config references where applicable, pricing-page retrieval dates for any dollar claims, cleanup confirmation.

Glossary drill

Review every bold term introduced in Detailed Core Concepts and Interview Q&A. Write one sentence definitions from memory; correct against the lesson. Unfamiliar AWS console navigation becomes a lab note, not a guess.

Budget and cleanup sworn statement

I verified budgets remain enabled with alert subscribers. I deleted or scheduled deletion for orphan lab resources from lesson 55 named ____. I labeled every savings or cost figure with pricing-page retrieval dates or marked it illustrative. Signed ________ date ________.

Interview story to memorize

Craft a 60-second story: TechLearningPro needed Glue, Athena, and Lake Formation platform ops; you chose a design with explicit trade-offs; you proved it with tests; you governed cost with budgets and tags; you left an ADR so the next engineer does not reverse the decision blindly.

Self-assessment rubric

Area	Weight
Conceptual accuracy and service selection	20%
Working design/lab and reproducibility	25%
Security and least-privilege	20%
Reliability, observability, troubleshooting	20%
Documentation, ADR, cleanup	15%

Final synthesis before summary

Mastery of Glue, Athena, and Lake Formation platform ops is not logo recognition—it is owning failure modes, cost meters, identity boundaries, and the written decision that keeps TechLearningPro coherent under change.

Lesson Summary

Glue, Athena, and Lake Formation give TechLearningPro a governed lake query platform: Catalog as metastore, Glue jobs as operational ETL compute, Athena workgroups as cost-controlled SQL, and Lake Formation as the permission plane for analysts. Platform engineers own IAM, networking, encryption, workgroups, and showback; data engineers own pipeline logic and models in /learn/data-engineering. Tag Project=tlp-lesson55; budget before DPU-hours and scans; clean up lab jobs and databases.

Next: Amazon Redshift and Analytics Architecture for warehouse serving of Monday-board metrics. Deep services: AWS Data Engineering Services.

AWS Glue, Athena and Lake Formation

AWS Glue, Athena and Lake Formation

Lesson Overview

Learning Objectives

Prerequisites

Detailed Core Concepts

Glue Data Catalog

Crawlers vs explicit DDL

Glue ETL / jobs (ops view)

Athena

Lake Formation

Governance split

Cost meters

Security

Architecture and Design Diagrams

Technical Explanation

Query path

Why partitions and columnar formats matter

Glue job failure modes (ops)

Lake Formation mental model

Multi-account

Athena vs ad-hoc S3 Select

Practical Implementation or Design Walkthrough

Problem

Steps

Enterprise Real-World Scenarios

Monday board bill shock

Crawler overwrites curated schema

Glue job in public subnet

LF hybrid confusion

Results bucket public-ish

Best Practices

Common Mistakes and Anti-Patterns

Troubleshooting and Design Review

Interview Questions and Answers

Hands-On Exercises

Exercise: Ingestion and query architecture

Challenge ADR

Lab evidence checklist

Narrative: SELECT * postmortem

Continuity

Deep dive: Glue Data Quality / crawlers scope

Deep dive: Iceberg/Hudi/Delta on AWS

Deep dive: Cost anomaly for analytics

Extended enterprise scenario: exam season concurrency

Extended enterprise scenario: GDPR delete

Extended practice scenario 1 — 55 aws glue athena and lake formation

Extended practice scenario 2 — 55 aws glue athena and lake formation

Extended practice scenario 3 — 55 aws glue athena and lake formation

Extended practice scenario 4 — 55 aws glue athena and lake formation

Extended practice scenario 5 — 55 aws glue athena and lake formation

Extended practice scenario 6 — 55 aws glue athena and lake formation

Extended practice scenario 7 — 55 aws glue athena and lake formation

Extended practice scenario 8 — 55 aws glue athena and lake formation

Extended practice scenario 9 — 55 aws glue athena and lake formation

Extended practice scenario 10 — 55 aws glue athena and lake formation

Extended practice scenario 11 — 55 aws glue athena and lake formation

Extended practice scenario 12 — 55 aws glue athena and lake formation

Extended practice scenario 13 — 55 aws glue athena and lake formation

Extended practice scenario 14 — 55 aws glue athena and lake formation

Extended practice scenario 15 — 55 aws glue athena and lake formation

Extended practice scenario 16 — 55 aws glue athena and lake formation

Extended practice scenario 17 — 55 aws glue athena and lake formation

Extended practice scenario 18 — 55 aws glue athena and lake formation

Extended practice scenario 19 — 55 aws glue athena and lake formation

Extended practice scenario 20 — 55 aws glue athena and lake formation

Extended practice scenario 21 — 55 aws glue athena and lake formation

Extended practice scenario 22 — 55 aws glue athena and lake formation

Extended practice scenario 23 — 55 aws glue athena and lake formation

Extended practice scenario 24 — 55 aws glue athena and lake formation

Extended practice scenario 25 — 55 aws glue athena and lake formation

Compliance evidence pack for auditors / platform leads

Glossary drill

Budget and cleanup sworn statement

Interview story to memorize

Self-assessment rubric

Final synthesis before summary

Lesson Summary

Further Reading