Amazon EKS and Kubernetes on AWS

AWS Cloud & DevOps Engineering · Lesson 18 · AWS Application Platform

Lesson Overview

The progress API and enrollment service do not become “Kubernetes” because a slide deck said containers were inevitable. They become Kubernetes when TechLearningPro needs a portable control plane for many services, custom controllers, multi-team namespaces, or a platform shape that outgrows the opinionated comfort of ECS. Amazon EKS is AWS’s managed Kubernetes offering: AWS runs the Kubernetes control plane for you; you own the data plane choices—managed node groups, self-managed nodes, or Fargate profiles—and you own the day-two reality of networking, add-ons, access, and cost.

Containers with Docker for AWS Workloads taught images and processes. Amazon ECS and AWS Fargate taught an AWS-native orchestrator with a smaller surface. This lesson places EKS in that decision space without cloning the full Kubernetes course. You will learn what AWS manages, what you still operate, how pods get network and identity at a high level, and how to sketch a multi-service progress and enrollment path with ingress and autoscaling concepts—then delete the cluster before the budget alert has to shout.

Deep Kubernetes primitives—Deployments, Services, Controllers, RBAC theory, Operators—live in /learn/kubernetes. Here the focus is AWS-shaped: control plane versus data plane, VPC CNI awareness, add-ons, cluster access history and modern access entries, an IRSA tease for pod identity, and an honest ECS versus EKS framework for TechLearningPro. The next lesson, Amazon ECR and Container Image Supply Chain, is where those images live and how you promote them.

Learning Objectives

By the end of this lesson you will be able to:

1. Separate the EKS managed control plane from the data plane you choose and pay for.
2. Compare managed node groups, self-managed nodes, and Fargate profiles for TechLearningPro workloads.
3. Explain VPC CNI awareness at a high level: pods as VPC citizens with IP implications.
4. Describe EKS add-ons as versioned cluster software you must keep intentional.
5. Distinguish historical aws-auth ConfigMap patterns from IAM access entries without inventing current-only APIs.
6. Tease IRSA (IAM Roles for Service Accounts) as pod identity to AWS APIs—and know where deep identity design continues.
7. Apply an ECS versus EKS decision framework for progress, enrollment, and platform services.
8. Sketch a multi-service app on EKS with Ingress and Horizontal Pod Autoscaler concepts.
9. Cross-link correctly to /learn/kubernetes and /learn/docker instead of duplicating those courses.
10. Plan budget alerts, tags, and full cluster deletion for labs.
11. Identify common EKS failure modes: networking, access, version skew, and idle control-plane cost.
12. Write a short ADR defending EKS or ECS for a named TechLearningPro service.
13. Explain why image registries (ECR) are the next operational dependency after the cluster exists.
14. Map observability and reliability expectations for Monday board traffic on EKS without pretending Kubernetes is free.

Prerequisites

• Containers with Docker for AWS Workloads and comfort with images, registries at a conceptual level, and process isolation.
• Amazon ECS and AWS Fargate so you can compare orchestrators with shared vocabulary.
• VPC, IAM, and load balancing foundations from Module 1 and earlier Module 2 lessons—private subnets, security groups, least privilege.
• Optional but valuable: skim Kubernetes for Deployment/Service/Ingress vocabulary; this lesson will not re-teach the API objects in full.
• Budget alert in the sandbox account before creating any EKS cluster. Control planes and worker nodes bill while they exist.
• Local or CloudShell access with aws, and willingness to install or use kubectl and optionally eksctl as documented in current AWS and project docs—verify versions against official guides at lab time.

Detailed Core Concepts

What EKS is for in TechLearningPro

Amazon Elastic Kubernetes Service (EKS) runs the Kubernetes API server and associated control-plane components as a managed service. You create a cluster; AWS provides a highly available control plane endpoint (with private/public endpoint options you configure). Applications still run as pods on compute you attach: EC2-backed nodes or Fargate. EKS does not remove Kubernetes complexity; it removes the undifferentiated heavy lifting of operating etcd and API servers yourself—while introducing AWS-specific networking, IAM, and add-on concerns.

Choose EKS when TechLearningPro needs Kubernetes portability, a rich ecosystem of controllers, multi-tenant namespace boundaries for platform teams, or skills alignment with a broader Kubernetes estate. Choose ECS when the team wants AWS-native simplicity for a bounded set of services and does not need the Kubernetes API. Choose neither when Lambda and an API Gateway path already fit the event shape—see earlier serverless lessons.

Control plane versus data plane

Control plane (managed by AWS for EKS): the Kubernetes API, scheduler, and control-plane components AWS operates. You do not SSH into control-plane instances. You pay for the cluster while it exists (verify current pricing dimensions in AWS documentation—do not memorize invented numbers). You choose Kubernetes version, upgrade cadence, endpoint access, logging to CloudWatch (when enabled), and encryption settings per current cluster configuration options.

Data plane (your responsibility to design): worker capacity that runs pods. Patterns:

• Managed node groups: AWS manages the ASG lifecycle of EC2 nodes joined to the cluster, with AMI and scaling knobs you configure. Familiar mental model from Compute Services: EC2, AMIs and Auto Scaling.
• Self-managed nodes: you own the ASG or instances and the join bootstrap. More flexibility, more operational burden—usually not the lab default.
• Fargate profiles: serverless data plane for matching pods; you do not manage EC2 for those workloads. Trade-offs include profile selectors, networking constraints, and suitability for certain DaemonSets or privileged patterns—verify current Fargate-on-EKS limits in AWS docs.

The Monday board fails when teams treat “managed control plane” as “managed platform.” Nodes still need capacity, CNI still needs IP space, and Deployments still need healthy probes.

Cluster networking awareness (VPC CNI)

On EKS, the default networking story for many clusters uses the Amazon VPC CNI plugin so pods receive addresses from the VPC (or related IP management features depending on configuration). High-level consequences for TechLearningPro:

• Pod density is limited by ENI and IP allocation behavior on node instance types—tiny nodes can run out of pod IPs before they run out of CPU.
• Security groups and network policies (when enabled) become part of the blast-radius design; do not open world access because “Kubernetes Services are internal.”
• Subnet sizing for private app subnets must anticipate pod IPs, not only node IPs.
• Prefix delegation and related CNI features exist to improve density—enable them deliberately after reading current VPC CNI documentation, not from hallway rumor.

This lesson stays at awareness. Deep CNI tuning, custom networking modes, and policy engines belong in platform engineering follow-ups and the Kubernetes course where network policy theory lives.

Add-ons

EKS add-ons package cluster software—commonly VPC CNI, CoreDNS, kube-proxy, and optionally other AWS-published add-ons such as observability or storage helpers depending on what you enable. Treat add-on versions as production artifacts: pin, upgrade with the cluster version story, and avoid “click latest in console” during an incident. A CNI upgrade is a networking change. Schedule it like one.

Cluster access: history and modern practice

Historically, many EKS clusters mapped IAM principals to Kubernetes RBAC via the aws-auth ConfigMap in the kube-system namespace. Mis-editing that ConfigMap locked teams out; over-editing granted cluster-admin to the wrong role. AWS has moved EKS access management toward IAM access entries (and related access policies) so IAM-to-Kubernetes permissions can be managed as first-class AWS API resources rather than only as a fragile ConfigMap. Mentality for this course:

• Understand that both eras exist in the wild; production clusters may still carry aws-auth history.
• Prefer the access model documented as current for new clusters in the Region you use—verify EKS cluster access management at lab time.
• Never grant cluster-admin to human break-glass roles without an owner, logging, and an expiry story.
• Separate CI deploy roles from human admin roles.

Do not invent API field names here; read current docs when you implement.

IRSA tease: pods calling AWS APIs

Kubernetes ServiceAccounts identify pods inside the cluster. AWS APIs need IAM. IAM Roles for Service Accounts (IRSA) associates an IAM role with a Kubernetes ServiceAccount via OIDC trust so pods receive temporary credentials without long-lived keys on disk. TechLearningPro’s progress API on EKS should read course assets from a specific S3 prefix using IRSA, not an access key baked into the image.

This lesson teases IRSA as the correct identity shape. Full pod identity design, trust policy conditions, and comparison with emerging pod identity helpers belong with IAM deep dives and later security modules—do not confuse “the pod can call S3” with “the pod is secure.” Network path and RBAC still matter.

ECS versus EKS for TechLearningPro

Default bias for a small progress API and enrollment worker: ECS/Fargate unless the organization already standardized on EKS. Default bias for a multi-team learning platform with many deployable services and shared ingress: EKS can win—if you fund platform ownership.

Images, ingress, and autoscaling at sketch level

Pods pull images from a registry—almost always Amazon ECR in this course (next lesson). Ingress controllers or AWS Load Balancer Controller patterns expose HTTP services; verify which controller your platform standardizes. Horizontal Pod Autoscaler (HPA) scales replicas on CPU, memory, or custom metrics; cluster autoscaling or managed node group scaling grows nodes when pods are pending. Application autoscaling without node capacity is a pending-pod incident waiting for Monday.

Cross-link: container image literacy in /learn/docker; Kubernetes object deep dives in /learn/kubernetes.

Cost drivers without fake prices

You pay for the EKS control plane while the cluster exists. You pay for EC2 or Fargate data plane. You pay for load balancers, NAT, CloudWatch logs if verbose, and idle PersistentVolumes if you create them carelessly. Labs: smallest viable node types or Fargate profiles, short lifetime, delete cluster and node groups when done. An abandoned EKS cluster is an expensive textbook left open overnight.

Namespaces as team boundaries (without fake multi-tenancy)

TechLearningPro may place progress-api and enrollment-worker in separate namespaces with separate ResourceQuotas and NetworkPolicies. Namespaces are not hard multi-tenant security boundaries by themselves—RBAC, admission policy, and network controls make them meaningful. Do not promise “namespace equals compliance isolation” in an ADR without the controls list.

Kubernetes version and platform cadence

EKS supports specific Kubernetes versions on a published lifecycle. Running far behind loses security patches and add-on compatibility; running on day-zero of a new minor version without soak time risks the Monday board. Platform teams pick an N-1 style cadence, test add-ons in non-prod, and upgrade control plane then nodes per current AWS upgrade guidance. Application teams pin their tested platform version in the ADR—not “whatever the cluster was last month.”

Architecture and Design Diagrams

Control plane versus data plane:

mermaidCopy
flowchart TB
  subgraph cp["EKS managed control plane"]
    API["Kubernetes API server"]
    SCH["Scheduler / controllers AWS operates"]
  end
  subgraph dp["Data plane you design"]
    MNG["Managed node group EC2"]
    FGP["Fargate profile pods"]
  end
  DEV["kubectl / CI / platform"] --> API
  API --> MNG
  API --> FGP
  MNG --> PODS["progress-api / enrollment pods"]
  FGP --> PODS

Learner request sketch:

mermaidCopy
sequenceDiagram
  participant L as Learner
  participant ALB as Load balancer / Ingress
  participant P as progress-api pods
  participant E as enrollment pods
  participant S3 as S3 via IRSA
  L->>ALB: HTTPS progress request
  ALB->>P: Route to Service
  P->>S3: Fetch course asset metadata
  P->>E: Internal call if needed
  P-->>L: Progress response

ECS versus EKS decision (conceptual):

textCopy
Need Kubernetes API / operators / multi-team K8s platform?
  yes → EKS (+ fund platform ownership)
  no  → Prefer ECS/Fargate for bounded AWS-native services
Either way → Images in ECR (next lesson); IAM least privilege; budget + cleanup

Technical Explanation

Creating a cluster without cargo-cult YAML

Common lab paths: AWS Console guided flow, eksctl for a declarative cluster spec, or infrastructure as code (CloudFormation/Terraform/CDK) in later modules. Whatever tool you use, decide explicitly:

• VPC and private subnets for nodes (public nodes are a teaching anti-pattern for the progress API).
• Cluster endpoint: public access locked down by CIDR for labs, or private-only with a bastion/SSM/VPN path for stricter designs—verify trade-offs in current docs.
• Kubernetes version: choose a supported version you intend to upgrade deliberately.
• Logging: enable API and audit logs to CloudWatch when you need forensics; remember log ingestion cost.

eksctl and Console wizards create convenience defaults. Read the resulting VPC CNI, node IAM role, and access configuration before you call the cluster “done.”

Managed node groups in practice

A managed node group ties an EKS-optimized AMI, instance types, subnets, scaling min/desired/max, and labels/taints. Use labels so progress API pods land on the right pool; use taints for dedicated pools (for example, batch workers). Prefer current EKS-optimized AMIs; do not build snowflake node images in the first lab. Scale min to what you will pay for overnight—often zero in disposable labs after you finish exercises, which means deleting the group or the cluster.

Node IAM roles need permissions for ECR pull, CNI, and other AWS integrations. Prefer AWS managed policy sets documented for node roles, then trim only when you understand the blast radius—blind * on s3:* is not “making CNI work.”

Fargate profiles in practice

A Fargate profile selects namespace (and optional labels) whose pods run on Fargate. Good for isolating enrollment workers without managing another node group. Not a free escape from networking design: pods still need VPC config, and some workloads (DaemonSets, certain hostPath patterns) do not map cleanly. If your Ingress controller expects DaemonSet nodes, Fargate-only clusters need a different ingress strategy—design before you delete the node group “to save money.”

Pod networking and Services

Pods get IPs; Kubernetes Services provide stable virtual IPs and DNS names inside the cluster (progress-api.progress.svc.cluster.local style names). ClusterIP for internal; LoadBalancer or Ingress for external HTTP. On AWS, LoadBalancer Services and Ingress controllers typically provision Classic/NLB/ALB resources depending on annotations and controller—verify the controller your platform uses. Security groups on nodes and, where supported, on pods must allow the path from the load balancer to the pod port.

Pending pods with FailedCreatePodSandBox or IP allocation errors often mean CNI/IP exhaustion—not “Kubernetes hates us.” Check node type, ENI limits, and subnet available IPs.

Ingress and TLS sketch

For TechLearningPro’s learner-facing progress API, terminate TLS at the load balancer with a certificate you manage (ACM patterns from earlier edge lessons), route to an Ingress or Service, and keep pods private. Do not expose NodePorts on public node IPs as a production pattern. Path-based routing can send /progress and /enrollment to different Services—useful for a multi-service sketch without inventing a service mesh.

Autoscaling layers

1. HPA: more pods when CPU/memory/custom metrics rise.
2. Node / cluster autoscaling: more nodes when pods are unschedulable due to capacity.
3. Application design: queue-based workers scale on depth (enrollment grading style work) rather than only CPU.

If HPA wants 20 pods and the node group max is 2 tiny instances, you learn pending pods. Align max replicas with max nodes and instance size. For labs, keep numbers tiny and prove the signal chain once.

Access, RBAC, and break-glass

IAM authenticates to the cluster; Kubernetes RBAC authorizes API verbs on resources. Map deploy roles to namespace-scoped Roles, not cluster-admin. Break-glass admin should be rare, logged, and reviewed. If your cluster still uses aws-auth, treat edits as change-controlled. If you use access entries, treat them as IAM-level production config with the same discipline.

IRSA flow (conceptual)

1. Cluster has an OIDC issuer associated for IRSA.
2. IAM role trust policy allows the ServiceAccount subject.
3. ServiceAccount annotation (or equivalent documented association) points at the role ARN.
4. Pod uses the AWS SDK default credential chain to obtain temporary credentials.

Failure modes: wrong audience/subject in trust policy, missing annotation, pod not using the ServiceAccount, network path to STS/OIDC blocked. Diagnose with pod identity, not by pasting access keys into manifests “temporarily.”

Observability minimum

Control plane logs (when enabled), CloudWatch Container Insights or your chosen metrics stack, pod logs to a central sink, and ALB/target metrics for the edge. Alarm on: 5xx at the edge, pod restart loops, pending pods, node NotReady, and cluster version approaching end of support. Delete verbose lab log retention when you destroy the cluster.

Security baseline for an EKS-hosted progress API

• Private subnets for nodes; restricted API endpoint.
• Images from ECR with immutable digests in production deploys (next lesson).
• IRSA least privilege per ServiceAccount.
• Secrets via AWS Secrets Manager / Parameter Store patterns (upcoming lessons)—not plaintext in ConfigMaps committed to git.
• Admission controls and policy engines as the platform matures—do not claim “we run EKS” as a security control by itself.
• Regular node AMI and add-on patching.

Reliability and AZ thinking

Spread nodes across at least two AZs. Run at least two replicas of the progress API Deployment behind a Service. PodDisruptionBudgets protect Monday capacity during voluntary drains. Persistent state still belongs in RDS/ElastiCache/S3—not in a single pod’s filesystem—same cattle lesson as EC2 ASGs.

When not to use EKS

Single small service, team with no Kubernetes skills, short-lived experiment better on ECS Fargate or Lambda, or a platform that cannot fund upgrades and add-on ownership. EKS abandoned is worse than ECS boring.

Data plane capacity math for the Monday board

Suppose the progress API needs two replicas minimum for AZ loss, each requesting 500m CPU and 512Mi memory, plus kube-system and Ingress pods. Two t3.medium-class nodes may schedule the lab; they may not schedule a sudden HPA burst to ten replicas plus sidecars. Write the resource requests honestly. Overstated requests strand capacity; understated requests cause noisy-neighbor throttling. Fargate sidesteps node bin-packing but still needs correct ephemeral storage and CPU/memory shapes per current Fargate-on-EKS docs. Capacity planning is an engineering artifact, not a hope that the scheduler will invent cores.

Cluster endpoint exposure

Public endpoint access simplifies laptop kubectl for labs when locked to your IP CIDR. Private-only endpoints reduce internet attack surface and require a path into the VPC (VPN, Session Manager port forward patterns, or a bastion you then must harden). Enterprises often prefer private API with federated access. Labs may use restricted public access—document the CIDR, remove 0.0.0.0/0 if you ever opened it for convenience, and rotate thinking toward private as you mature. The Kubernetes API is a privileged plane; treat it like one.

Add-on ownership matrix

Unowned add-ons become surprise Monday outages when a node image requires a CNI version you never tested.

Mapping ECS concepts to EKS (translation table)

Translation helps ECS-native engineers; it does not erase Kubernetes RBAC, Controllers, or CRDs. When the translation stops fitting, stop forcing ECS mental models onto Operators.

Workload identity boundaries beyond IRSA

IRSA covers calls from pods to AWS APIs. It does not replace Kubernetes RBAC, nor does it replace mutual TLS between services if your threat model requires it. TechLearningPro should decide whether enrollment may call progress over plain ClusterIP HTTP inside the VPC-backed cluster network, or whether a service mesh / mTLS path is required later. Do not bolt on a mesh in the first EKS week “because production looks like that on Twitter.” Start with network policies (when ready), clear ServiceAccounts, and IRSA scopes; graduate to mesh when multi-team lateral movement risk justifies the operational cost.

Resource requests, limits, and noisy neighbors

Without requests, the scheduler packs optimistically and Monday latency collapses when enrollment batch and progress API share a node. Without limits, one leaky process consumes the node. With limits far below requests, you confuse operators. Set requests from measured steady state; set limits with headroom; use LimitRanges and ResourceQuotas per namespace so a student-facing experiment namespace cannot starve production namespaces on a shared cluster. Separate node groups with taints when compliance or noisy-neighbor risk is high.

Storage classes and why the progress API should avoid local disk loyalty

EKS can provision EBS or EFS-backed PersistentVolumes through CSI drivers packaged as add-ons or Helm charts depending on your platform standard. The progress API should still keep learner state in RDS/S3 patterns from Module 1. If you attach a PVC “just for logs,” you reintroduce pets. Use sidecars or node log agents shipping to CloudWatch or OpenSearch; delete lab PVCs when you delete the cluster so volumes do not linger on the bill.

Control plane logging that is useful, not theatrical

API, audit, authenticator, controller manager, and scheduler logs can stream to CloudWatch when enabled. Audit logs help answer “who deleted the Deployment?” Authenticator logs help answer “why did this IAM role fail?” Verbose everything forever is a FinOps problem. Enable what your runbooks reference; set retention deliberately; ensure only security/platform roles read audit streams. Labs: enable briefly to see the shape, then disable or delete the log groups in cleanup if you created them solely for learning.

Upgrade rehearsal without heroics

A sober upgrade path: read EKS version release notes and deprecated API notices; upgrade non-prod cluster; upgrade add-ons to compatible versions; run smoke tests on progress and enrollment; upgrade node AMI / node group; only then touch production with a change window. Application teams must not use removed APIs—kubectl deprecations are a delivery problem, not only a platform problem. Pin client versions in CI. Never upgrade production on Friday before a course launch email without a tested rollback story (redeploy previous manifests on a healthy cluster version—you cannot casually downgrade all control planes in every scenario, so soak time matters).

Multi-cluster when one cluster is not a strategy

Some organizations run one giant EKS cluster for everything. Blast radius and noisy neighbors argue for separating prod from non-prod at minimum, and sometimes separating platform system components from app workloads. Multi-cluster multiplies add-on ownership and ECR pull configuration. TechLearningPro’s early platform can start with one non-prod and one prod cluster per Region, with clear promotion of manifests and images—not six half-configured sandboxes named after engineers.

Cost narrative operators should rehearse

Speak in dimensions, not fantasy invoices: control plane hours while the cluster exists; EC2 or Fargate vCPU/GB hours; load balancer hours; NAT vs VPC endpoint trade-offs; CloudWatch log ingestion; idle PersistentVolumes; Cross-AZ data transfer for chatty microservices. The remediation is the same as Module 1: tags, budgets, delete unused, rightsize requests, prefer fewer chatty hops. An ADR that picks EKS must name who watches these dimensions weekly.

Security groups for pods (awareness)

Depending on your CNI configuration and supported features in Region, you may assign security groups to pods for finer network control than node-level groups alone. Awareness level here: useful for isolating enrollment workers from directly reaching data stores that only progress should touch; complexity rises quickly. Verify current Amazon VPC CNI security group for pods documentation before designing around it. Misconfigured SGs cause mysterious timeouts that look like application bugs.

GitOps versus pipeline apply

EKS invites GitOps (cluster desired state in git) or CI kubectl apply / Helm from pipelines. Either can work. Pick one paved path for TechLearningPro services so progress and enrollment do not diverge—one team Helm-from-laptop, another Flux, another raw apply. Platform should publish a template. Secrets still do not belong in git; upcoming Parameter Store and Secrets Manager lessons matter even more on Kubernetes than on a single EC2 guest.

Connecting this lesson to ECR before you need a hero

Every successful deploy assumes the image digest exists, the pull role works, and the tag was not overwritten. If ImagePullBackOff appears during your lesson 18 lab while using a public image, you still preview the failure mode. When you move to private ECR images, lesson 19’s IAM and digest discipline become the difference between a clean rollout and a Monday incident that “Kubernetes caused” when the registry policy was the real root cause.

Operator decision tree when pods will not schedule

textCopy
Pending?
  → describe pod events
  → Insufficient CPU/memory? raise nodes or lower requests
  → IP allocation / ENI? CNI capacity / subnet IPs / instance size
  → affinity/taint miss? fix schedule constraints or Fargate profile
  → PVC unbound? storage class / AZ mismatch
ImagePullBackOff?
  → digests, ECR IAM, endpoints, rate limits
CrashLoopBackOff?
  → logs, probes, config, IRSA denials mistaken for crashes

Carry this tree into on-call notes; it prevents random node restarts as a first response.

Practical Implementation or Design Walkthrough

Problem

In a sandbox account, create a small EKS cluster, deploy a two-service sketch (progress API and a stub enrollment service), expose HTTP via Ingress or LoadBalancer Service as your chosen controller allows, demonstrate replica scaling concepts, then delete everything. Prefer managed node groups or Fargate profiles—not both sprawling. Budget alert first.

Steps (verify CLI flags against current docs)

1. Confirm budget alert and tag strategy: Project=tlp-lesson18, Owner=<you>, Environment=sandbox.
2. Choose a Region and two private subnets (plus public subnets if your load balancer pattern needs them).
3. Create a cluster with a supported Kubernetes version using Console, eksctl, or IaC. Enable only the logging you will read.
4. Attach a small managed node group (min=1 or 2, small instance types) or define a Fargate profile for the target namespaces.
5. Update kubeconfig using the documented aws eks update-kubeconfig flow; confirm kubectl get nodes (or Fargate-ready namespaces).
6. Create namespaces progress and enrollment.
7. Deploy two simple web Deployments (even nginx or a tiny app image from a public gallery for the lab—or images you pushed to ECR if you already completed patterns from the containers lesson). Expose ClusterIP Services.
8. Install or use an Ingress controller / AWS Load Balancer Controller per current AWS documentation for EKS; create an Ingress that routes / to progress and /enrollment to enrollment—or two separate LoadBalancer Services if Ingress setup exceeds lab time. Record which path you chose and why.
9. Scale progress Deployment to 2 replicas; optionally apply a basic HPA on CPU with tiny thresholds to observe replica changes under synthetic load—or manually scale and explain what HPA would automate.
10. Optional IRSA tease: annotate a ServiceAccount and role that allows s3:ListBucket on a lab bucket; confirm the pod can call AWS without embedded keys. Skip if time-boxed, but write the intended trust story in your notes.
11. Capture evidence: kubectl get deploy,svc,ingress -A (as relevant), node list, and a screenshot or CLI output of the HTTP path.
12. Cleanup: delete Ingress/Services (to remove load balancers), delete Deployments, delete node groups or Fargate profiles, delete the cluster, confirm no orphan ENIs/load balancers/NAT you created only for the lab, and verify tagged billable resources are gone.

Illustrative deploy shape (adapt images and probes; not a copy-paste production manifest):

yamlCopy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: progress-api
  namespace: progress
spec:
  replicas: 2
  selector:
    matchLabels:
      app: progress-api
  template:
    metadata:
      labels:
        app: progress-api
    spec:
      containers:
        - name: api
          image: public.ecr.aws/nginx/nginx:stable
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80

Replace with your real progress image from ECR when available. Prefer digests in production.

Failure paths to test

• Remove the node group scaling capacity or shrink nodes until a replica stays Pending; read events with kubectl describe pod.
• Break security group path from load balancer to nodes/pods; observe timeouts; restore.
• Attempt kubectl with an IAM principal lacking access entries/aws-auth mapping; observe denial; fix via documented access path—not by making the API public to the world.

Enterprise Real-World Scenarios

TechLearningPro: EKS because “everyone uses Kubernetes”

A team migrated the progress API from ECS to EKS in two weeks without a platform owner. Add-ons drifted, CNI IP exhaustion hit during a course launch email, and nobody owned Ingress certificate rotation. The board failed for a subset of learners. Repair: either return bounded services to ECS or fund an EKS platform team with version cadence, add-on ownership, IP capacity monitoring, and an ADR that states why Kubernetes is required.

Managed node groups without AZ spread

Nodes landed in one subnet “because the Terraform module defaulted.” An AZ impairment took the board dark. Fix: subnets in at least two AZs, topology-aware replica spreading, PodDisruptionBudgets.

aws-auth edit lockout

An engineer removed the wrong mapRoles entry while “cleaning YAML.” No IAM principal could administer the cluster. Recovery required AWS support paths / documented break-glass depending on cluster configuration age. Fix: access entries (where applicable), change control, and tested break-glass—never casual ConfigMap edits on production.

Fargate profile mismatch

Enrollment pods stayed Pending because the Fargate profile selected the wrong namespace label. On-call scaled a node group that was never in the scheduling path. Fix: kubectl describe pod events first; align profile selectors; document which workloads are Fargate versus EC2.

IRSA replaced with keys in the image

A contractor “temporarily” baked access keys for S3 into the progress image to unblock a demo. Keys leaked via a copied tag to a shared environment. Fix: IRSA only; rotate keys; scan images; treat registry promotion as trust promotion (next lesson).

Ecommerce flash sale on a shared cluster

Retail partner traffic spiked; HPA worked; cluster autoscaler lagged; pods Pending; checkout-like enrollment flows timed out. Fix: warm node capacity before known launches, align HPA max with node max, separate noisy tenants by node pools or clusters when needed.

Healthcare customer isolation story

A healthcare content customer requires stricter isolation. TechLearningPro evaluates dedicated node groups with taints, tighter NetworkPolicies, private API endpoint, and encrypted secrets—not a shared playground namespace with cluster-admin developers. EKS can support the shape; compliance is the control set, not the logo.

Course launch email without capacity headroom

Marketing schedules a launch while the node group max equals the steady-state desired count. HPA fires; pods Pending; learners see intermittent 502s at the Ingress. The incident channel debates “Kubernetes networking” for thirty minutes before someone notices Pending. Fix: pre-scale node desired before launch, align HPA max with node max, and add a pending-pod alarm that pages before Twitter does.

Shared cluster tenancy gone wrong

A data-science namespace without quotas scheduled GPU-sized requests onto the only general node group, evicting progress pods under pressure. Fix: ResourceQuotas, separate node pools with taints/tolerations, and a platform policy that non-prod experimentation cannot share the learner-facing pool.

Addon silent skew after cluster upgrade

Control plane moved to a new minor version; VPC CNI add-on stayed behind; intermittent pod networking failures appeared under load. Fix: add-on compatibility matrix owned by platform; upgrade add-ons in the same change program as the control plane; smoke-test DNS and pod-to-pod before declaring success.

Break-glass drama

Only one engineer’s IAM mapping could administer production. Vacation plus incident equaled extended outage. Fix: at least two break-glass principals, documented access entries, and a tested procedure stored outside the cluster.

Best Practices

• Fund platform ownership before proliferating clusters.
• Prefer managed node groups or clear Fargate profiles over snowflake self-managed nodes for most teams.
• Size subnets and instance types for pod IP density, not only CPU.
• Pin and upgrade add-ons deliberately with the cluster version story.
• Use IAM access entries (or carefully controlled aws-auth on legacy clusters) with least privilege.
• IRSA for AWS API calls from pods; never long-lived keys in images.
• Multi-AZ nodes and multi-replica Deployments for learner-facing APIs.
• Digests for production image pins; ECR next lesson.
• Budget alerts; delete lab clusters the same day.
• Document ECS versus EKS decision per service, not per fashion.
• Keep Secrets out of git manifests; integrate secrets managers in upcoming lessons.
• Monitor pending pods and NotReady nodes as first-class alerts.
• Cross-train on /learn/kubernetes for API depth; use this lesson for AWS integration depth.
• Restrict cluster endpoint access; prefer private where operationally feasible.
• Tag cluster, node groups, and load balancers for cost allocation.

Common Mistakes and Anti-Patterns

Mistake: Treating managed control plane as managed application platform.
Symptom: Outages in CNI, Ingress, or node capacity with no owner.
Cause: No platform team / no add-on ownership.
Fix: Assign owners; write upgrade runbooks; or choose ECS.

Mistake: Single-AZ node group for production progress API.
Symptom: AZ event takes board dark.
Cause: Subnets or ASG limited to one AZ.
Fix: Multi-AZ capacity and replica spreading.

Mistake: World-open cluster endpoint.
Symptom: Attack surface on Kubernetes API.
Cause: Convenience networking.
Fix: CIDR restrict or private endpoint with a controlled path.

Mistake: HPA without node capacity.
Symptom: Pending pods during traffic spike.
Cause: Node max too low / no cluster autoscaling.
Fix: Align layers; load test before course launches.

Mistake: Cluster-admin for CI and humans alike.
Symptom: Wide blast radius, unauditable changes.
Cause: Access convenience.
Fix: Namespace Roles; separate break-glass.

Mistake: Embedding AWS keys in pod env from git.
Symptom: Credential leaks.
Cause: Skipping IRSA.
Fix: IRSA + secret stores.

Mistake: Leaving lab clusters running for weeks.
Symptom: Surprise bill (control plane + nodes + LB).
Cause: No cleanup checklist.
Fix: Delete cluster; verify orphans.

Mistake: Ignoring VPC CNI IP exhaustion.
Symptom: Sandbox errors / Pending pods on “plenty of CPU.”
Cause: Small nodes / small subnets.
Fix: Larger prefix capacity, bigger nodes, or fewer pods per node; read CNI docs.

Mistake: Running DaemonSet-dependent Ingress on Fargate-only without a plan.
Symptom: No ingress pods schedule.
Cause: Data plane mismatch.
Fix: EC2 nodes for those components or alternate ingress design.

Mistake: Cloning the entire Kubernetes course into every EKS ticket.
Symptom: Slow delivery, confused ownership.
Cause: Wrong layer of abstraction.
Fix: Platform provides paved path; apps consume Deployments/Services; deep K8s study stays in /learn/kubernetes.

Troubleshooting and Design Review

kubectl cannot authenticate. Check AWS credentials, Region, cluster name, access entries or aws-auth mapping, and endpoint network path (VPN/DNS/SG).

Nodes NotReady. CNI issues, IAM for nodes, subnet routing, or resource pressure—kubectl describe node and node logs paths per AWS docs.

Pods Pending. Events: insufficient CPU/memory, IP allocation, affinity/taints, or Fargate profile miss.

ImagePullBackOff. ECR permissions, wrong digest/tag, or networking to registry—preview next lesson’s IAM pull story.

Ingress address empty / pending. Controller not installed, subnet tags for discovery missing (common ALB controller requirement—verify current docs), or quota issues.

5xx after deploy. Readiness probes, security groups, wrong Service selector labels, dependency failures (database), or IRSA denials misread as app bugs.

Design review checklist:

• Why EKS instead of ECS for this service?
• Who owns add-ons and version upgrades?
• Multi-AZ data plane?
• How do pods authenticate to AWS APIs?
• What is the image source and pin strategy?
• What is pending-pod alerting?
• Lab cleanup evidence?

Interview Questions and Answers

Q1. What does EKS manage versus what you manage?

EKS manages the Kubernetes control plane. You manage data plane capacity (nodes or Fargate), cluster networking configuration, add-ons, application manifests, access policies, and most day-two operations.

Q2. When would you choose ECS over EKS for TechLearningPro?

When services are bounded, the team is AWS-native, and you do not need the Kubernetes API/ecosystem. ECS/Fargate often delivers faster with less platform overhead.

Q3. What is a managed node group?

An EKS-integrated EC2 Auto Scaling capacity pool using EKS-optimized AMIs where AWS helps manage the node lifecycle joining the cluster, while you set sizes, scaling, and subnets.

Q4. How do Fargate profiles decide where pods run?

They select pods by namespace and labels; matching pods run on Fargate instead of EC2 nodes, subject to current feature limitations.

Q5. Why does VPC CNI IP space matter?

Pods consume VPC IPs (in common configurations); exhausting IPs prevents scheduling even when CPU remains. Subnet and instance ENI limits become capacity limits.

Q6. What was aws-auth used for, and what should you look for now?

It mapped IAM principals to Kubernetes RBAC via a ConfigMap. Newer clusters increasingly use IAM access entries—verify current AWS guidance and migrate carefully on existing clusters.

Q7. What problem does IRSA solve?

It gives pods temporary IAM credentials through a ServiceAccount association so applications can call AWS APIs without long-lived access keys.

Q8. How do HPA and node autoscaling differ?

HPA changes pod replica counts. Node/cluster autoscaling changes worker capacity so those pods can schedule. You typically need both stories aligned.

Q9. What are EKS add-ons?

Managed packages of cluster software (such as VPC CNI, CoreDNS, kube-proxy) versioned and upgraded as part of platform operations.

Q10. Why pin images by digest in production?

Tags can move; digests immutably identify content. Promotes reproducible deploys and safer rollback—expanded in the ECR lesson.

Q11. What is a PodDisruptionBudget for?

It limits voluntary disruptions so drains and upgrades cannot remove too many progress API pods at once during Monday traffic.

Q12. How do you stop an EKS lab from billing?

Delete workloads and load balancers, delete node groups/Fargate profiles, delete the cluster, and verify no orphaned NAT/LB/volumes remain. Confirm with tagged resource inventory.

Hands-On Exercises

Exercise: Multi-service sketch on EKS with cleanup

Problem statement. Create a sandbox EKS cluster, deploy progress and enrollment style services, expose HTTP, demonstrate scaling concepts, document ECS vs EKS choice, and destroy all billable resources.

Tasks.

1. Budget alert + tags before create.
2. Cluster + managed node group or Fargate profile.
3. Two Deployments + Services in separate namespaces.
4. Ingress or LoadBalancer exposure with evidence of HTTP success.
5. Scale replicas; note what would be required for HPA + node scaling in production.
6. Short write-up: IRSA plan for S3 read (even if not fully implemented).
7. Full cleanup evidence.

Evaluation rubric.

Challenge ADR

Decision: For TechLearningPro’s progress API in environment X, choose EKS or ECS. If EKS, specify node group vs Fargate, multi-AZ, access model, add-on owners, image registry, and IRSA roles at a high level. Include cost-control constraints (max nodes, delete policy for non-prod) without inventing dollar amounts. State consequences: platform skills required, upgrade cadence, and pending-pod monitoring.

Lab evidence checklist

Record Region, cluster name/ARN, Kubernetes version, endpoint access mode, node instance types or Fargate profile selectors, subnet IDs, Ingress/LB DNS names, Deployment replica counts, IAM access approach, budget alert ID, and cleanup timestamps. This is the homework quality bar.

Detailed narrative: a careful first cluster day

You start with Budgets. You refuse to create a cluster until an alert exists. You pick two private subnets across AZs that already have enough IPs for pods. You create the cluster on a supported version. You add a managed node group with min=2 small nodes for a multi-AZ sketch—or a Fargate profile for progress and enrollment if you are proving serverless data plane. You authenticate with update-kubeconfig and confirm access. You deploy two Services and prove DNS inside the cluster with a debug pod if needed. You expose through a load balancer. You hit the URL from an allowed path. You scale to two replicas and terminate one pod to watch replacement—the same cattle instinct as ASGs. You then delete Ingress, workloads, node group, and cluster, and you do not leave a load balancer orphaned overnight. That loop is competence.

Walkthrough notes: Ingress choice under time pressure

If installing the AWS Load Balancer Controller exceeds the lab clock, two LoadBalancer Services still teach exposure, security groups, and cleanup of load balancer orphans. Document that production TechLearningPro prefers a single Ingress (or documented gateway pattern) to reduce LB sprawl and centralize TLS. The pedagogical goal is the path from Service to learner HTTPS, not collecting controller CRDs. When you do use the Load Balancer Controller, verify subnet tags and IAM for the controller ServiceAccount (IRSA again) against current AWS docs—missing subnet tags are a classic “Ingress pending forever” cause.

Walkthrough notes: proving multi-service without a mesh

Progress and enrollment can communicate with Kubernetes DNS and ClusterIP. Curl from a progress pod to http://enrollment.enrollment.svc.cluster.local (adjust names) proves service discovery. That is enough for the sketch. Record that production may later add retries, timeouts, and auth between services—omitting them in the lab is fine if the write-up names the gap. Do not pretend a successful curl equals a hardened microservice architecture.

Walkthrough notes: cleanup order that saves money

1. Delete Ingress / LoadBalancer Services first and wait until AWS load balancers disappear.
2. Delete Deployments, HPAs, and leftover PVCs.
3. Delete managed node groups or Fargate profiles; confirm EC2 instances terminate.
4. Delete the EKS cluster.
5. Delete lab log groups, unused security groups you created, and any NAT or endpoints created solely for the exercise if your account owner wants them gone.
6. Re-run inventory filtered by Project=tlp-lesson18.

Skipping step 1 is how “I deleted the cluster” still leaves an ALB billing quietly.

Comparing mental models: ASG replace versus pod replace

Terminating an EC2 instance in an ASG and deleting a pod in a Deployment train the same reliability instinct: cattle, not pets. Differences matter: pod replacement is usually faster; node replacement rebalances many pods; ImagePullBackOff has no exact ASG twin; IRSA replaces instance profiles at the pod grain. Engineers who learned lesson 8 well adapt to EKS faster than those who only memorized kubectl flags.

Platform checklist before calling EKS “production ready”

• Version and add-on owners named
• Multi-AZ data plane
• Access model documented (access entries and/or aws-auth legacy notes)
• IRSA for AWS calls; no static keys
• Ingress TLS and certificate renewal owner
• Pending-pod and NotReady alarms
• Image digest policy (lesson 19)
• Backup/DR story for cluster config and for data stores (data stores are not etcd hobbies)
• Cost tags and budget
• Upgrade calendar

If more than two boxes are empty, you have a demo cluster, not a production platform—ECS may still be the braver choice.

Lesson Summary

EKS gives TechLearningPro a managed Kubernetes control plane; you still design the data plane, networking, access, add-ons, and cost. Use managed node groups or Fargate profiles deliberately, treat VPC CNI IP space as capacity, prefer least-privilege access entries (and respect aws-auth history on legacy clusters), and tease IRSA as the path for pod AWS credentials. Choose EKS when the Kubernetes API and platform investment are justified; otherwise prefer ECS for bounded services. Deep pod and controller theory stays in /learn/kubernetes; image promotion and scanning come next in Amazon ECR and Container Image Supply Chain. Labs that leave clusters running fail the operational half of the course.

Further Reading

• Amazon EKS User Guide
• EKS networking and Amazon VPC CNI
• Managed node groups
• EKS Pods on AWS Fargate
• Cluster access management
• IAM roles for service accounts
• EKS add-ons
• Amazon ECS vs EKS decision guidance (AWS containers docs)