Amazon ECR and Container Image Supply Chain

AWS Cloud & DevOps Engineering · Lesson 19 · AWS Application Platform

Lesson Overview

Every progress API pod and enrollment worker task is only as trustworthy as the bytes it pulls. Amazon Elastic Container Registry (ECR) is where TechLearningPro stores those bytes on AWS: repositories, tags, digests, scan findings, lifecycle expiry, and cross-account pull patterns. Without a deliberate image supply chain, EKS and ECS become very expensive ways to run unknown software at Monday scale.

Containers with Docker for AWS Workloads taught how images are built. Amazon ECS and AWS Fargate and Amazon EKS and Kubernetes on AWS consume images at runtime. This lesson sits between build and run: how you authenticate push and pull with IAM, why digests beat floating tags in production, how basic and enhanced scanning awareness fits promotion gates, how tag immutability and lifecycle policies prevent registry landfill, and how cross-account pull works at a high level. Deep DevSecOps—signing policy engines, admission webhooks, full SBOM governance—is deferred to Secure Software Supply Chain and DevSecOps. Here you learn the ECR-shaped foundation those controls assume.

You will set up scanning and promotion across environments in a lab mindset, keep budget and cleanup habits, and preview AWS Systems Manager and Fleet Operations as the next operational surface for hosts and parameters that still surround containers.

Learning Objectives

By the end of this lesson you will be able to:

1. Create and organize ECR repositories for TechLearningPro services with clear naming and tags.
2. Explain tags versus image digests and why production deploys should pin digests.
3. Authenticate Docker/OCI clients to ECR using IAM-backed login flows without long-lived registry passwords.
4. Describe basic versus enhanced scanning awareness and where to verify current scan capabilities in AWS docs.
5. Enable tag immutability for release repositories so tags cannot be silently overwritten.
6. Design lifecycle policies that expire untagged and old images without deleting the digest you still run.
7. Sketch cross-account pull patterns with resource policies at a high level.
8. Apply a promote-on-trust model: scan before promote, immutable promotion tags, trusted base images.
9. Connect ECR to ECS task definitions and EKS pod specs as image consumers.
10. Avoid registry cost and risk anti-patterns: latest in production, unscoped IAM, immortal untagged layers.
11. Run a hands-on promotion lab with evidence and cleanup.
12. Point forward to DevSecOps for signing and policy-as-code without pretending this lesson finished supply chain security.
13. Write an ADR for image promotion across sandbox, staging, and production accounts.
14. Use stable CLI verbs such as aws ecr describe-images and get-login-password while verifying flags against current documentation.

Prerequisites

• Containers with Docker for AWS Workloads and local familiarity with build/tag/push concepts from /learn/docker.
• Amazon ECS and AWS Fargate and/or Amazon EKS and Kubernetes on AWS so you know who pulls images.
• IAM foundations: roles for CI push and runtime pull, least privilege.
• Budget alert before creating many large images or enabling features that add scan cost—verify current pricing dimensions in AWS docs; do not memorize invented prices.
• Docker or another OCI-compatible client available for the lab, or a CI runner equivalent.

Detailed Core Concepts

What ECR is for

Amazon Elastic Container Registry is a managed OCI-compatible container registry integrated with IAM, encryption, replication options, and scanning features. You create a repository per image product (for example progress-api, enrollment-worker), push manifests and layers, and grant pull permissions to ECS task roles, EKS node roles, and IRSA-adjacent pull paths depending on architecture. ECR Private is the default story for TechLearningPro application images. Amazon ECR Public exists for publishing public images; do not confuse “public gallery base images” with “our progress API should be public.”

ECR is the wrong place to store non-image blobs as a general artifact store—use S3 for generic artifacts. It is the right place for container images that ECS/EKS/Lambda (container images) will pull.

Repositories, tags, and digests

A repository holds related images for one named artifact. An image is addressed by:

• Tag: a mutable (unless immutability is on) human string such as v1.4.2, staging, or the anti-pattern latest.
• Digest: an immutable content hash (sha256:…) identifying the manifest.

Pushing progress-api:staging twice can move the tag to new bytes if the repository allows tag mutation. The digest of the previously running production image does not change—but if your deploy config said :staging, you silently changed production at the next pull/restart. Production pins digests (or immutable tags that never move). Floating tags are for humans and for carefully gated promotion pointers, not for unreproducible prod.

List what is in a repo with documented CLI such as:

bashCopy
aws ecr describe-images --repository-name progress-api --region <region>

Verify parameters in current AWS CLI reference. Read image tags and imageDigest fields from the response.

Push and pull authentication via IAM

ECR does not use a long-lived username/password you paste into git. Typical flow:

1. IAM principal (user or role) is allowed ecr:GetAuthorizationToken and repository-level actions (ecr:BatchCheckLayerAvailability, ecr:PutImage, ecr:InitiateLayerUpload, … for push; ecr:BatchGetImage, ecr:GetDownloadUrlForLayer for pull—confirm the exact action set in AWS docs for your use case).
2. Client retrieves a short-lived token, commonly via aws ecr get-login-password piped into docker login against the registry URI.
3. Push/pull proceeds over HTTPS to the regional registry endpoint.

ECS and EKS integrate pull permissions through execution roles / node roles / pull-through configurations as documented—apps should not embed registry passwords. CI push roles should not be the same as production runtime roles.

Image scanning awareness (basic and enhanced)

ECR can scan images for known vulnerabilities in operating system and some language packages depending on scan type and configuration. Historically and in current product language you will encounter:

• Basic scanning awareness: often associated with Clair-based results on push or manual scan depending on settings—verify what your account and Region offer today.
• Enhanced scanning awareness: integrates with broader AWS vulnerability intelligence (commonly discussed alongside Amazon Inspector integration in current docs)—verify enablement steps and findings delivery in the ECR image scanning documentation.

Course rule: scan before promote. Do not treat a green pipeline that only compiles as a security gate. Findings need severity policy: block criticals on production promotion unless a documented exception exists. This lesson does not invent CVE scoring thresholds as law—your org sets them; TechLearningPro should set them explicitly for progress and enrollment images.

Tag immutability

Repository image tag mutability can be MUTABLE (default mental model) or IMMUTABLE. Immutable repositories reject overwriting an existing tag. That prevents “someone force-pushed v1.4.2 with different bytes.” Combine immutability with digest pins for defense in depth. Use mutable tags only in scratch/dev repositories where convenience outweighs risk—and never as the sole production identifier.

Lifecycle policies

Untagged images and old tags accumulate storage cost and confusion. Lifecycle policies define rules such as: expire untagged images older than N days; keep last M images for a tag prefix; expire images matching dev-* sooner than prod-*. Design rules so you never expire the digest still referenced by production ECS task definitions or EKS Deployments. Dry-run thinking: list what would match before applying aggressive expiry in shared registries.

Cross-account pull (high level)

Enterprises often build in a shared “artifacts” account and pull into workload accounts. Patterns:

• Repository resource policy granting ecr:BatchGetImage and related pull actions to the workload account root or specific roles.
• Or replicate images to the workload Region/account using ECR replication features where appropriate.

Do not make repositories world-pullable. Prefer explicit principals. Push remains tightly held by CI roles in the artifacts account. Verify current policy examples in AWS docs—resource policy JSON shapes evolve; copy from official samples, not from memory of a blog post.

Supply chain basics (foundation only)

Trust chain for TechLearningPro images:

1. Trusted base images from Amazon ECR Public verified sources or internal golden bases—not random Docker Hub tags without digest pins.
2. Build in CI with least privilege; no developer laptops as the only path to prod tags.
3. Scan on push or before promote.
4. Promote by digest into environment-specific tags (staging, prod) only after gates—or better, deploy digests directly.
5. Immutable release tags in release repos.
6. Runtime pull by ECS/EKS with scoped IAM.

Signing, admission controllers, provenance attestations, and organization-wide policy-as-code deepen in /learn/aws-cloud-devops-engineering/secure-software-supply-chain-and-devsecops. Do not claim you are “done” with supply chain after creating an ECR repo.

Encryption and private access

ECR encrypts images at rest with AWS managed or customer managed KMS keys depending on configuration. Prefer private subnets’ workloads pulling via VPC endpoints for ECR (com.amazonaws.region.ecr.api and ecr.dkr interface endpoints, plus S3 gateway for layers—as documented) so node traffic does not require NAT for registry pulls. Endpoint cost versus NAT cost is a design choice; for private EKS/ECS this is a common hardening step.

Naming and multi-service layout

Suggested TechLearningPro layout:

• techlearningpro/progress-api
• techlearningpro/enrollment-worker
• techlearningpro/frontend (if containerized)

Separate repositories beat one mega-repo with unrelated tags. Prefix with org or domain for clarity in shared accounts. Tag CI builds with git SHA (sha-abc1234) and promote semantic versions only when release management agrees.

Cost drivers without fake prices

Storage for images and layers, data transfer across Regions/accounts, and scanning features can add cost. Lifecycle policies are FinOps controls. Deleting a repository does not fix an orphaned pipeline that rebuilds hourly into a forgotten repo—disable the pipeline too. Labs: small images, few tags, delete repositories after the exercise if policy allows, or expire aggressively.

Pull-through caches and upstream trust (awareness)

ECR can be configured, in supported patterns, to cache or pull through from upstream registries so builds do not hammer public registries and so you keep copies in-account. Awareness only here: if you use pull-through, you still owe scanning and pinning; caching a vulnerable base does not sanitize it. Verify current pull-through cache setup in AWS documentation before relying on it in CI.

Manifest lists and multi-architecture images

TechLearningPro may run Graviton (Arm) nodes and x86 CI builders. Multi-arch manifests let one tag resolve to architecture-specific digests. Pinning a digest pins a specific manifest—understand whether you pinned the index or a child manifest when debugging “wrong arch” pull failures on EKS/ECS. Build matrices belong in CI; registry layout should make arch explicit in tags when multi-arch is not used (-arm64 suffixes) to avoid silent scheduling pain.

Architecture and Design Diagrams

Build → scan → promote → run:

mermaidCopy
flowchart LR
  CI["CI build role"] --> ECR_DEV["ECR repo tags sha-*"]
  ECR_DEV --> SCAN["Scan findings"]
  SCAN -->|"gate pass"| PROMOTE["Promote digest to staging/prod tags"]
  PROMOTE --> ECR_REL["Release repo or tags"]
  ECR_REL --> ECS["ECS tasks"]
  ECR_REL --> EKS["EKS pods"]

Auth sequence (conceptual):

mermaidCopy
sequenceDiagram
  participant CI as CI role
  participant IAM as IAM
  participant ECR as Amazon ECR
  participant RT as ECS/EKS pull role
  CI->>IAM: Assume role
  CI->>ECR: GetAuthorizationToken / login
  CI->>ECR: PutImage layers + manifest
  RT->>ECR: Pull by digest with IAM
  ECR-->>RT: Layers

Cross-account sketch:

textCopy
Artifacts account: CI pushes progress-api@sha256:…
Workload account: task/node role principal allowed by repo resource policy
Runtime: pull digest only; no world-open policy

Technical Explanation

Registry URI and regionality

ECR repositories are regional. The registry URL shape includes account ID and Region (verify exact URI format in docs). Cross-Region pulls are possible with policies and network path but add latency and transfer considerations; replication can place images closer to workload Regions. TechLearningPro production in us-east-1 should not casually pull from a forgotten lab Region.

IAM policy mental model

Separate personas:

• CI push role: upload layers and put images to specific repository ARNs; deny delete in production repos if separation of duties requires it.
• Runtime pull role: batch get image and layer download on specific repos; no put image.
• Human break-glass: limited, audited, temporary.

Wildcard arn:aws:ecr:*:*:repository/* for every engineer is how a sandbox becomes a breach narrative. Scope by repository name prefix.

Tag immutability versus digest pins

Immutability prevents tag overwrite. Digest pins prevent “whatever tag currently means.” Use both for release trains. For mutable staging tags used as floating pointers, accept that staging is intentionally moving—and never let production Deployments reference staging.

Scanning in the promotion gate

Practical gate for TechLearningPro:

1. CI builds and pushes sha-<gitsha>.
2. Scan runs (on push or explicit API/console start—verify).
3. Pipeline reads findings (console, EventBridge patterns, Inspector/ECR integrations as documented).
4. If policy fails, do not promote.
5. If pass, either update an immutable vX.Y.Z tag on that digest (if allowed) or record the digest in the deploy artifact for ECS/EKS.

Do not parse scan JSON by folklore; use documented finding fields and your platform’s standard tool.

Lifecycle policy examples (conceptual)

Rules you might encode (adapt to JSON schema in current docs):

• Expire untagged images older than 14 days.
• Keep only the last 10 images tagged sha-*.
• Keep images tagged v* for 90 days unless still referenced—operational process must update deploys before expiry.

Always reconcile with disaster recovery: can you rebuild from git if the image is gone? Prefer rebuildability over immortal registries, but do not expire the only copy of a hotfix still in production.

Cross-account resource policies

A repository policy grants principals in account B permission to pull. Combine with identity policies in account B that allow the pull actions. Both sides must permit—resource policy alone or identity alone is a common misconfiguration. Test with a deny path: role without permission should fail pull; role with permission should succeed on a known digest.

ECS and EKS as consumers

• ECS: task definition image field points at ECR URI with tag or digest; execution role needs pull permissions.
• EKS: Pod spec image field; nodes or imagePullSecrets patterns—on AWS, IAM roles for nodes or IRSA-related pull mechanisms are preferred over stored dockerconfig secrets. ImagePullBackOff is frequently IAM or bad digest—not “Kubernetes networking” alone.

Link mental model: registry health is a dependency of every deploy. Alarm on pull failures during rollouts.

Base image hygiene

Pin base digests in Dockerfiles. Rebuild when bases publish CVEs. Prefer minimal images for the progress API (distroless or slim variants where compatible) to reduce scan noise and attack surface—without breaking required CA certs and runtime dependencies. Document exceptions when a fat image is required for a vendor agent.

Replication and DR awareness

ECR replication can copy repositories to other Regions for DR or multi-Region reads. Treat replication as part of the promotion design, not a blind checkbox. Failover runbooks must reference digests available in the target Region.

Observability

CloudTrail for PutImage and policy changes; scan finding dashboards; storage metrics; failed pull metrics from ECS/EKS events. Alert when production deploy pulls a tag instead of a digest if your policy forbids it (admission or CI check).

Registry as a product boundary

Platform teams own repository templates: KMS settings, immutability default, lifecycle defaults, scan configuration, and resource policy scaffolding. App teams own Dockerfiles and promotion requests. When every squad invents its own repo settings, you get mutable prod tags in one service and immutable in another with no scanning. Standardization is reliability.

What “promotion” means operationally

Promotion is not docker tag theater on a laptop. Promotion is a controlled update of the deploy artifact—Helm values, ECS task definition revision, Terraform variable, or GitOps manifest—so that staging and production reference a digest that passed scan and integration tests. Optional convenience tags (prod) may point at that digest for humans, but controllers should still resolve digests. If a human can push directly to prod from a workstation, you do not have a supply chain; you have a hope chain.

Deny-by-default pull testing

After creating a repository policy for cross-account access, prove a negative: assume a role that should not pull and confirm docker pull or a batch-get API call fails. Prove a positive with the runtime role. Evidence of both belongs in the lab write-up. Teams that only test the happy path discover open policies months later during an audit.

Layer caching, rebuild speed, and why registry design still matters

CI speed depends on layer reuse. Stable base layers should change rarely; application layers should change often. When every build invalidates the base because Dockerfiles are poorly ordered, you push more bytes, pay more transfer, and lengthen promote time. ECR does not fix Dockerfile hygiene—/learn/docker does—but registry lifecycle and retention must assume frequent SHA pushes. Keeping every failed experimental tag forever is how storage grows while delivery stays slow.

Immutable infrastructure meets mutable human language

Product managers say “deploy version 1.8.” Engineers must translate that into a digest. Release notes should include the digest or an immutable tag that is guaranteed not to move. Chat messages that say “I pushed latest” are not an audit trail. TechLearningPro change tickets for enrollment and progress should paste imageDigest from describe-images or from the CI output. Culture beats tooling when someone tries to bypass the pipeline.

Encryption contexts and CMK pitfalls

If you choose a customer managed KMS key for a repository, the CI push role and the runtime pull roles need key permissions (kms:Decrypt / encrypt as required for the flow—verify current guidance). Teams enable CMK for compliance, forget the key policy, and diagnose the failure as “ECR is down.” Design review: repository policy, identity policy, and KMS key policy must agree. Labs may use AWS managed encryption to reduce moving parts; production ADRs should state the key owner and rotation expectation.

Replication versus multi-Region pull

Pulling cross-Region on every pod start couples runtime to WAN latency and transfer cost. Replication (or promoting the same digest into a Region-local repository) keeps pulls local. For TechLearningPro’s primary Region, build local. For a DR Region, replicate scanned digests you are willing to run, and ensure task definitions/manifests in DR reference the DR registry URI—not the primary URI with fingers crossed on DNS.

Event-driven responses to scans (awareness)

Findings can drive EventBridge rules toward chat topics or pipeline fail states depending on how you integrate scanning today—verify current EventBridge/Inspector/ECR integration docs rather than inventing event names here. The architectural idea: do not rely on humans noticing a console badge. Wire severity to automation that blocks promote or opens a ticket with an owner.

SBOM awareness without claiming completion

Software Bills of Materials help answer “what is inside this digest?” ECR and adjacent AWS tooling continue to evolve around visibility into package contents; deeper SBOM attestation and policy belong in the DevSecOps lesson. Here: know that a digest without contents visibility is harder to incident-respond when a CVE drops on a transitive dependency. Prefer images you can rebuild from a locked dependency file in git.

Registry namespaces and blast radius

A single AWS account with hundreds of repositories still has a shared IAM blast radius if push roles are wildcards. Split high-sensitivity images (payment-adjacent experiments, private customer content processors) into tighter accounts or tighter role boundaries. TechLearningPro’s progress API is sensitive for learner trust even when it is not a bank; treat its repo policies accordingly.

Tagging strategy that survives humans

Recommended pattern:

• sha-<gitsha> on every CI build (mutable repo OK).
• vMAJOR.MINOR.PATCH immutable tags on release repo only after gate.
• Environment pointers (staging, prod) optional and never the only prod reference—update them as a convenience after digest deploy succeeds.
• Ban latest in prod via review checklist now; admission later.

Write this in the ADR so new engineers do not invent final-final-v2.

How ECS task definition revisions and EKS GitOps relate to ECR

When you register a new ECS task definition revision, you are promoting a digest if you did the job right. When you merge a GitOps commit changing a Deployment image field, same promotion. Rolling back means pointing at the previous digest that still exists in ECR—or rebuilding it from git if lifecycle removed it. Therefore lifecycle retention must exceed your maximum rollback window, or your rebuild pipeline must be trusted under pressure. State that window explicitly (for example, keep release digests at least as long as your rollback SLO).

Performance: pull latency at scale-out

Cluster scale-out of fifty new pods pulling a multi-hundred-megabyte image simultaneously stresses registry and network paths. Mitigations: smaller images, warm nodes with already-pulled layers, VPC endpoints, and avoiding cross-Region pulls. Course launches that scale from two to many replicas should pre-pull or pre-scale before the email, not during first-request storms.

Security review questions for any ECR repo

• Who can PutImage?
• Who can delete images or change lifecycle?
• Is tag immutability on for releases?
• Are scans required before promote?
• Are resource policies least privilege?
• Are images encrypted with a key whose policy is understood?
• CloudTrail reviewed for unexpected PutImage?
• Are public settings impossible for private app repos?

Answering these aloud in design review catches more risk than buying another scanner logo.

Mapping lesson 18 failures to lesson 19 causes

Keep the table in incident channels bridging app, platform, and security.

Practical Implementation or Design Walkthrough

Problem

Create an ECR repository for a stand-in progress API image, push a build, inspect digests and tags, enable scanning awareness, configure tag immutability appropriate to the repo purpose, add a lifecycle policy for untagged expiry, simulate promotion to a staging pointer or digest-based deploy note, attempt a denied pull with a role lacking permission, then clean up images/repos per sandbox policy. Budget alert first.

Steps (verify every flag against current docs)

1. Confirm budget alert; tag resources Project=tlp-lesson19.
2. Create repository tlp-progress-api (name as allowed) with encryption settings you understand; consider enabling tag immutability for a tlp-progress-api-release repo and keeping a mutable tlp-progress-api-dev if you want both modes.
3. Authenticate Docker to ECR using aws ecr get-login-password and docker login as documented for your Region/account.
4. Build a small image locally (or in CI):

bashCopy
docker build -t tlp-progress-api:sha-lab1 .

5. Tag for ECR URI and push. Record the digest from aws ecr describe-images or from the push output.
6. Enable or confirm image scanning configuration on the repository; trigger or wait for a scan; open findings in console. Write two sentences on how TechLearningPro would gate promotion on severity—without inventing tool-specific APIs.
7. Demonstrate tag risk: in a mutable repo, push a different image to the same tag (if allowed) and observe the digest change; explain why prod must not follow that tag blindly.
8. Create a lifecycle policy that expires untagged images after a short window suitable for labs; confirm JSON against current lifecycle policy schema in docs.
9. Write a promotion note: “staging deploy uses digest sha256:…”. Optionally put that digest into a sample ECS task definition snippet or EKS Deployment snippet (do not need a live cluster if budget-constrained—document the consumer fields).
10. Optional cross-account: if you have a second sandbox account, add a repository policy allowing a specific role to pull; test pull; remove policy after.
11. Cleanup: delete images, delete repositories you created for the lab, revoke temporary policies, and ensure CI credentials used are rotated or temporary roles expired. Leave no orphaned large layers if your account bills for them.

Failure paths

• Push with a role missing ecr:PutImage (or related upload actions)—confirm AccessDenied.
• Pull with runtime-like role missing download actions—confirm failure.
• Reference a wrong digest in a sample manifest—explain ImagePullBackOff / ECS resource pull errors you would expect.

Enterprise Real-World Scenarios

TechLearningPro: latest in production

The progress API Deployment used progress-api:latest. A leftover CI job pushed a broken build to latest. Nodes restarted overnight and pulled bad bytes. Monday board failed. Fix: digest pins; immutable release tags; forbid latest in prod admission checks (full policy in DevSecOps lesson).

Untagged immortal registry

Developers pushed unique SHA tags but also left untagged manifests from failed multi-stage experiments. Storage grew for a year. Finance asked questions. Fix: lifecycle policy on untagged and on sha-* retention counts; educate on rebuild-from-git.

Scan findings ignored

Enhanced or basic scanning showed critical CVEs in a base image. Pipeline was green because tests passed. Customer security review failed. Fix: promote gate on findings; rebuild on patched bases; documented exceptions with expiry.

Cross-account wildcard policy

Artifacts account granted * principals pull on all repos to “make EKS work.” A compromised role in a sibling account pulled internal images. Fix: principal-specific policies; separate sensitive repos; CloudTrail review.

ECS execution role too wide

Execution role could pull every repository including experimental ML images with different compliance posture. Fix: per-service execution roles scoped to needed repository ARNs.

Tag overwrite on “immutable” process that was not

Policy said immutable tags; repository setting left mutable. Hotfix reused v2.3.1 with new digest; canaries and prod disagreed about what v2.3.1 meant. Fix: enable immutability; cut v2.3.1-hotfix.1; pin digests in deploy.

Healthcare promotion story

Healthcare learner data path requires provenance. TechLearningPro progresses from SHA builds → scanned digest → staging soak → prod digest recorded in change ticket. ECR is the store; the promotion record is the compliance artifact. Signing comes in the DevSecOps lesson.

The “temporary” public repo

A team toggled a repository toward public access to share an image with a contractor quickly. The progress API internal build became fetchable more broadly than intended. Fix: never use public ECR for proprietary app images; use cross-account policies with time-bounded roles; rotate and delete exposed tags; treat the event as an incident.

CI role shared across all microservices

One push role could write to every repository. A compromised pipeline for a low-value service pushed a trojan tag to progress-api because IAM was not scoped. Fix: per-repo or per-prefix push roles; separate pipelines; repository policies as defense in depth where applicable.

Region drift between build and run

Images landed in eu-west-1 while EKS ran in us-east-1. Pulls worked intermittently through cross-Region paths until a network control blocked them. Fix: build/promote into the workload Region; use replication deliberately; pin registry URI in deploy templates per environment.

Scan exception without expiry

Security granted a permanent exception for a critical CVE “until the vendor patches.” Eighteen months later the CVE was still in prod digests. Fix: time-boxed exceptions with owners and automatic reopen; rebuild cadence SLAs.

Lifecycle vs. legal hold confusion

A cleanup job expired images under investigation for an incident review. Fix: legal/security hold process that pauses lifecycle rules or copies digests to an evidence bucket; communicate before aggressive expiry in shared registries.

Best Practices

• One repository per deployable image product.
• Pin production by digest; treat floating tags as non-prod or explicit pointers with gates.
• Enable tag immutability on release repositories.
• Scan before promote; define severity policy in writing.
• Lifecycle-expire untagged and stale CI tags.
• Least-privilege push vs pull roles; no registry passwords in git.
• Prefer private pulls via VPC endpoints for private subnets.
• Trusted, pinned base images; rebuild on CVE.
• Cross-account pulls via explicit resource policies.
• Tag AWS resources for cost; delete lab repos.
• Document promotion in ADRs and change tickets.
• Defer signing/admission deep dives to the DevSecOps lesson—but do not skip scanning basics here.
• Align ECS task definitions and EKS manifests with the same digest discipline.
• Verify CLI and console steps against current AWS documentation every lab.

Common Mistakes and Anti-Patterns

Mistake: Deploying :latest to production.
Symptom: Non-reproducible rollouts; surprise broken mornings.
Cause: Convenience tagging.
Fix: Digests / immutable versions; policy checks.

Mistake: Mutable release tags without immutability.
Symptom: Same version string, different bytes.
Cause: Repository left mutable.
Fix: IMMUTABLE on release repos; new version for hotfixes.

Mistake: No lifecycle policy.
Symptom: Rising storage cost; noisy describe-images.
Cause: Forever retention by default habit.
Fix: Expire untagged and old CI tags; keep what prod references.

Mistake: Ignoring scan findings.
Symptom: Audit failure; known exploited CVEs in prod.
Cause: Tests ≠ security gates.
Fix: Block promote on policy; patch bases.

Mistake: Long-lived Docker login tokens in CI variables.
Symptom: Credential leak risk.
Cause: Misunderstood ECR auth.
Fix: get-login-password per job with IAM roles.

Mistake: World-open repository policy.
Symptom: Unexpected pulls; data exfil of proprietary images.
Cause: Troubleshooting with Principal: *.
Fix: Explicit accounts/roles only.

Mistake: Same IAM role for push and runtime.
Symptom: Compromised app can push malware tags.
Cause: Role reuse.
Fix: Separate push/pull personas.

Mistake: Deleting images still referenced by ECS/EKS.
Symptom: Scale-out ImagePullBackOff.
Cause: Lifecycle too aggressive.
Fix: Coordinate retention with deploy pins; prefer digest redeploy from CI rebuild if deleted.

Mistake: Claiming supply chain complete after ECR create.
Symptom: False confidence.
Cause: Skipping signing/admission/provenance.
Fix: Use this lesson as foundation; continue in DevSecOps module lesson.

Mistake: Building only on laptops for prod.
Symptom: “Works on my machine” images; no scan trail.
Cause: No CI.
Fix: CI build + scan + promote.

Troubleshooting and Design Review

Cannot docker login to ECR. Check IAM GetAuthorizationToken, clock skew, wrong Region/account URI, and CLI profile.

Push denied mid-layer. Missing upload actions on repository ARN; repo name mismatch; KMS key policy denying the role if CMK used.

Pull denied from ECS/EKS. Execution/node role lacks pull actions; wrong account; repository policy missing for cross-account; VPC endpoint/S3 endpoint misconfig for private pulls.

ImagePullBackOff with existing tag. Tag moved or deleted; digest mismatch; auth; rate or network path.

Scan not appearing. Feature not enabled on repo/registry; wrong scan type for account; wait/propagation; verify Region support in docs.

Lifecycle deleted needed image. Policy matched more than intended—restore from rebuild; tighten rules; protect prod digests via process.

Design review:

• Are prod references digests?
• Is immutability on for release repos?
• Who can push? Who can pull?
• What is the scan gate?
• What expires when?
• Cross-account principals explicit?
• Cleanup plan for labs?

Interview Questions and Answers

Q1. Why prefer digests over tags in production?

Tags can move to new content unless immutability prevents overwrite. Digests immutably identify the manifest you tested and scanned.

Q2. How does ECR authentication work at a high level?

IAM-authorized clients obtain a short-lived authorization token and use it with the container client to push or pull; long-lived registry passwords are not the design center.

Q3. What is tag immutability?

A repository setting that rejects attempts to push an image using a tag that already exists, preventing silent retag of the same version string.

Q4. What problem do lifecycle policies solve?

They automatically expire images that match rules (such as untagged or old CI tags), controlling cost and clutter without manual archaeology—when designed not to delete live production digests carelessly.

Q5. Basic vs enhanced scanning—how should you speak about them?

As ECR capabilities for discovering vulnerabilities, with enhanced scanning typically associated with deeper AWS integration such as Inspector—always verify current feature descriptions and enablement in official docs for your account.

Q6. How do cross-account pulls work conceptually?

The repository resource policy allows principals from another account to perform pull actions, and that account’s identity policy must also allow those actions.

Q7. What IAM separation should exist for CI vs runtime?

CI can push to specific repos; runtime can only pull what it needs. Compromise of an app should not grant push.

Q8. How do ECS and EKS consume ECR images?

Through image URIs in task definitions or pod specs, with execution/node/pull roles authorized for ECR download APIs.

Q9. Why pin base images?

Unpinned bases change underneath you, altering CVE posture and behavior. Digests make rebuilds intentional.

Q10. Is creating an ECR repository enough for supply chain security?

No. It is necessary infrastructure. Scanning gates, promotion discipline, least privilege, and later signing/admission controls complete the story.

Q11. What does aws ecr describe-images help you verify?

Tags, digests, pushed-at times, and related image metadata for a repository—useful for confirming what production should pin.

Q12. How do VPC endpoints change the pull path?

Private subnets can reach ECR API and Docker endpoints (and S3 for layers) without NAT, reducing exposure and often simplifying private-cluster networking—verify endpoint set in docs.

Hands-On Exercises

Exercise: Scan and promote across environments (lab simulation)

Problem statement. In a sandbox, create ECR repositories, push a progress-api style image, record digests, exercise scanning awareness, apply immutability and lifecycle controls, document a staging→prod promotion by digest, test a denied pull, and clean up.

Tasks.

1. Budget + tags.
2. Create mutable dev repo and immutable release repo (or one repo with clear tag policy and immutability on).
3. Push sha-lab image; capture digest via describe-images.
4. Configure/confirm scanning; note findings handling policy for TechLearningPro.
5. Write promotion record mapping digest → staging → prod (prod may be paper deploy).
6. Add lifecycle rule for untagged expiry; justify retention.
7. Negative IAM test for pull/push.
8. Cleanup repositories/images/policies.

Evaluation rubric.

Challenge ADR

Decision: TechLearningPro image promotion across sandbox, staging, and production. Specify repository layout, immutability, scan gate, digest pinning in ECS/EKS, cross-account pull (yes/no and how), lifecycle retention, and what is deferred to the DevSecOps lesson (signing, admission). Consequences: CI role design, rebuild SLAs if images expire, and operational ownership of the registry.

Lab evidence checklist

Repository names/ARNs, mutability settings, example digest, scan configuration evidence, lifecycle policy JSON reference, IAM role names used for push/pull tests, sample consumer image URI with digest, budget alert ID, cleanup timestamps.

Detailed narrative: promoting the Monday build

A developer merges a progress API fix. CI assumes the push role, builds, tags sha-9f3c, pushes to tlp-progress-api. Scan runs. Critical base CVE appears; pipeline fails promotion. Platform rebuilds on a patched base digest, push succeeds, findings clear the gate. Release engineering records digest sha256:abc… in the staging ECS task definition revision and EKS GitOps commit. After soak, the same digest is referenced in production—not a new “prod rebuild” with different bytes. Lifecycle rules later remove old sha-* images no longer referenced, while the production digest remains until the next successful promote. That story is the supply chain minimum. Signing and cluster admission become the next hardening chapter—not a substitute for this discipline.

Lab narrative: seeing a tag move with your own eyes

In the mutable repository, you push image A as demo. You run aws ecr describe-images and record digest A. You change the Dockerfile by one line, rebuild, and push again as demo. Describe-images now shows digest B for the same tag. You write in your notes: any ECS service or EKS Deployment following :demo will eventually run B without a conscious promote. You then push digest B into an immutable release repository as v0.0.1-lab and confirm a second push of v0.0.1-lab is rejected. That single contrast teaches more than a slide on immutability.

Lab narrative: wiring a consumer snippet

Even without starting a cluster, paste the digest into a sample:

yamlCopy
# EKS sketch — prefer digest in production
image: <account>.dkr.ecr.<region>.amazonaws.com/tlp-progress-api@sha256:<digest>

jsonCopy
{
  "image": "<account>.dkr.ecr.<region>.amazonaws.com/tlp-progress-api@sha256:<digest>"
}

(The JSON shape stands in for an ECS container definition field—verify exact task definition structure in current ECS docs.) Your deliverable explains which IAM role would pull this URI on ECS versus EKS. That explanation is graded as seriously as a successful docker push.

Lab narrative: lifecycle dry reasoning

Before applying a lifecycle policy, list images and mark which digests a hypothetical production deploy might still need. Draft rules that expire untagged first, then old sha-*, while keeping v* longer. Apply in sandbox only. Confirm with describe-images over time or with the lifecycle preview mechanisms documented by AWS if available in your console experience. Never practice first-destructive-lifecycle in the shared production registry.

Connecting ECR to Systems Manager next

Images are not the only operational artifact. Host patches, parameter values, and session access still matter for nodes and for incidental EC2. The next lesson on Systems Manager covers fleet operations that complement—not replace—container registries. A patched node pulling a vulnerable image is still a vulnerable service; a perfect image on an unmanaged, SSH-open node fails a different exam. Hold both standards.

Promotion evidence package for enterprise homework

Submit: repository ARNs; mutability settings; before/after describe-images output for the tag-move experiment; scan screenshot or CLI finding summary; lifecycle policy document; IAM deny/allow test notes; consumer snippets with digests; ADR; cleanup evidence. Graders should be able to replay your reasoning without logging into your account.

Anti-drama rollback drill

Record digest N (current) and digest N-1 (previous good). Write the exact manifest/task definition change to roll back. Confirm N-1 still exists in ECR. If lifecycle would have deleted N-1, fix retention before you need it at 02:00. Rollback drills are supply chain tests as much as deployment tests.

Why TechLearningPro cares more than a toy app would

Learner progress integrity and course content trust are the product. A poisoned enrollment worker image can corrupt grades; a poisoned progress API can leak data. Registry controls are product controls. Treat PutImage on production repositories with the same seriousness as database schema migrations—because for container platforms, they are the migration of the running brain.

Operator runbook snippet: bad image in production

1. Identify the digest currently running (ECS task definition / EKS pod status).
2. Compare to the last known good digest in the change ticket.
3. Roll back the consumer reference to the good digest; do not “rebuild latest” under panic unless the good digest is gone.
4. Disable or pause the CI job that can overwrite mutable tags if that path exists.
5. Preserve the bad digest for forensics; do not lifecycle-delete during investigation.
6. Open a supply-chain incident: who pushed, which role, which findings were ignored, which gate failed.
7. Only after containment, patch Dockerfile/base, scan, promote a new digest through staging.

Operator runbook snippet: cannot pull in private subnets

Check IAM on the pull role first, then repository policy for cross-account, then security groups and routes to ECR endpoints, then presence of the ecr.api, ecr.dkr, and S3 gateway endpoints as documented for private pulls. Confirm the image URI Region matches the workload Region. Only after those layers should you suspect “ECR outage.”

FinOps hygiene specific to registries

Set a monthly review of repository storage by tag prefix. Chargeback tags on repositories (Service=progress-api). Delete abandoned lab repos quarterly. Prefer multi-stage slim images so storage and pull times drop together. Scanning and replication have costs—enable them where risk demands, not on every toy repo by default. The budget alert from the lab start still applies when someone enables enhanced scanning across hundreds of repos without a plan; verify pricing dimensions in AWS docs before org-wide enablement.

Dual-consumer consistency (ECS and EKS)

If TechLearningPro runs enrollment on ECS and progress on EKS during a migration, both must pin the same promotion rules. A digest certified for ECS staging is the same bytes EKS should run—do not “rebuild for Kubernetes” unless the Dockerfile truly differs. Dual rebuilds create split-brain CVEs. One pipeline, one digest, two consumers.

Lesson Summary

ECR is TechLearningPro’s source of truth for container bytes on AWS: repositories, IAM-authenticated push/pull, digests over floating tags, scanning before promote, immutable release tags, lifecycle hygiene, and explicit cross-account pull. ECS and EKS only run what the registry—and your promotion gates—allow. Treat this lesson as the foundation; deepen signing, provenance, and admission in Secure Software Supply Chain and DevSecOps. Next, AWS Systems Manager and Fleet Operations extends operational control to fleets and parameters that still surround your containers. Labs that leave huge unscanned mutable repos behind fail both security and FinOps.

Further Reading

• Amazon ECR User Guide
• Private registry authentication
• Image scanning
• Tag immutability
• Lifecycle policies
• Repository policies / cross-account access
• ECR interface VPC endpoints
• AWS CLI ecr describe-images