Container Image Scanning

Introduction

Container Image Scanning inspects OCI images — OS packages (apk, deb, rpm) and application layers — for CVEs before deploy. CI integrates Trivy or Grype at build time; Kubernetes admission controllers (Kyverno, OPA Gatekeeper) enforce scan policy at deploy time as defense in depth. Staff engineers treat "scanned in CI" and "blocked at admission" as complementary — CI catches early; admission catches bypassed or stale images.

The story

A platform team passed CI with a green build but ops deployed an image tagged :latest pulled from a cache — not the CI-scanned digest. Prod ran a base image with CVE-2024-3094 (xz backdoor class incident pattern). After the incident they enforced: CI Trivy scan → push by digest only → Kyverno ClusterPolicy rejecting images without scan annotation or with Critical CVEs. A developer tried to kubectl apply an unscanned image from laptop — admission webhook denied. CI-only scanning alone would have missed the bypass.

Understanding the topic

Two enforcement points: build pipeline scan (shift-left feedback) and cluster admission (shift-right guarantee).

• Trivy: comprehensive scanner — OS vulns, app deps, misconfigs, secrets; trivy image --severity CRITICAL,HIGH --exit-code 1.
• Grype (Anchore): fast CVE matching against Grype DB; pairs with Syft for SBOM; similar CI integration.
• CI placement: scan after docker build, before registry push; fail or warn per policy.
• Admission controllers: Kyverno verifyImages with attestations; Gatekeeper OPA policies on image digest and CVE allowlists.
• Digest pinning: deploy app@sha256:abc... not app:v1.2.3 — ties deploy to scanned artifact.

Internal architecture

Image scan architecture — CI scan, attest, push; admission verify on deploy.

docker build → tag with git SHA
   ↓
Trivy / Grype scan image
   ↓
Policy: CRITICAL → fail build
   ↓
Push to registry (ECR/GCR/Harbor)
   ↓ optional: Cosign sign + scan attestation
Deploy to Kubernetes
   ↓
Admission webhook (Kyverno verifyImages)
   ↓ deny unscanned / CRITICAL CVE
Pod scheduled

Visual explanation

Two diagrams show where Container Image Scanning lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Add CI step after build: trivy image --format sarif --output trivy.sarif $IMAGE.
2. Fail on Critical (and optionally High) with --exit-code 1 --severity CRITICAL,HIGH.
3. Push image by digest; record digest in deploy manifest (GitOps) — not floating tags.
4. Install Kyverno; apply verifyImages policy requiring registry from allowlist + optional Cosign signature.
5. Configure admission to reject :latest tags in prod namespaces.
6. Nightly re-scan images in registry — new CVEs affect already-pushed digests.

Production implementation

GitHub Actions Trivy + Kyverno verifyImages policy:

# CI
  - name: Build
    run: docker build -t $REGISTRY/app:${{ github.sha }} .
  - name: Trivy scan
    uses: aquasecurity/trivy-action@master
    with:
      image-ref: $REGISTRY/app:${{ github.sha }}
      format: sarif
      output: trivy-results.sarif
      severity: CRITICAL,HIGH
      exit-code: 1
  - name: Push
    run: docker push $REGISTRY/app:${{ github.sha }}

# Kyverno ClusterPolicy (excerpt)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-image-digest
spec:
  validationFailureAction: Enforce
  rules:
    - name: disallow-latest
      match: { resources: { kinds: [Pod] } }
      validate:
        message: "Use digest-pinned images, not :latest"
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Execution workflow

Real-world use

Aqua Trivy is CNCF-adjacent and widely adopted in CI templates. Anchore Grype/Syft powers SBOM-first workflows. Google Distroless and Chainguard images reduce CVE surface — scanning still required. NSA/CISA Kubernetes hardening guidance recommends admission control for image provenance. xz utils backdoor (2024) reinforced "scan AND pin digest AND verify admission."

Enterprise use cases

Multi-cluster Kubernetes org: Harbor registry with Trivy built-in; images cannot be pulled until scan completes. Prod clusters enforce Kyverno; dev clusters warn only. Critical CVE blocks prod promote; High creates JIRA via webhook. Base image team publishes golden images rescanned daily — app teams FROM corp/base:2024-06-18.

• Air-gapped: Grype DB mirrored internally; scan offline in CI.
• Serverless: scan container Lambda layers or Cloud Run images same as K8s.
• Exception: break-glass namespace with audit logging — not prod default.

Production case study

Insurtech on EKS (SOC2 + state regulators): auditors asked for proof unscanned images cannot reach prod.

• Challenge: Trivy in CI but developers kubectl applied local builds to staging that leaked to prod-like env.
• Decision: Kyverno enforce digest pinning + Trivy Critical block; ECR scan on push; GitOps only path to prod.
• Outcome: Audit passed; one attempted :latest deploy blocked with clear webhook message.
• Metric: Critical CVE MTTR 8 hours; zero unscanned prod deploys in 11 months.

Trade-offs

• CI scan only: fast feedback but bypassable via manual deploy or wrong tag.
• Admission only: dev finds out at deploy — late feedback; combine with CI.
• Fail on any CVE: blocks on unfixable base image CVEs — waiver + golden base updates.
• Trivy vs Grype: similar; Trivy adds misconfig/secret scan; Grype pairs tightly with Syft SBOM.
• Scan time: large images add 1–5 min — cache Trivy DB in CI.

Security implications

Container scanning closes supply chain gaps but has its own threat model:

• Scan DB must update daily — stale DB misses new CVEs (xz, Log4Shell patterns).
• Admission webhook failure mode: fail-open vs fail-closed — prod must fail-closed with HA webhook.
• Registry credentials in CI need push-only scope; admission uses read-only pull verification.
• Distroless/minimal bases reduce findings but don't eliminate scanning — compliance still requires evidence.

Scalability analysis

Image scanning at scale:

• 500 microservices × 10 builds/day = 5000 scans — cache Trivy DB layer; scan in parallel matrix.
• Registry-side scanning (Harbor, ECR enhanced scanning) offloads CI but adds latency before pull.
• Re-scanning all prod digests nightly when NVD updates — automate with registry API + alert.
• SBOM per image stored in OCI referrers — query across fleet without re-scanning.

Staff engineer insights

• CI green + :latest deploy is the most common way image scanning becomes theater — digest pin or it didn't happen.
• Admission without CI scan means developers learn at 4pm deploy — angry and slow; do both.
• Golden base images rescanned on schedule beat per-app chasing OS CVEs in Alpine packages.
• Re-scan deployed digests when NVD updates — building clean today doesn't mean safe tomorrow.

Best practices

• Scan image in CI before push — same artifact that gets deployed.
• Deploy by digest; treat tags as human-readable aliases only.
• Cache Trivy/Grype vulnerability DB in CI for speed.
• Upload SARIF; integrate with GitHub/GitLab security dashboard.
• Pair scan with SBOM (Syft) stored as OCI artifact for incident queries.

Common mistakes

• Scanning Dockerfile but not the built image — misses actual layer contents.
• CI scans tag v1.2.3 but deploy pulls different digest — policy bypass.
• Admission webhook single replica — outage blocks all deploys or fail-open hole.
• Ignoring base image updates — app code clean, OS CVEs accumulate for months.

Advanced interview questions

Hands-on exercise

Lab: Build a Docker image FROM an old base with known CVEs. Run Trivy with exit-code 1 on Critical. Fix by updating base. Add Kyverno policy rejecting :latest. Attempt deploy with floating tag — capture denial message.

• Export SARIF and view in GitHub Security.
• Deploy same image by digest after scan passes.
• Document CI + admission defense-in-depth diagram.

Summary

You can enforce container image scanning at build (Trivy/Grype) and deploy (admission controllers), with digest pinning linking the two. Explain why CI-only scanning fails when someone deploys the wrong tag — admission is the non-bypassable backstop.