Build Caching

Introduction

Build caching in CI reuses Docker layers, dependency downloads, and compiled outputs across runs — cutting wall clock and cost when cache keys are stable and remote cache is shared across runners and developers.

The story

CI bill doubled when the team switched to fresh VMs every job — no cache, 4-minute npm ci every run. Platform enabled three layers: GitHub Actions cache: npm, Docker GHA cache (cache-from: type=gha), and Turborepo remote cache. Median PR pipeline: 18 min → 4 min. Then a leaked env var in cache key caused cross-branch cache bleed — fix: sanitize keys, scope by lockfile hash, never include secrets in cache paths.

Understanding the topic

CI caching operates at three layers — dependency, container layer, and task output — each with different key semantics and invalidation rules.

• Dependency cache: npm/yarn/pip/m2/gradle caches keyed on lockfile hash; restore before install, save after.
• Layer cache (Docker): BuildKit cache mounts and registry/GHA cache for RUN apt/npm layers; invalidate when Dockerfile or early layers change.
• Remote cache (Turbo/Nx/Bazel): content-addressed task outputs shared org-wide; hash = sources + deps + tool version + env allowlist.
• Cache key design: stable prefix + lockfile hash + runner OS; avoid branch name in key unless intentional isolation.
• Invalidation: lockfile bump, base image digest change, compiler flag change — must bust cache predictably.

Internal architecture

Three-layer cache stack in a typical containerized Node CI job:

Job start
   ↓
[1] Dependency cache (actions/cache or setup-node cache: npm)
    key: ${{ runner.os }}-npm-${{ hashFiles('package-lock.json') }}
    restore → ~/.npm
   ↓
npm ci (fast if cache hit)
   ↓
[2] Turbo/Nx remote cache
    task hash → hit → skip build/test, restore dist/
   ↓
[3] Docker layer cache (build-push-action)
    cache-from: type=gha
    cache-to: type=gha,mode=max
    RUN npm ci layer reused if package-lock unchanged
   ↓
Push image (only uncached layers upload)

Visual explanation

Two diagrams show where Build Caching lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Measure baseline: log install and build duration without cache across 20 runs.
2. Add dependency cache keyed on lockfile — verify hit rate >80% on unchanged deps PRs.
3. Enable Docker BuildKit cache export to GHA or registry; order Dockerfile for max layer reuse (deps before source COPY).
4. Connect Turborepo/Nx remote cache with team token; declare env vars affecting build in config.
5. Monitor hit rate dashboard; alert if sudden drop (key bug or lockfile churn).

Production implementation

Combined caching in GitHub Actions — npm, Turbo remote, Docker GHA:

• Bazel: --remote_cache=grpcs://cache.example.com with auth; RBE shares cache + execution.
• Gradle: build-cache + dependencies cache in GitLab CI cache: key: ${CI_COMMIT_REF_SLUG}-gradle.
• Avoid caching secrets or .env in actions/cache paths — public fork risk on cache poisoning.

# Dockerfile — cache-friendly layer order
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

FROM node:20-alpine AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npx turbo run build --filter=@acme/api

# .github/workflows/ci.yml excerpt
- uses: actions/setup-node@v4
  with:
    node-version: 20
    cache: npm

- uses: actions/cache@v4
  with:
    path: .turbo
    key: ${{ runner.os }}-turbo-${{ hashFiles('package-lock.json') }}-${{ hashFiles('turbo.json') }}

- uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    cache-from: type=gha
    cache-to: type=gha,mode=max
  env:
    TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
    TURBO_TEAM: acme

Execution workflow

Real-world use

Docker BuildKit (2019+) made layer caching standard. GitHub Actions cache v2 (2023) improved npm/yarn integration. Turborepo Vercel Remote Cache and Nx Cloud commercialized cross-machine task caching — DORA lead time improvements often trace to cache hit rate increases documented in Vercel case studies.

Enterprise use cases

Shopify uses extensive remote caching for Ruby and JS monorepos. Google Bazel Remote Cache is mandatory at scale. GitHub Actions cache limit 10GB per repo — large teams use external S3-backed cache or Turbo/Nx cloud instead of relying solely on actions/cache.

• Self-hosted runners: persistent disk cache on runner VM — faster than upload/download but drift risk.
• Registry layer cache: ECR/GCR as cache backend for BuildKit in AWS-native pipelines.
• Cache warming: main branch post-merge job populates cache for feature branches.

Production case study

A data platform team (Node + Docker, 200 PRs/week) cut CI spend 55% with three-layer caching. Initial misconfiguration cached dist/ with stale test results — added Turbo --force on main and test target never cached without coverage flag in hash.

• Baseline: 14 min median, $0.08/run, 40% runner time on npm ci.
• Stack: setup-node cache + Turbo remote + BuildKit GHA cache.
• Incident: wrong globalEnv in turbo.json — cache hit skipped lint on env change.
• Fix: globalEnv includes LINT_STRICT; hit rate 72%; median 3.5 min.

Trade-offs

• Remote cache: massive speedup; trust and auth required — poisoned cache = bad builds.
• Local runner disk cache: fastest restore; not portable across runner pool.
• Aggressive cache keys: high hit rate; stale output if invalidation misses edge case.
• Conservative keys (branch in key): safer isolation; lower hit rate, higher cost.

Security implications

Cache stores are secondary attack surface — poisoned cache can inject malicious artifacts into builds.

• Authenticate remote cache (TURBO_TOKEN, Nx Cloud) — no public write access.
• Don't include API keys or branch names with secrets in cache key computation inputs logged publicly.
• Public repo forks cannot access base repo cache on GitHub — by design; don't weaken for convenience.
• Periodically purge remote cache on security incident or compromised runner.

Scalability analysis

Cache upload/download becomes bottleneck; storage quotas hit at high PR volume.

• GitHub Actions 10GB cache limit — evict LRU; monorepos need external remote cache.
• Large node_modules cache tar upload can exceed job time savings — measure net benefit.
• Multi-region CI: replicate remote cache or accept cross-region latency on restore.
• Cache stampede on main after lockfile mega-bump — warm cache job before opening flood of PRs.

Staff engineer insights

• Dockerfile order matters more than cache backend — COPY source before deps kills layer reuse.
• If cache hit rate drops suddenly, check lockfile churn or someone added timestamp to Dockerfile.
• Test jobs with coverage should include coverage config in task hash or use --force on main.
• Measure dollars saved — finance funds remote cache subscription when you show CI bill delta.

Best practices

• Order Dockerfile: base → system deps → package manifests → npm ci → COPY source → build.
• Cache key includes lockfile hash and tool version (node:20 vs node:22).
• Use remote cache for monorepos — local actions/cache insufficient at scale.
• Run cache-bust full build on main after major dependency upgrades.

Common mistakes

• Caching test results without inputs in hash — green CI, broken code.
• COPY . . before npm ci in Dockerfile — every commit invalidates dep layer.
• Sharing unscoped cache across prod and experimental branches with different env.

Advanced interview questions

Hands-on exercise

Compare CI job duration with and without Docker GHA cache and npm cache on the same commit.

• Log layer IDs reused vs rebuilt between runs.
• Reorder Dockerfile badly — measure cache invalidation impact.
• Document hit rate formula for your stack.

# Run 1: no cache
docker buildx build --no-cache -t demo:no-cache .

# Run 2: with GHA cache (in Actions)
# cache-from: type=gha / cache-to: type=gha,mode=max

# Locally simulate layer cache:
docker buildx build --cache-from type=local,src=/tmp/buildcache \
  --cache-to type=local,dest=/tmp/buildcache-new,mode=max \
  -t demo:cached .

Summary

You can implement build caching with layer cache, dependency cache, and remote cache — designing keys, Dockerfile order, and monitoring hit rates to cut CI time and cost without sacrificing correctness.