Secrets in CI

Introduction

Secrets in CI are credentials pipelines use to reach databases, clouds, and APIs — and the primary exfiltration target in supply-chain attacks. Staff engineers eliminate long-lived plaintext keys (no AKIA... in repo settings) in favor of OIDC federation, HashiCorp Vault dynamic secrets, Sealed Secrets for GitOps, and platform log masking. This lesson is how production platforms authenticate pipelines without secrets sprawl.

The story

A startup's GitHub Actions log printed `AWS_ACCESS_KEY_ID=AKIA...` when a debug step echoed environment variables — bots scraped it within minutes. $28k in EC2 mining later, the platform lead replaced every static key with GitHub OIDC → IAM role trust, enabled org secret scanning, and added pre-commit hooks blocking secret patterns. No long-lived cloud keys have existed in repo settings since.

Understanding the topic

Modern CI secrets architecture:

• OIDC federation: CI platform issues short-lived JWT; cloud IAM/Azure/GCP trusts issuer + subject claim; pipeline assumes role per job — no stored cloud keys.
• Vault: central secret store; CI authenticates via JWT/k8s auth; retrieves dynamic DB credentials TTL 1h; audit log every access.
• Sealed Secrets / SOPS: encrypt secrets at rest in git for GitOps; cluster controller decrypts; CI never holds prod kube secrets.
• Log masking: platform redacts known secret values in output; custom masking for hex tokens; never echo env in debug steps.

Internal architecture

OIDC-first secrets flow (preferred pattern)

Pipeline job starts
   ↓
CI platform mints OIDC JWT (aud, sub, ref claims)
   ↓
Cloud STS validates issuer + subject condition
   ↓
Short-lived session credentials (15min–1h)
   ↓
Job completes — credentials expire automatically

Visual explanation

Two diagrams show where Secrets in CI lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Audit existing secrets: repo/org settings, CI variables, Jenkins credentials store — classify by environment and rotation owner.
2. Replace cloud static keys with OIDC: configure IAM identity provider for GitHub/GitLab issuer; trust policy conditions on repo, environment, branch.
3. For database/application secrets, integrate Vault: CI job authenticates with OIDC JWT → Vault role → dynamic credentials with TTL.
4. GitOps deploy secrets via Sealed Secrets or External Secrets Operator — CI pushes only sealed blobs, never plaintext.
5. Enable log masking, secret scanning (GitHub Advanced Security, gitleaks in CI), and block PRs containing secret patterns.

Production implementation

GitHub Actions OIDC to AWS — no AKIA keys in secrets:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/gha-payments-prod
          aws-region: us-east-1
          role-session-name: gha-${{ github.run_id }}

      - run: aws ecs update-service --cluster prod --service payments --force-new-deployment

# IAM trust policy (AWS) — condition on repo + environment
# "Condition": {
#   "StringEquals": {
#     "token.actions.githubusercontent.com:sub":
#       "repo:myorg/payments-api:environment:production"
#   }
# }

Execution workflow

Real-world use

undefined

Enterprise use cases

Vault dynamic database credentials in GitLab CI:

Production case study

Case study: FinTech eliminated 340 static AWS keys from CI in 90 days.

• Baseline: keys in GitHub org secrets, GitLab variables, Jenkins credentials — no inventory.
• Phase 1: OIDC roles per repo/environment; block new static keys via org policy.
• Phase 2: Vault for DB and API keys; GitLab JWT auth; dynamic TTL 30 min.
• Outcome: secret scanning clean; PCI audit finding closed; zero mining incidents post-migration.

Trade-offs

• OIDC pros: no rotation, scoped per repo/env, auditable via cloud trail, zero secrets in CI settings.
• OIDC cons: initial IAM/policy complexity; debugging trust condition mismatches frustrates teams.
• Vault pros: dynamic secrets, central audit, multi-cloud abstraction.
• Vault cons: Vault itself is infrastructure to operate HA; latency on secret fetch.
• Static secrets cons: rotation toil, leak persistence, broad blast radius — avoid except legacy bridge.

Security implications

Secrets threat model in CI:

• Fork PR exfiltration: never expose org secrets to workflows running untrusted fork code.
• Log leakage: mask secrets; ban `env | sort` debug; use `::add-mask::` in GitHub Actions for dynamic values.
• Over-scoped IAM roles: deploy role writes one service, not `*:*` admin.
• Secrets in artifacts: scan build outputs — .env files accidentally uploaded.
• Third-party actions: compromised action steals ${{ secrets.X }} — pin SHA, audit action code.

Scalability analysis

Enterprise secrets management at scale:

• Environment-scoped secrets: prod secrets only in production environment — not org-wide variables.
• Vault namespaces: isolate teams; JWT auth per namespace; rate limit secret reads.
• Rotation automation: static secrets that remain use automated rotation Lambda; alert 14 days before expiry.
• Secret inventory: dashboard of all CI secrets, owner, last rotated, which pipelines consume.
• Break-glass: documented emergency credential with 4-eyes approval and automatic expiry.

Staff engineer insights

• If you see AKIA in a workflow file or log, stop and fix — that's P1, not tech debt.
• OIDC trust policy conditions are your authorization model — treat `sub` claim as RBAC.
• Vault is worth it above ~50 secrets or any dynamic DB credential requirement.
• Log masking is defense in depth — never rely on it instead of eliminating static secrets.

Best practices

• OIDC for all cloud deploy auth — static keys are legacy debt.
• Scope secrets to environments, not org-wide — prod ≠ staging.
• Never pass secrets as CLI args — use env vars or secret files (masked).
• Run gitleaks/trufflehog on every PR; pre-commit hooks locally.
• Rotate break-glass credentials immediately after incident use.

Common mistakes

• Storing base64 "encoded" secrets in git — encoding is not encryption.
• OIDC role with trust `repo:org/*:*` — any repo in org assumes prod role.
• Vault root token in CI — defeats Vault purpose; use JWT auth roles.
• Printing partial secrets for debug ("first 4 chars") — bots reconstruct.

Advanced interview questions

Hands-on exercise

Exercise: Rewrite this insecure workflow to use OIDC. Identify every security flaw in the original.

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - run: |
          echo "Key: ${{ secrets.AWS_ACCESS_KEY_ID }}"
          export AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}
          export AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws s3 sync ./build s3://prod-customer-data

Summary

You understand secrets in CI: OIDC federation over static AKIA keys, Vault for dynamic credentials, Sealed Secrets for GitOps, and log masking — the architecture staff platform engineers deploy to pass audits and survive fork-PR attacks.