Self-Hosted Runners

Introduction

Self-hosted runners execute CI/CD jobs on infrastructure you operate — VMs, bare metal, or Kubernetes pods — instead of vendor-hosted pools. Staff engineers choose them for private network access, custom hardware, cost at scale, and compliance zones. This lesson covers isolation, autoscaling, and the pet vs cattle operational model that determines whether runners become a security liability or a platform asset.

The story

A crypto exchange ran GitHub Actions self-hosted runners as long-lived VMs with Docker socket mounted. A malicious PR container escaped to the host, pivoted via the runner's IAM role, and drained a staging wallet. The rebuild: ephemeral K8s pods per job, no persistent disk, IMDSv2, network policies blocking egress except allowlisted registries — runners became cattle terminated after every job.

Understanding the topic

Self-hosted runner fundamentals:

• Why self-host: VPC database access, GPU/ARM builds, air-gapped environments, data residency, cost optimization beyond ~50k hosted minutes/month.
• Isolation: each job should run in a disposable environment — container, VM snapshot revert, or fresh pod. Persistent home directories leak secrets between jobs.
• Autoscaling: queue depth triggers scale-out (Actions Runner Controller on K8s, VMSS for Azure, ASG for AWS). Scale-in when idle to save cost.
• Pet vs cattle: pets are manually patched snowflake runners; cattle are immutable, scripted, replaced not repaired — staff engineers insist on cattle.

Internal architecture

Ephemeral runner on Kubernetes (reference architecture)

CI platform job queue
   ↓
Runner controller (ARC / GitLab Runner / Jenkins K8s plugin)
   ↓
Schedules ephemeral pod (job container + sidecar)
   ↓
Job executes in isolated network namespace
   ↓
Pod terminated — no state survives

Visual explanation

Two diagrams show where Self-Hosted Runners lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Register runner pool with CI platform — GitHub org/repo runners, GitLab Runner manager, Jenkins K8s cloud.
2. Controller watches job queue — when depth exceeds threshold, provisions new runner instance (pod/VM).
3. Job executes with scoped credentials (OIDC role, short-lived token) — never prod keys on runner filesystem.
4. Post-job hook terminates instance or reverts snapshot — cache artifacts uploaded to remote storage, not local disk.
5. Metrics: queue wait time, runner utilization, job failure rate by runner version — alert on drift from golden image.

Production implementation

GitHub Actions Runner Controller (ARC) — ephemeral runner scale set:

# values.yaml (simplified) — actions-runner-controller
githubConfigUrl: https://github.com/myorg
githubConfigSecret: controller-pat

runnerScaleSet:
  minRunners: 0
  maxRunners: 50
  runnerGroup: "vpc-builds"

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/myorg/actions-runner:2.311.0-custom
        resources:
          requests:
            cpu: "2"
            memory: 4Gi
        env:
          - name: RUNNER_EPHEMERAL
            value: "true"
    securityContext:
      runAsNonRoot: true
    nodeSelector:
      workload: ci-runners
    tolerations:
      - key: "ci-only"
        operator: "Exists"

Execution workflow

Real-world use

undefined

Enterprise use cases

GitLab Runner on Kubernetes — docker executor with autoscaling:

Production case study

Case study: Data platform team scaled self-hosted runners 0→200 concurrent.

• Starting point: 8 pet VMs, 45-min queue at peak, quarterly security findings on unpatched kernels.
• Target: ARC on dedicated EKS node group; min 0 max 200; custom image with pre-warmed toolchains.
• Isolation: fork PRs forced to GitHub-hosted; internal PRs only on self-hosted VPC runners.
• Outcome: queue p95 38 min → 95 sec; security audit passed; runner cost 40% below equivalent hosted minutes.

Trade-offs

• Self-hosted pros: private network, custom images, potentially lower unit cost at high volume, compliance control.
• Self-hosted cons: you own CVE patching, isolation failures, scaling logic, and on-call for runner infrastructure.
• Ephemeral pros: strong isolation, no cross-job secret leakage.
• Ephemeral cons: cold start latency, cache must be remote (S3, registry layer cache).
• Pet runners pros: fast warm caches on local SSD.
• Pet runners cons: snowflake config, secret persistence, highest breach risk — avoid for multi-tenant CI.

Security implications

Self-hosted runners are high-value targets — they hold CI credentials and network access:

• Never run public fork PRs on self-hosted runners with org network access — route untrusted jobs to hosted pool only.
• Disable privileged containers unless absolutely required; mount Docker socket = host root.
• IMDSv2 on cloud VMs: prevent SSRF from job to instance metadata credentials.
• Network egress allowlist: jobs reach package registries and git only — block lateral movement.
• Runner registration tokens: rotate; scope to runner group; never log during registration.

Scalability analysis

Operating runner fleets at enterprise scale:

• Queue SLO: target p95 queue wait < 2 min — scale maxRunners accordingly.
• Golden AMI/image: weekly rebuild from Packer/Tekton — patch drift kills compliance.
• Remote caching: Bazel remote cache, sccache, Docker registry layer cache — ephemeral without cold penalty.
• Multi-pool routing: GPU pool, linux-small, linux-large, windows — job labels route efficiently.
• Cost visibility: tag runner nodes; compare $/minute vs hosted — include engineer on-call cost.

Staff engineer insights

• If untrusted code can run on a runner with prod network access, you don't have CI — you have a remote code execution vulnerability.
• Cattle runners: automate replacement, never SSH to fix — debug via logs and repro on fresh instance.
• Autoscale min=0 saves money but adds cold start — warm pool of 2-3 for business hours if SLAs demand.
• Measure queue wait, not just job duration — users feel queue time as "CI is slow."

Best practices

• Ephemeral runners by default — one job, one instance, terminate after.
• Separate runner groups: public-untrusted (hosted), internal-standard, internal-privileged (VPC).
• Pre-install tools in image — don't curl bash in every job.
• Use OIDC from runner jobs to cloud — no static keys on runner disk.
• Automate runner image updates — monthly CVE patch cycle minimum.

Common mistakes

• Shared home directory between jobs — .npmrc or .docker credentials leak to next job.
• Running self-hosted for all repos including public OSS — unnecessary attack surface.
• maxRunners unlimited — runaway matrix bankrupts cloud account.
• Privileged DinD on shared runners — container escape to host is matter of when.

Advanced interview questions

Hands-on exercise

Exercise: Write a threat model table: rows = fork PR, internal PR, main deploy; columns = runner pool, network access, credential type. Identify one critical gap.

# Template
| Scenario      | Runner pool | Network      | Credentials |
|---------------|-------------|--------------|-------------|
| fork PR       | ?           | ?            | ?           |
| internal PR   | ?           | ?            | ?           |
| main deploy   | ?           | ?            | ?           |

Summary

You understand self-hosted runners: isolation through ephemeral infrastructure, autoscaling on queue depth, and pet-vs-cattle operations — the platform engineering decisions that keep private CI both fast and defensible.