GitOps Basics

Introduction

GitOps declares cluster (and often infra) desired state in git — manifests, Helm charts, Kustomize overlays — and lets controllers continuously reconcile live state to match. Argo CD and Flux watch repos; on drift they sync or alert. Deploy becomes `git merge`; rollback becomes `git revert`. The reconcile loop — compare desired (git) vs actual (cluster), act on diff — is the same pattern as Kubernetes controllers, applied to your entire delivery model.

Staff engineers treat git as the single source of truth, enforce PR review on prod manifest changes, and understand when drift detection vs auto-sync is appropriate. GitOps complements CI (build artifact) — CI pushes digest to git or triggers image updater; GitOps deploys what git declares.

The story

Before GitOps, fourteen engineers had kubectl access to prod. A Friday typo in a replica count caused an outage; nobody knew who applied it — audit log empty. Platform team deployed Argo CD: all prod changes via PR to `deploy/apps/checkout-api/overlays/prod`. Drift from manual kubectl corrected or blocked. Rollback of a bad release was git revert + sync — three minutes with full audit trail. One engineer tried kubectl patch during an incident; Argo flagged OutOfSync within 30 seconds and auto-sync restored declared state — teaching the team to revert git instead.

Understanding the topic

GitOps core loop — four concepts:

• Declarative config: desired state in git (YAML, Helm, Kustomize) — not imperative kubectl scripts.
• Reconcile loop: controller compares git commit vs cluster; applies diff (sync) or reports drift.
• Argo CD: UI/CLI/API for Application CR; sync policies, health assessment, multi-cluster; Image Updater for digest bumps.
• Flux: GitOps Toolkit (source, kustomize, helm controllers); native CNCF; often GitHub Actions triggers vs Argo UI.
• Drift: manual kubectl edit creates OutOfSync — detect via alert or auto-heal depending on policy.

Internal architecture

Argo CD reconcile architecture:

Git repo (manifests/)
   checkout-api/overlays/prod/kustomization.yaml
   image: myreg/checkout-api@sha256:deadbeef…
   ↓ poll / webhook
Argo CD Application CR
   spec: { source: repo, path, destination: cluster/namespace }
   syncPolicy: { automated: { prune: true, selfHeal: true } }
   ↓ compare + kubectl apply / helm template
Kubernetes cluster (actual state)
   ↓ diff detected
OutOfSync → Sync → Healthy (or Degraded + alert)
Rollback: git revert digest commit → auto-sync

Visual explanation

Two diagrams show where GitOps Basics lives in the delivery path and how teams implement it in production.

Step-by-step explanation

1. Structure repo: app chart/base + overlays per env (dev/staging/prod) with Kustomize or Helm.
2. Register Argo CD Application pointing to overlay path and target cluster/namespace.
3. CI builds artifact; updates image digest in git via PR (or Argo Image Updater from registry).
4. Merge PR; Argo syncs; verify Application Healthy in UI and SLIs in observability.
5. Rollback: git revert digest commit; sync; investigate drift alerts for manual kubectl bypass.

Production implementation

Argo CD Application + Kustomize prod overlay:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://github.com/org/deploy-manifests.git
    targetRevision: main
    path: apps/checkout-api/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=false
---
# overlays/prod/kustomization.yaml
images:
- name: myreg/checkout-api
  digest: sha256:deadbeef...

# Rollback
# git revert <digest-bump-commit>
# argocd app sync checkout-api --prune

Execution workflow

Real-world use

Weaveworks coined GitOps; CNCF adopted Flux as graduated project. Intuit, Adidas, and Tesla publish Argo CD at scale. Terraform + GitOps (Crossplane, Atlantis) extends the pattern to cloud infra. Most Kubernetes-native orgs treat GitOps as default CD by 2025.

Enterprise use cases

Multi-cluster GitOps with ApplicationSet at a SaaS platform — 12 EKS clusters from one repo.

• ApplicationSet: generator from cluster registry; each cluster gets overlay path `overlays/{{name}}`.
• Promotion: PR moves digest from staging overlay to prod overlay — same artifact.
• Policy: OPA Gatekeeper + Argo CD pre-sync hooks block unsigned images.
• Drift: selfHeal true in dev; selfHeal false in prod — prod drift pages on-call before auto-fix.

Production case study

Platform team GitOps migration from Jenkins kubectl scripts — 200 microservices.

• Before: SSH kubectl from CI; no drift detection; rollback = re-run old Jenkins job (20 min).
• After: Argo CD + Kustomize overlays; rollback git revert + sync (3 min); OutOfSync alerts.
• Outcome: kubectl prod access revoked for app teams; change failure rate down 40%.

Trade-offs

• Benefit: auditable deploys, PR-reviewed infra, easy rollback via revert, drift visibility.
• Cost: repo structure learning curve; sync delays vs direct kubectl; secret management complexity.
• selfHeal risk: auto-healing prod hides manual incident fixes — policy per environment.
• Secret sprawl: never plaintext secrets in git — use Sealed Secrets, External Secrets, SOPS.

Security implications

Git repo becomes crown jewel — protect with branch protection, CODEOWNERS, and signed commits.

• Argo CD repo credentials scoped read-only; deploy via git merge not direct cluster admin for app teams.
• SSO + RBAC on Argo UI — prod Application sync restricted to platform role.
• Verify artifact signatures in pre-sync hook before applying manifest.
• Audit all sync events — who merged PR triggering deploy.

Scalability analysis

Large mono-repo manifests stress Argo CD reconciliation performance.

• Shard repos by domain or use ApplicationSet with path generators — avoid 500 Apps on one path.
• Webhook-triggered sync vs 3-minute poll — reduce lag for high-frequency deploys.
• Server-side apply and replace sync options for large ConfigMaps — avoid client-side apply limits.

Staff engineer insights

• GitOps is not "no CI" — CI builds artifact; GitOps deploys declared digest.
• Prod selfHeal false + drift alert is often safer than auto-healing over manual incident kubectl.
• Application health must include custom metrics — Running pods ≠ healthy checkout flow.
• Rollback = git revert, but emergency kubectl undo still valid — reconcile git within the hour.

Best practices

• Separate deploy repo from app repo — platform owns manifests; app teams propose digest PRs.
• Use Kustomize/Helm overlays per env — never duplicate entire manifest trees.
• Configure sync windows for prod — block auto-sync during business peak if needed.
• Export Argo CD metrics to Prometheus — sync failures and OutOfSync duration are platform SLOs.

Common mistakes

• Plaintext secrets in git — use Sealed Secrets or External Secrets Operator.
• selfHeal true in prod masking incident kubectl — reconcile git instead.
• Argo CD Application pointing to :latest tag — always pin digest in manifests.

Advanced interview questions

Hands-on exercise

Design GitOps repo structure for checkout-api with dev/staging/prod:

• Kustomize base + overlay layout.
• Argo CD Application YAML for prod.
• PR flow from CI digest bump to prod promotion.
• Rollback steps with git revert + argocd sync.

Summary

You understand GitOps basics: declarative git manifests, Argo CD and Flux reconcile loops, drift detection, and production rollback via git revert — integrated with CI artifact promotion.