Architectural Trade-offs

Introduction

Architectural trade-offs are the explicit choices between competing quality attributes — consistency vs availability, speed vs safety, cost vs performance. Google staff engineers document what was optimized, what was sacrificed, and how the org will detect a wrong bet — trade-off illiteracy is how systems accumulate contradictory optimizations.

Real production story

Two Google Cloud teams debated Spanner vs Cloud SQL for a new control-plane metadata store. Team A optimized for strong consistency and global reads; Team B optimized for cost and familiar ops. Without a structured trade-off review, they shipped a hybrid that used Spanner for writes and async replica to PostgreSQL for reads — creating dual sources of truth. A partition event surfaced stale reads that authorized incorrect IAM changes. The remediation was not "pick Spanner" — it was an ADR that named CAP position, RPO/RTO, reconciliation, and a single authoritative store with documented read staleness bounds.

Business problem

Google engineers face trade-offs on every tier-0 decision. Hidden trade-offs — "we get both!" — become production contradictions that surface as security incidents or cost overruns.

• Decision velocity: Unresolved trade-offs block projects or produce compromised designs that fail neither goal well.
• Organizational alignment: Teams optimize different attributes without knowing they conflict until integration.
• Accountability: Postmortems need recorded rationale — not memory of a hallway conversation.

Architecture overview

Trade-off framework: List alternatives → score against prioritized attributes → choose → document dissent → define verification metrics and revisit trigger.

• ATAM-style scenarios: Stimulus → response → measure for each attribute.
• Reversibility: One-way doors get more review; two-way doors get experiments.
• Cost modeling: Include infra, toil, and coordination cost — not just latency.
• Dissent: Record minority opinions — they often predict failure modes.

Architecture motivation

Trade-off analysis is core staff work: Architecture is choosing among imperfect options under constraints — not finding the mythical best pattern.

• Force: Quality attributes conflict; resources are finite; time-to-market pressures real.
• Constraint: Decisions must be reversible where possible — option value matters.
• Outcome: ADRs capture alternatives, decision, and measurable success/failure criteria.

Internal architecture

Trade-off decision workflow embedded in Google design reviews:

• "Do nothing" must be a scored option — status quo has cost too.
• Revisit triggers: SLO burn, 2× cost, team size threshold — not calendar only.

Problem statement + prioritized QAs (ranked 1–5)
   ↓
Options (min 3 — including "do nothing")
   ↓
Trade matrix (attribute × option scores)
   ↓
Risk register (failure modes per option)
   ↓
Decision + ADR (chosen, rejected, revisit trigger)
   ↓
Verification plan (metrics proving bet)
   ↓
Quarterly ADR revisit in architecture forum

Data flow

Trade-offs affect data paths directly: Choosing eventual consistency changes read path; choosing sync coupling changes failure propagation — document on data-flow diagrams.

• Write trade-off: Strong consistency path — higher latency, lower anomaly risk.
• Read trade-off: Stale replica — lower latency, explicit staleness SLA to clients.
• Async trade-off: Better availability, harder debugging — invest in trace + reconciliation.

System design diagram

Two diagrams show the Architectural Trade-offs topology and the primary request/event path used in production at scale.

Production code example

ADR template with trade matrix — YAML front matter for Google-style architecture repo:

• Machine-readable front matter enables ADR lint in CI — missing revisit trigger fails merge.
• Dissent field preserves institutional memory when bet fails.

# docs/adr/4021-iam-metadata-store.md
---
adr: 4021
status: accepted
deciders: [staff-platform, security, finops]
revisit_trigger: "p99_read_ms > 50 OR monthly_cost > $120k"
quality_attributes:
  - { name: consistency, priority: 1, target: "linearizable reads for authz" }
  - { name: availability, priority: 2, target: "99.95% regional" }
  - { name: cost, priority: 3, target: "< $100k/mo at current scale" }
alternatives:
  - name: cloud_sql_primary
    rejected_because: "failover RPO violates authz correctness SLO"
  - name: spanner_plus_pg_replica
    rejected_because: "dual source of truth — incident 2024-Q2"
  - name: spanner_authoritative_with_cache
    chosen: true
verification:
  - metric: authz_staleness_violations_total == 0
  - metric: iam_read_p99_ms < 40
dissent: "FinOps prefers Cloud SQL — accept 20% cost delta for correctness"
---

Enterprise case study

Google Cloud IAM metadata store trade-off review: Hybrid Spanner/PostgreSQL caused stale authorization reads.

• Before: Implicit trade-offs; dual stores; IAM bug class from read lag.
• Decision: Single authoritative Spanner; read-through cache with max staleness header; ADR-4021.
• After: Authorization anomalies eliminated; 15% cost increase accepted with finance sign-off.

Trade-offs

• Consistency vs availability: Classic CAP — pick per operation, not per system.
• Build vs buy: Buy for commodity; build when differentiation or control is strategic — include exit cost.
• Centralize vs federate: Platform efficiency vs team autonomy — federate with guardrails.
• Speed vs correctness: Payments and IAM lean correctness; recommendations lean speed — do not mix defaults.

Security considerations

Security trade-offs are explicit: Fail-open vs fail-closed, encryption overhead, zero-trust complexity — document accepted risk.

• Fail-closed: Better integrity; availability hit during auth outage — business must accept.
• Encryption: CPU cost vs compliance — hardware acceleration and selective encryption.
• Third-party trust: Faster delivery vs supply chain risk — vendor review in trade matrix.

Scalability analysis

Trade-offs shift with scale: A decision correct at 1k QPS may be wrong at 1M — revisit triggers must include growth milestones.

• Cost crossover: Managed service cheaper until scale — model crossover point.
• Org crossover: Monolith maintainability breaks at team count N — Conway triggers review.
• Geo crossover: Single-region simplicity vs multi-region compliance requirements.

Failure scenarios

Trade-off failures: Wrong CAP choice during partition; cost-optimized design that cannot meet surge; over-engineered platform no team adopts.

• Worst of both worlds: Hybrid without reconciliation — fix by picking authoritative source.
• Premature optimization: Microservices at 5 engineers — merge or modular monolith.
• Analysis paralysis: No decision ships — time-box trade study with executive sponsor.

Staff engineer insights

• Staff engineers who say "it depends" without naming what it depends on are avoiding the job — pick attributes and rank them.
• Every ADR needs a "we were wrong if" section — trade-offs without falsification criteria are opinions.
• The rejected alternative is as important as the chosen one — it prevents re-litigation every year.

Interview questions

Architecture review questions

• Are at least three alternatives documented including status quo?
• Are quality attributes ranked numerically for this decision?
• Does ADR include rejected options and dissenting views?
• Are verification metrics and revisit triggers defined?
• Is cost (infra + toil) included in trade matrix?
• Does data-flow diagram reflect chosen consistency/availability position?

Summary

Architectural trade-offs at Google scale require explicit attribute ranking, scored alternatives, ADRs with dissent and revisit triggers, and verification metrics — because undocumentated trade-offs become production contradictions and repeated postmortems.