What Makes Good Architecture

Introduction

Good architecture is not minimal lines of code or the trendiest topology — it is the set of structures that let Amazon-scale teams change the system safely while meeting explicit quality attributes. At staff level you judge architecture by whether it survives Prime Day traffic, org churn, and partial failure without heroic on-call work.

Real production story

During a Prime Day pre-warm, a catalog team merged a "small refactor" that collapsed three bounded contexts into one shared PostgreSQL schema. Checkout p99 latency jumped 4× when inventory writes contended with browse reads. The post-incident review reframed architecture: every change must declare which quality attributes it optimizes, which it sacrifices, and how SLO burn will be measured. Within two quarters, team boundaries realigned to service ownership and ADRs became a merge gate for cross-domain schema changes.

Business problem

Amazon retail ships thousands of deploys weekly across hundreds of teams. Without shared criteria for "good," local optimizations — shared tables, synchronous chains, hidden coupling — compound into revenue-impacting incidents during peak.

• Revenue at risk: A 200ms checkout regression during Prime Day can cost tens of millions in abandoned carts.
• Velocity paradox: Teams that skip architectural constraints ship faster for one quarter, then spend the next three untangling correlated outages.
• Organizational scale: New engineers must infer design intent from code archaeology unless architecture is explicit and documented.

Architecture overview

Good architecture satisfies prioritized quality attributes under real constraints, exposes trade-offs in ADRs, and embeds operability (metrics, tracing, runbooks) as first-class design — not post-incident bolt-ons.

• Definition: Structures that maximize business optionality while keeping change cost predictable as org and traffic grow.
• When to invest: When change failure rate, MTTR, or cross-team coordination dominate delivery cost.
• When to defer: Pre-PMF prototypes with <10 engineers — optimize learning speed, document intentional shortcuts.
• Litmus test: Can you degrade gracefully and explain the blast radius during an outage without opening the codebase?

Architecture motivation

Why staff architects define "good" upfront: Good architecture balances changeability, operability, and measurable quality attributes — not diagram aesthetics. The naive alternative (optimize for feature count only) works until traffic, compliance, and team count make every change a system-wide negotiation.

• Force: Independent teams must deploy without breaking neighbors — blast radius and contract clarity are non-negotiable.
• Constraint: No multi-year rewrite; evolution happens via strangler patterns and reversible decisions.
• Outcome: Designs pass a review where a new engineer explains data flow, failure mode, and SLO in ten minutes.

Internal architecture

Amazon-style layered responsibility — good architecture separates concerns with explicit contracts:

• Each horizontal slice has an owner, SLO, and on-call rotation — "good" means no shared-on-call mystery meat services.
• Dependencies flow downward; events flow outward — never hide synchronous chains across team boundaries.

Client (web / mobile / Alexa)
   ↓
Edge (Route 53 · CloudFront · ALB · API GW)
   ↓
Application services (stateless, team-owned)
   ↓
Domain boundaries (catalog · cart · payments)
   ↓
Persistence adapters (DynamoDB · RDS · cache)
   ↓
Async plane (SQS · SNS · EventBridge)
   ↓
Observability (X-Ray · CloudWatch · runbooks)

Data flow

Happy path vs architectural path: User requests enter at the edge, authenticate once, route to the owning service, persist within one consistency boundary, and emit side effects asynchronously with idempotent consumers.

• Write path: Validate → apply domain rule → single-transaction persist → outbox event → downstream projection.
• Read path: Serve from authoritative store or materialized view; never stitch cross-service reads synchronously at scale.
• Failure path: Timeouts, bulkheads, and cached fallbacks documented per endpoint — not invented during Sev-1.

System design diagram

Two diagrams show the What Makes Good Architecture topology and the primary request/event path used in production at scale.

Production code example

Architecture fitness function — automated gate Amazon platform teams wire into CI:

• Fitness functions encode "good" as executable policy — diagrams alone do not survive merges.
• Pair with ADR links in PR template so human context accompanies automated checks.

// architecture-fitness.ts — fails build if cross-domain imports detected
import { Project } from "ts-morph";

const FORBIDDEN_EDGES: Record<string, string[]> = {
  "catalog-service": ["payments-service", "fulfillment-service"],
  "cart-service": ["inventory-internal-db"],
};

export function assertArchitectureBoundaries(root: string): void {
  const project = new Project({ tsConfigFilePath: `${root}/tsconfig.json` });
  const violations: string[] = [];

  for (const [service, blocked] of Object.entries(FORBIDDEN_EDGES)) {
    const source = project.getSourceFileOrThrow(`src/${service}/index.ts`);
    for (const imp of source.getImportDeclarations()) {
      const mod = imp.getModuleSpecifierValue();
      if (blocked.some((b) => mod.includes(b))) {
        violations.push(`${service} illegally imports ${mod}`);
      }
    }
  }

  if (violations.length) {
    throw new Error(`Architecture violation:\n${violations.join("\n")}`);
  }
}

Enterprise case study

Amazon catalog modernization: A team inherited a "good enough" monolith that became the integration hub for twelve squads.

• Before: One schema change required fourteen approvers; weekly Sev-2 incidents from lock contention.
• Decision: Strangler extraction of read models, event-driven inventory sync, ADR-gated cross-domain APIs.
• After: Deploy frequency per squad 3×; checkout p99 stable through two Prime Days; MTTR dropped from hours to minutes.

Trade-offs

• Modularity vs delivery speed: Strict boundaries slow initial features but accelerate change after team count exceeds ~8 per domain.
• Consistency vs availability: Strong checkout invariants may require synchronous coordination; browse can tolerate stale reads — optimize per attribute.
• Standardization vs autonomy: Paved-road templates reduce toil; over-centralization becomes a platform bottleneck.
• Documentation vs code truth: ADRs drift — tie architecture reviews to deploy gates and diagram updates, not wiki archaeology.

Security considerations

Security is structural: Good architecture minimizes blast radius, isolates tenant data, and makes audit trails first-class.

• Identity: mTLS and IAM roles at service boundaries — not long-lived shared API keys in config repos.
• Data: PII stays in owning context; cross-service events carry references, not full payloads.
• Supply chain: Signed artifacts, dependency scanning, and least-privilege deploy roles per service.

Scalability analysis

Scale is multidimensional at Amazon: QPS, SKU cardinality, regional expansion, and team headcount all stress different parts of the design.

• Horizontal scale: Stateless tiers autoscale; stateful tiers partition by tenant, region, or product category.
• Hot keys: Flash deals and celebrity launches create partition hotspots — pre-warm caches and async inventory reservation.
• Cost architecture: Good design tracks $/transaction; inefficient sync chains show up in infra bills before they show up in latency dashboards.

Failure scenarios

Good architecture assumes failure: Dependencies slow, regions partition, and deploys introduce bugs — degraded behavior must be designed, not improvised.

• Cascade: Retry storms from catalog to inventory amplify load — circuit breakers and bulkheads at every hop.
• Partial write: Dual writes without outbox leave cart and inventory inconsistent — compensating actions and reconciliation jobs required.
• Operational blindness: Missing golden signals during Prime Day — treat observability as a quality attribute with SLOs.

Staff engineer insights

• Good architecture is boring on the happy path — if your runbooks read like hero stories, the design failed operability.
• Ask "what quality attribute wins when two conflict?" in every review; silence means the team will optimize locally and pay later.
• Amazon-scale systems fail at interfaces — spend review time on contracts, idempotency keys, and backpressure, not box colors.

Interview questions

Architecture review questions

• Are top three quality attributes ranked explicitly for this system?
• Can a new engineer trace a request end-to-end in under ten minutes from docs?
• Is degraded behavior documented when each critical dependency fails?
• Does every cross-team integration have an ADR and versioned contract?
• Will this design survive 10× traffic and 3× team size without rewrite?
• Are golden signals (latency, traffic, errors, saturation) defined with SLOs?

Summary

Good architecture at Amazon scale means structures that survive peak traffic, organizational growth, and partial failure while keeping change cost predictable. Staff architects make quality-attribute trade-offs legible, wire observability into the design, and enforce boundaries with fitness functions and ADRs.