Transformer Architecture

Introduction

Every modern LLM is a transformer — a neural net built around self-attention. Attention lets each token 'look at' every other token in the context and decide which ones matter, producing rich, context-aware representations.

Beginner analogy: reading a sentence, your eyes jump back to earlier words for context ("it referred to the cat"). Attention is that jumping, mathematised.

Understanding the topic

Core concepts:

• Self-attention: every token attends to every other.
• Multi-head: many parallel attention 'views'.
• Stack of N layers (12 in BERT-base, 80+ in GPT-4).
• Positional encodings tell the model token order.
• Decoder-only transformers (GPT, Claude, Llama) generate text.

Syntax reference

Visual workflow / architecture:

Input tokens
    │
    ▼
┌──────────────┐
│  Embedding   │
└──────┬───────┘
       ▼
┌────────────────────┐
│ Multi-Head         │
│ Self-Attention     │ ◄── core innovation
└──────┬─────────────┘
       ▼
┌──────────────┐
│ Feed-Forward │
└──────┬───────┘
       ▼
┌──────────────┐
│ Repeat × N   │  (12 → 80+ layers)
└──────┬───────┘
       ▼
   Next-token logits

Real-world use

The 2017 paper Attention Is All You Need launched the era — BERT, GPT, T5, Llama, Claude all use it.

Best practices

• You don't need to implement attention to build agents.
• Knowing it explains context-length costs, attention sinks, KV cache.

Common mistakes

• Memorising the math without understanding the implications for prompting.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Why is attention so important?
• Q2. Why do longer contexts cost quadratically more?
• Q3. Decoder-only vs encoder-decoder transformers?
• Q4. What does multi-head buy you?