Attention Mechanism

Introduction

Attention lets the model decide, for each token, which other tokens matter most. It's how 'it' in a sentence figures out what noun it refers to. Mathematically, it's a weighted sum where weights come from token similarity.

Beginner analogy: Reading 'The cat sat on the mat. It was fluffy.' — your brain instantly attends to 'cat' when processing 'It'. Self-attention does this with math.

Understanding the topic

Core concepts to understand:

• For each token: compute Query, Key, Value vectors.
• Score = softmax(Q · Kᵀ / √d) — how relevant is each token?
• Output = weighted sum of Value vectors.
• Multi-head = multiple attention computations in parallel.

Syntax reference

Visual workflow / architecture:

Token "it"  ─── Q ──┐
                         │
All tokens ─── K, V ─────┼──> attention scores → weighted V
                         │            │
                         ▼            ▼
                  softmax(Q·Kᵀ)   context vector for "it"

Real-world use

Attention is what gives LLMs long-range coherence — keeping track of who's who and what's what across thousands of tokens. Flash Attention is a faster implementation now used in every major LLM.

Best practices

• Don't implement from scratch — use PyTorch or vendor APIs.
• Watch the visualisation in the original paper or Karpathy's video.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. What does attention compute?
• Q2. What are Q, K, V?
• Q3. Why divide by √d in the softmax?