Context Windows Explained

Introduction

The context window is the maximum number of tokens an LLM can consider in one request — both your prompt and its reply. GPT-4 Turbo offers 128k, Claude 3.5 Sonnet 200k, Gemini 1.5 Pro 2M. Beyond the limit, tokens are silently dropped or rejected.

Beginner analogy: A context window is the model's short-term memory. Fits a paragraph? Easy. Fits a book? Harder. Try to fit two books? The model forgets the start by the time it reaches the end.

Understanding the topic

Core concepts to understand:

• Window includes input + output. Forget this and your reply gets truncated.
• Models perform worse near the middle of very long contexts (lost in the middle).
• Bigger window ≠ better answer — relevance still beats raw length.
• Pricing scales with token count — long contexts are expensive.
• RAG exists to keep the window small & focused.

Syntax reference

Visual workflow / architecture:

Context Window (e.g. 128k tokens)
┌──────────────────────────────────────────┐
│ system prompt          ▓                 │
│ few-shot examples      ▓▓                │
│ retrieved context      ▓▓▓▓▓▓            │
│ chat history           ▓▓▓                │
│ user message           ▓                 │
│ → reserved for reply   ░░░░ (≤ 4k)       │
└──────────────────────────────────────────┘
                       ↑ exceeded? oldest tokens dropped

Real-world use

Cursor and GitHub Copilot use sophisticated context selection to fit only the most relevant code into the model's window. They don't dump the whole repo.

Best practices

• Always leave ~10–20% of the window for the reply.
• Place key instructions at the start AND repeat them at the end.
• Use RAG to retrieve only the most relevant chunks, not entire docs.

Common mistakes

• Sending a 200k-token request and wondering why the model ignored the middle.
• Forgetting that streaming counts against your window too.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. What is a context window?
• Q2. What is the 'lost in the middle' effect?
• Q3. How do you handle a chat that exceeds the window?
• Q4. Why is a larger window not always better?