Token Prediction

Introduction

An LLM generates text one token at a time. After processing your prompt, it outputs a probability distribution over the entire vocabulary, samples one token, appends it, and repeats. This loop — called autoregressive decoding — is what produces fluent text.

Beginner analogy: Like writing an essay one syllable at a time, each based on everything you've written so far.

Understanding the topic

Core concepts to understand:

• Autoregressive: each token depends on all previous tokens.
• Temperature controls randomness (0 = deterministic, 1+ = creative).
• Top-k / top-p sampling restricts choices.
• Greedy always picks the most likely token (boring but safe).

Syntax reference

Visual workflow / architecture:

Prompt: "The capital of France is"
              │
              ▼
         {Paris: 0.92, London: 0.03, Lyon: 0.02, ...}
              │ sample
              ▼
            "Paris"
              │ append
              ▼
"The capital of France is Paris"
              │
              ▼  next iteration → "."
              ...

Real-world use

Every chat completion you've ever seen is an autoregressive loop. Speculative decoding speeds this up by predicting multiple tokens with a draft model then verifying with the big model.

Best practices

• Use temperature=0 for facts, 0.7 for creativity.
• Set max_tokens to control cost.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. How does autoregressive generation work?
• Q2. What does temperature do?
• Q3. Difference between greedy and top-p sampling?