Tokens Explained

Introduction

A token is the smallest chunk of text an LLM processes — roughly ¾ of an English word. Every API call is billed in tokens (in and out), every model has a token limit, and every prompt change is really a token change.

Beginner analogy: Tokens are like Lego bricks. Common words ('the') are single bricks; rare words ('pseudonymisation') are 3–4 bricks. The model thinks in bricks, not letters or words.

Understanding the topic

Core concepts to understand:

• 1 token ≈ 4 characters ≈ 0.75 English words.
• Numbers, code, emojis and non-English text use more tokens.
• Pricing is per-million tokens — input is cheaper than output on most models.
• Use OpenAI's tiktoken or Anthropic's tokenizer to count before sending.
• Reducing tokens reduces cost & latency proportionally.

Syntax reference

Visual workflow / architecture:

Input:  "Generative AI is amazing!"
            ↓ tokenizer
[Generative][ AI][ is][ amazing][!]
       ↓
   5 tokens (input)

Cost = tokens_in × in_price + tokens_out × out_price

Real-world use

A company running 10M GPT-4o calls/day at 1000 tokens each pays roughly $25K/day. Cutting prompts 20% saves $5K/day — that's why token discipline is real money.

Best practices

• Count tokens during development with a tokenizer SDK.
• Prefer short synonyms when meaning is identical.
• Avoid duplicate context — say it once, well.

Common mistakes

• Pasting whole files when only a function is needed.
• Repeating the same instructions in system AND user messages.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Define a token.
• Q2. Why is non-English text usually more expensive?
• Q3. How would you reduce token usage by 30% on a chat app?
• Q4. Where does Anthropic's pricing model differ from OpenAI's?