Tokens & Embeddings

Introduction

Two foundations of every LLM and agent: tokens (the units the model sees) and embeddings (numeric vectors that capture meaning). You'll touch both every day — pricing is per token, RAG is built on embeddings.

Beginner analogy: tokens are LEGO bricks the model snaps together. Embeddings are GPS coordinates for meaning — words near each other in 'meaning space' have similar coordinates.

Understanding the topic

Core concepts:

• Token ≈ 3-4 characters or roughly ¾ of an English word.
• Tokenizer (BPE / SentencePiece) splits text deterministically.
• Pricing & context limits are per token, not per word.
• Embedding: a fixed-size vector (e.g. 1536 floats) representing meaning.
• Cosine similarity between embeddings = semantic similarity.

Syntax reference

Visual workflow / architecture:

Text: "Cancel my flight to Tokyo"
   │ tokenize
   ▼
[Cancel][ my][ flight][ to][ Tokyo]
   │
   │ embed (separately for RAG)
   ▼
[0.12, -0.04, 0.88, …, 0.03]  ← 1536-d vector
                                      │
                                      ▼
                           Vector DB search
                                      │
                                      ▼
                       Most similar docs / past tickets

Real-world use

Every chatbot bill is computed in tokens. Every RAG system, semantic search, recommendation engine — all run on embeddings.

Best practices

• Use tiktoken / Anthropic tokenizers to estimate cost before sending.
• Pick the cheapest embedding model that meets quality (text-embedding-3-small often wins).
• Re-embed when you change models — vectors aren't compatible across providers.

Common mistakes

• Counting words instead of tokens → underestimated bills.
• Mixing embeddings from two models in the same DB.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. How many tokens ≈ 1 English word?
• Q2. What is an embedding?
• Q3. Why is cosine similarity used?
• Q4. Scenario: your RAG returns garbage. Could embedding choice be the cause?