Tokens & Embeddings

Introduction

LLMs don't see words — they see tokens (small chunks of text) converted into embeddings (lists of numbers that capture meaning). Understanding these two concepts is essential because they drive cost, context limits and search quality.

Beginner analogy: A token is a syllable; an embedding is a coordinate on a 'meaning map' where similar words sit close together. 'King' and 'queen' end up near each other; 'pizza' is far away.

Understanding the topic

Core concepts to understand:

• Token ≈ ¾ of a word in English. 1000 tokens ≈ 750 words.
• Pricing is per token (input + output) — long prompts get expensive fast.
• Embedding = vector (e.g. 1536 numbers) representing meaning.
• Similar meanings → similar vectors → cosine similarity ≈ 1.
• Embeddings power semantic search, RAG, recommendation.

Syntax reference

Visual workflow / architecture:

Text:  "Hello world"
         │
         ▼  tokenizer (BPE)
      [15496, 1917]
         │
         ▼  embedding model
   [0.12, -0.84, 0.33, ..., 0.07]   ← 1536-dim vector
         │
         ▼  store in Vector DB
       Pinecone / Chroma / pgvector

Search:  query → embed → find nearest vectors → return original text

Real-world use

OpenAI text-embedding-3-small (1536-dim) is the workhorse for RAG. Cohere embed-v3 and Voyage AI are popular alternatives. Every modern AI search system — Perplexity, ChatGPT memory, Notion AI Q&A — runs on embeddings.

Best practices

• Always count tokens before sending — use tiktoken for OpenAI.
• Pick an embedding model and stick with it; mixing models breaks similarity.
• Normalise embeddings (L2) for accurate cosine similarity.

Common mistakes

• Embedding very long documents in one shot — chunk first!
• Forgetting that token cost includes output as well as input.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. What is a token?
• Q2. What is an embedding and what is it good for?
• Q3. Why does cosine similarity matter for embeddings?
• Q4. How would you estimate the cost of a 50-page document?