Embeddings (RAG)

Introduction

Embeddings power RAG. Choosing the right embedding model and dimension affects accuracy, storage cost, and search speed. Get this layer right and the rest falls into place.

Beginner analogy: Like choosing fingerprint resolution — too low and you confuse people, too high and storage/cost balloon.

Understanding the topic

Core concepts to understand:

• OpenAI text-embedding-3-small — 1536-dim, cheap, great default.
• OpenAI text-embedding-3-large — 3072-dim, top accuracy.
• Cohere embed-v3, Voyage AI — strong alternatives.
• BGE, E5 — open-source, free to self-host.
• Always normalise (L2) for accurate cosine similarity.

Syntax reference

Visual workflow / architecture:

Document chunk
       │
       ▼  embedding model
[0.12, -0.84, ..., 0.07]  (1536 floats)
       │
       ▼  store
   Vector DB (Pinecone / pgvector / Chroma)

Real-world use

OpenAI's embedding API processes billions of vectors monthly across customers. Most production RAG uses one of: OpenAI, Cohere, Voyage, BGE.

Best practices

• Pick one model and stick with it across your corpus.
• Normalise vectors.
• Re-embed if you upgrade models.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Name three popular embedding models.
• Q2. Why normalise embeddings?
• Q3. What happens if you mix two embedding models?