AI Retrieval Systems

Introduction

Retrieval is the 'R' in RAG and the most under-invested part of most pipelines. Great retrieval makes mediocre LLMs shine; bad retrieval makes great LLMs hallucinate.

Beginner analogy: Garbage in, garbage out — even GPT-4 can't answer correctly if you give it the wrong document.

Understanding the topic

Core concepts to understand:

• Hybrid search (vector + BM25) for recall.
• Re-ranking with cross-encoders for precision.
• Metadata filtering (tenant, language, date).
• Query rewriting — expand or rephrase queries with an LLM.
• Evaluate with recall@k and NDCG.

Syntax reference

Visual workflow / architecture:

Query
   │
   ▼  query rewrite (LLM)
   ▼  expanded query
   │
   ├──> Vector search (top 50)
   ├──> BM25 search   (top 50)
   │       │
   │       ▼ merge
   │
   ▼ Re-rank (cross-encoder) → top 5
   │
   ▼ pass to LLM

Real-world use

Perplexity and You.com invest heavily in retrieval. Cohere Rerank is a popular hosted re-ranker.

Best practices

• Always use hybrid + re-rank in production.
• Evaluate retrieval separately from generation.
• Tune chunking and metadata for your domain.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Why is retrieval the most important RAG component?
• Q2. What is a re-ranker and why use one?
• Q3. How do you evaluate retrieval?