AI Model Training Basics

Introduction

Training an LLM happens in three stages: pre-training on the entire internet (predict the next token), supervised fine-tuning on curated examples, and reinforcement learning from human feedback (RLHF) to align responses with what humans actually want.

Beginner analogy: Pre-training is like reading the whole library. Fine-tuning is like attending a specialist course. RLHF is like having a tutor say 'good answer' or 'rewrite that' until you sound polite and helpful.

Understanding the topic

Core concepts to understand:

• Pre-training — months on thousands of GPUs, trillions of tokens.
• SFT — supervised fine-tuning on Q&A or instruction pairs.
• RLHF / DPO — humans rank responses; model learns preferences.
• Fine-tuning (you) — adapt a base model to your domain with small data.
• LoRA & QLoRA — cheap fine-tuning with adapter layers.

Syntax reference

Visual workflow / architecture:

Stage 1: PRE-TRAINING       (trillions of tokens · months)
              │
              ▼
Stage 2: SFT                 (50k–100k Q&A examples)
              │
              ▼
Stage 3: RLHF / DPO          (human preference ranking)
              │
              ▼
       Released base model
              │
              ▼
Stage 4: YOUR FINE-TUNING    (LoRA on domain data)

Real-world use

Training GPT-4 reportedly cost >$100M in compute. Most companies never pre-train — they fine-tune Llama or use OpenAI's API. Bloomberg GPT is a rare full pre-train on financial data.

Best practices

• Don't pre-train. Fine-tune or prompt-engineer.
• Use LoRA/QLoRA — 100× cheaper than full fine-tuning.
• Evaluate on held-out test data; never trust training loss alone.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. What are the three stages of LLM training?
• Q2. What is RLHF and why does it matter?
• Q3. What is LoRA?
• Q4. When should you fine-tune vs prompt-engineer?