Error Analysis

Introduction

Error analysis bucketises failures from your eval set — what kind of mistakes is the prompt making? Each bucket suggests a different fix.

Beginner analogy: Like a doctor categorising symptoms before treating — diagnosis precedes prescription.

Understanding the topic

Core concepts to understand:

• Bucket failures: format, factual, missing constraint, off-topic, refused too much.
• Sort buckets by impact & ease of fix.
• Each bucket → a targeted prompt edit or new example.
• Repeat until top buckets are below threshold.

Syntax reference

Visual workflow / architecture:

Failure types        Count   Fix
─────────────────────────────────────
Wrong format         12      Add JSON example
Hallucinated source   8      Add 'cite source' rule
Too long              5      Add length limit
Refused valid input   3      Soften refusal rule

Real-world use

Standard practice at every AI startup with a working eval pipeline.

Best practices

• Hand-label 30–50 failures before automating.
• Track each bucket trend over versions.
• Pair with regression tests — fixing bucket A shouldn't break bucket B.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. What is error analysis?
• Q2. How bucket failures?
• Q3. How decide what to fix first?
• Q4. Risk of fixing one bucket?