AI Debugging Tasks

Introduction

Debugging AI is harder than debugging code — outputs are stochastic. These tasks teach systematic AI debugging.

Understanding the topic

Core concepts to understand:

• Diagnose: chatbot suddenly hallucinates after a model upgrade.
• Fix: agent loops infinitely on certain inputs.
• Investigate: RAG returns irrelevant chunks for some queries.
• Cost spike: bill 5×'d overnight — find the cause.
• JSON output: occasional malformed responses break the parser.

Syntax reference

Visual workflow / architecture:

Bug
  │
  ▼
Reproduce on a fixed seed input
  │
  ▼
Inspect logs (full prompt + completion)
  │
  ▼
Form hypothesis (chunking? retrieval? model?)
  │
  ▼
Change one thing → re-run → measure
  │
  ▼
Add eval to prevent regression

Real-world use

These are real on-call scenarios at any AI startup.

Best practices

• Always reproduce first.
• Change one variable at a time.
• Add a test that catches the bug forever.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Walk through how you'd debug a sudden hallucination spike.
• Q2. Tools you'd use?
• Q3. How do you prevent regression?