AI Debugging Tasks

Introduction

Debugging agents is its own skill. These tasks teach you to read traces, isolate variables and apply structured hypotheses.

Understanding the topic

Core concepts:

• Trace says the planner produced an empty plan. Why?
• Agent calls the wrong tool 5% of runs. Diagnose.
• Cost spike at midnight — find the cause.
• Latency p95 doubled after a model upgrade.
• Hallucinated citation in RAG output.
• Tool returns 200 but agent loops anyway.
• Memory store fills with duplicate facts.
• Workflow stuck in 'pending' for hours.
• Multi-agent debate spirals into pleasantries.
• Eval pass rate dropped 10% — bisect.

Syntax reference

Visual workflow / architecture:

trace → hypothesis → minimal repro → fix → re-eval

Real-world use

Real debugging logs from AI platforms — Langfuse / Braintrust / Helicone replays.

Best practices

• Always reproduce on a minimal trace before fixing.
• Bisect on the eval set after fixes.

Common mistakes

• Fixing a symptom without a repro.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Debugging order for an agent failure?
• Q2. Why bisect on the eval set?
• Q3. Scenario: success rate dropped 10% — first artefact you open?