AI Monitoring

Introduction

Agents fail in new ways: silent hallucinations, infinite loops, runaway cost. Monitoring needs metrics, traces, and evals — not just logs.

Beginner analogy: NASA mission control — telemetry, alerts, replay, eval.

Understanding the topic

Core concepts:

• Metrics: success rate, p95 latency, $/run, tokens/run.
• Traces: full step-by-step with inputs/outputs.
• Eval: nightly golden set + production sampling.
• Alerts on regressions.
• Drift detection on tool / model behaviour.

Syntax reference

Visual workflow / architecture:

trace ─► OpenTelemetry / Langfuse / Braintrust
metric ─► Prometheus / Datadog
eval   ─► Braintrust / OpenAI Evals
alert  ─► PagerDuty / Slack

Real-world use

Langfuse, Braintrust, Helicone, Arize Phoenix — purpose-built for AI monitoring.

Best practices

• Trace EVERY run.
• Sample evals on prod traffic, not just CI.

Common mistakes

• Logs without traces — useless for agents.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Four AI-specific metrics.
• Q2. Trace vs log — difference?
• Q3. Scenario: success rate dropped 10%. Three investigation steps?