Evaluation Frameworks

Introduction

Mature teams use evaluation frameworks — composable, model-agnostic eval suites that run in CI.

Beginner analogy: Like Jest for prompts. Repeatable, scoped, fast.

Understanding the topic

Core concepts to understand:

• Frameworks: Promptfoo, OpenAI Evals, DSPy, Braintrust, Patronus.
• Components: dataset, scorers, comparators.
• Run on every PR; gate merges on regressions.
• Visualise across versions and models.

Syntax reference

Visual workflow / architecture:

tests/
  evals/
    support_quality.yml
    extract_accuracy.yml
  datasets/
    support_v1.json
  scorers/
    contains.js
    judge.js

Real-world use

Promptfoo, OpenAI Evals, LangSmith are de-facto standards in 2026.

Best practices

• Treat eval suites as first-class repos.
• Version datasets like prompts.
• Visualise deltas between runs.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Pick a framework and justify.
• Q2. Scorer types?
• Q3. CI integration?
• Q4. Eval drift — how detect?