Prompt Testing Methods

Introduction

Prompts need both unit-style tests (single input → expected behaviour) and regression tests (the whole eval set). Modern tooling makes both easy.

Beginner analogy: Like software — unit tests for behaviours, integration tests for features.

Understanding the topic

Core concepts to understand:

• Unit: 'classify_refund' returns category='refund'.
• Regression: full test set, scored, must beat last version.
• Adversarial: prompt-injection attempts, weird inputs.
• Smoke tests in production traffic (canary prompts).
• Frameworks: Promptfoo, LangSmith, Braintrust, OpenAI Evals.

Syntax reference

Visual workflow / architecture:

tests/
  prompts/
    classify_refund.test.json
    summarise_ticket.test.json
  regressions/
    full_set_v3.json
  adversarial/
    injection_attacks.json

Real-world use

Mature AI teams run prompt tests in CI on every PR, just like code tests.

Best practices

• Block PR on regression failures.
• Re-run adversarial tests monthly.
• Pair tests with telemetry from prod.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Unit vs regression tests for prompts?
• Q2. What are adversarial prompt tests?
• Q3. Tools used?
• Q4. CI for prompts — how?