Prompt Optimization

Introduction

Prompt optimization turns prompt engineering from art into engineering: define an eval set, measure baseline, iterate with versions, ship the winner. Tools: PromptLayer, Braintrust, LangSmith, OpenAI Evals, Anthropic Workbench.

Beginner analogy: A/B testing for prompts — keep what wins on the metric you care about.

Understanding the topic

Core concepts:

• Build a fixed eval set of 50-200 cases.
• Score with rules, regex, LLM-as-judge or human review.
• Iterate: tweak prompt → re-run → compare → keep winner.
• Track cost AND quality — sometimes cheaper prompts win.
• Automate evals in CI before any prompt change.

Syntax reference

Visual workflow / architecture:

prompt v1 ─► run on evals ─► score 71%
prompt v2 ─► run on evals ─► score 78%   ✅ ship
prompt v3 ─► run on evals ─► score 76%
prompt v4 ─► run on evals ─► score 84%   ✅ ship

Real-world use

Anthropic, OpenAI and Google all ship internal eval harnesses; every serious AI startup builds one.

Best practices

• Eval set must reflect production traffic.
• Use LLM-as-judge with rubrics for open-ended outputs.

Common mistakes

• Shipping prompt changes without evals — silent quality drops.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Three components of a prompt eval system?
• Q2. When use LLM-as-judge vs human review?
• Q3. Scenario: a 'better' prompt costs 4x — how do you decide?