Prompt Optimization

Introduction

Prompt optimisation is the systematic process of improving prompts using evaluation, A/B testing and iteration. It treats prompts like code — you measure, refactor, and prove improvement before shipping.

Beginner analogy: Like SEO for prompts — small wording changes can dramatically affect output quality and cost.

Understanding the topic

Core concepts to understand:

• Build a test set of representative inputs + expected outputs.
• Score each prompt on accuracy, format compliance, latency, cost.
• A/B test variants — keep the winner.
• Use tools: Promptfoo, LangSmith, Braintrust, OpenAI Evals.

Syntax reference

Visual workflow / architecture:

Prompt v1 ──┐
Prompt v2 ──┼──> Test set (50 inputs)
Prompt v3 ──┘             │
                          ▼
              Score: accuracy · cost · latency
                          │
                          ▼
                  Pick winner → ship
                          │
                          ▼
                   Monitor in prod

Real-world use

LangSmith (LangChain) and Braintrust are the dominant prompt-eval platforms. Cursor reportedly runs thousands of evals per prompt change before shipping.

Best practices

• Always have a regression test set.
• Track cost AND quality — they trade off.
• Re-run evals every time you change models.

Common mistakes

• Optimising on vibes instead of metrics.
• Forgetting to re-test on the latest model version.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. How do you evaluate prompt quality?
• Q2. Name two prompt-eval tools.
• Q3. Why re-test prompts when the model updates?