Cost Optimization

Introduction

Cost optimisation blends model choice, prompt size, caching, and routing. Done well, a 50% cost cut is achievable without quality loss.

Beginner analogy: Like optimising a cloud bill — many small wins compound.

Understanding the topic

Core concepts to understand:

• Route easy queries to cheap models (Haiku, Mini); reserve top models for hard ones.
• Cache stable prompt prefixes (Anthropic, OpenAI support this).
• Compress retrieved context with reranking.
• Batch where latency allows (OpenAI Batch API = 50% off).
• Monitor cost-per-feature in dashboards.

Syntax reference

Visual workflow / architecture:

Easy query → cheap model
Hard query → frontier model
Cached prefix → 90% off the prefix tokens
Batch overnight → 50% off

Real-world use

Klarna saved millions yearly by routing simple queries to cheap models and reserving GPT-4 for the hard 20%.

Best practices

• Set budgets per feature.
• Show cost per request in dev tools.
• Re-evaluate model choice quarterly — pricing changes fast.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. How cut LLM cost 50%?
• Q2. What is model routing?
• Q3. When use Batch API?
• Q4. How monitor cost per feature?