Latency Optimization

Introduction

Latency optimisation makes AI features feel snappy. The biggest lever is streaming; the second is choosing smaller, faster models.

Beginner analogy: Like serving food while it cooks vs waiting for the whole meal — streaming feels faster even at the same throughput.

Understanding the topic

Core concepts to understand:

• Stream responses — perceived latency drops 50%+.
• Pick faster models for latency-sensitive UX (Haiku, Mini, Flash).
• Shorter prompts = faster first token.
• Pre-compute / cache common requests.
• Run side-by-side: small fast model for streaming, big model for offline rerun.

Syntax reference

Visual workflow / architecture:

User clicks → first token in 200ms (streaming) → full reply in 3s
vs
User clicks → blank for 3s → full reply at once

Real-world use

ChatGPT & Claude.ai stream by default — without it the product would feel broken.

Best practices

• Always stream user-facing replies.
• Use smaller model when time-to-first-token matters more than depth.
• Pre-warm caches for predictable queries.

Hands-on exercise

Interview preparation — practice these questions:

• Q1. Why does streaming feel faster?
• Q2. Smaller vs bigger model — when?
• Q3. Pre-compute strategies?
• Q4. How measure latency?