What is time-to-first-token versus total latency, and why do both matter?

beginner

Re-explain

Time-to-first-token (TTFT) is the wait from sending the request until the first token comes back. Total latency is the wait until the last token arrives.

Roughly, total = TTFT + output_tokens * time_per_output_token.

TTFT depends on prompt length, queue wait and prefill, the pass that reads the whole prompt before generating. It is what the user feels in streamed chat. Total latency depends on the number of output tokens and decode speed. It matters for JSON, tool calls and batch jobs, where nothing is usable early.

The two move independently. Trimming the prompt cuts TTFT. Trimming the answer cuts total latency.

So track both at p95, per route. A fast first token followed by a slow 800-token answer still fails a voice agent. One blended number hides which half is broken.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for AI System Design One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

A route's p95 TTFT doubled overnight, but answer length didn't change. Where do you look first? Check what adds prefill or wait time: a longer prompt from history or retrieved chunks, fewer prefix cache hits, or queueing at the provider.
A JSON extraction endpoint is too slow end to end. How do you bring total latency down? Cut output tokens first with a tight schema, short fields, no prose and an output token cap. Then try a faster model or split the work into parallel calls.
How does prefix caching change TTFT, and why does it do little for total latency on long answers? A cached prefix skips most of prefill, so TTFT drops. Decode still runs once per output token, so a long answer takes about as long as before.
How would you split a latency budget across retrieval, the model call and a tool call? Set a p95 target per stage, split the model's share into TTFT and total, time out each stage, and decide what you drop when one overruns.

What you can say

Time-to-first-token is how long you wait for the first token to come back, and total latency is how long you wait for the last one.
Roughly, total latency is TTFT plus the number of output tokens times the time per output token.
TTFT comes from queue wait and prefill, which is the pass over the whole prompt, so a longer prompt means a later first token.
In streamed chat, TTFT is what the user feels. For JSON, tool calls or batch jobs nothing is usable until the end, so total latency is what counts.
The two move independently. Trimming the prompt cuts TTFT, and trimming the answer cuts total latency.
So I track both at p95 per route, because one blended number hides which half is broken, and a fast first token still fails a voice agent if the 800-token answer is slow.

Weak answers to avoid

Treats latency as one number An average over the whole request hides whether prefill or decode is slow. Split TTFT from total and report each at p95 per route.
Says streaming fixes latency Streaming only shows tokens as they are generated. A JSON parser or tool call still has to wait for the last token, so total latency stays the same.
Offers one fix for both metrics Trimming the prompt cuts TTFT, and trimming the answer cuts total latency. Pick the fix for the metric that is failing on that route.
Only defines the two terms The definitions are the easy part. Say which workloads depend on which metric and how you would measure both in production.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the AI System Design cheatsheet.

← Back to all AI System Design questions

Pro · $10/mo

47 of 54 AI System Design answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →