What is streaming, and why do production chat apps almost always enable it?

beginner

Re-explain

Streaming sends the answer in small pieces as the model writes them, instead of one block at the end. You turn it on with a request flag.

{
  "model": "claude-opus-4-8",
  "max_tokens": 1024,
  "messages": [{ "role": "user", "content": "Explain streaming." }],
  "stream": true
}

Total generation time does not change. What changes is time to first token, the wait before the first word appears: hundreds of milliseconds instead of many seconds. Users read while the model writes, so the app feels fast.

A long single response can also trip a proxy or gateway timeout. Streaming avoids that, and it lets a user stop a wrong answer early.

The cost is complexity. Usage totals arrive at the end, and a check on the full text, such as JSON validation, cannot run before the user sees it.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for LLM APIs & Integration One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

How does streaming work on the wire, and what event sequence does the Anthropic API send? Server-sent events on one open HTTP response. Walk the message start, the content block deltas, the stop events, and where usage shows up.
A stream dies halfway through an answer. What can go wrong, and how should the client recover? Tell a dropped connection apart from an error event sent mid-stream. There is no resume, so choose between a full retry and keeping the partial text.
The output must pass a JSON or policy check. How do you keep streaming and still check it? Name the options: buffer and check before sending, parse partial JSON as it arrives, or stream the text but let only the final check trigger any action.
Your backend sits between the browser and the provider. How do you pass the stream through? Forward chunks as they arrive, turn off response buffering in proxies, keep the API key on the server, and cancel upstream when the user stops.
When would you leave streaming off? When code reads the output and the call is short, like classification or batch jobs. Long calls may still stream to avoid timeouts.

What you can say

Streaming means the API sends the answer in small pieces as the model writes them, not one block at the end, and you turn it on with a stream flag on the request.
The total generation time doesn't change. What changes is time to first token, so the first words show up in hundreds of milliseconds instead of many seconds.
Users start reading while the model is still writing, so the app feels fast even though the full answer takes just as long.
There are practical reasons too. A long single response can trip a proxy or gateway timeout, and streaming avoids that. It also lets a user stop a wrong answer early.
The cost is complexity. Usage totals only arrive at the end, and a check on the full text, like JSON validation, can't run before the user has already seen it.

Weak answers to avoid

Says streaming makes the model faster Total generation time is the same. Say that time to first token improves, and the app feels faster because users read while the model writes.
Only talks about user experience This misses the operational reasons. Mention that long single responses can hit proxy or gateway timeouts, and that users can stop a wrong answer early.
Ignores what streaming costs Streaming adds complexity. Usage totals arrive at the end, and a full-text check like JSON validation can't run before the user sees the output.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the LLM APIs & Integration cheatsheet.

← Back to all LLM APIs & Integration questions

Pro · $10/mo

45 of 52 LLM APIs & Integration answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →