Streaming sends tokens to the client as they are generated rather than waiting for the full response. It matters because it collapses perceived latency. With a 300-token answer, the user sees words within a second instead of staring at a spinner for five.
Architecturally, streaming forces choices through the whole stack:
Transport: Server-Sent Events (SSE) is the common choice for one-way token streams; WebSockets when you need bidirectional (voice, interrupts).
Connection handling: each generation keeps a connection open, which affects connection limits, memory use, and autoscaling.
Error handling gets harder: a failure mid-stream has already shown the user partial output.
Post-processing tension: output guardrails and JSON validation want the whole response, but streaming shows tokens before you can validate them.
Streaming improves perceived speed, but it does not reduce total cost or generation time. It also makes output validation harder because users see partial text.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
