Streaming sends tokens as the model writes them, so the user reads the first words in under a second, not a spinner. Total time barely changes. The wait just feels shorter.
It changes the architecture because the reply stops being one JSON object. You hold a connection open, usually with Server-Sent Events (SSE), a one-way stream over HTTP. Every hop must forward chunks without buffering, and proxies, gzip and gateways buffer by default.
The HTTP status goes out before the answer exists. So a mid-stream failure needs an in-band error event, not a 500. Token counts also arrive last, so keep a copy of the stream on the server for logging, caching and output checks.
let full = '';
for await (const chunk of stream) {
full += chunk.delta; // server-side copy
res.write(`data: ${JSON.stringify(chunk)}\n\n`);
}
res.write(`event: done\ndata: ${JSON.stringify(usage)}\n\n`);
await logAndCache(full); // only possible after the last chunk
If the user stops a bad answer, abort the upstream call. Otherwise you pay for tokens nobody ever sees.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓