Prompt caching stores the model's internal state for a prompt prefix, its KV cache, so a later request with the same prefix skips recomputing it. The match is prefix-based from the first token, so put stable parts first and the changing user turn last. One edited character early invalidates everything after it.
// Anthropic-style: mark where the stable prefix ends
const req = {
system: [
{ type: 'text', text: instructions + toolDocs,
cache_control: { type: 'ephemeral' } } // cache up to here
],
messages: [{ role: 'user', content: userTurn }] // changes every call
};
Cost falls because cached tokens are read back at a steep discount. Writing the cache can cost more than plain input, so it pays off only on repeats. Providers also set a minimum prefix size, so short prompts never cache.
Latency falls because caching shortens prefill, the pass over the input before the first output token. Time-to-first-token drops, but generation speed is unchanged. Entries usually expire after minutes of inactivity, so repeated, bursty traffic gains most.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓