How do you pick a max_tokens value, and what happens when it's set too low?Size it to the longest answer you expect plus a margin. Check the stop reason so you catch a truncated response and don't treat it as complete.
Why does output length drive latency more than prompt length does?The model reads the whole prompt in one parallel pass, but writes output one token per step. A long prompt mostly raises time to first token.
Tokens are the unit a model bills, limits and produces, so a request's token count decides what it costs, whether it fits, and how long it takes.
On cost, providers charge per token, and output tokens usually cost several times more than input tokens.
On fit, the prompt plus the output has to fit in the context window, or the call errors or the output gets cut off.
On latency, the model writes output one token at a time, so the time to finish grows with the length of the answer.
It adds up fast at scale, because a small change to the prompt can double the bill across millions of calls.
So I trim old chat history and set a real max_tokens on every request, sized to the answer I expect, because a cap that's too tight truncates the output.
Weak answers to avoid
Only says tokens cost moneyCost is one effect out of three. Also cover fitting in the context window and latency that grows with output length, or the answer sounds like a note about billing.
Treats input and output tokens as equalOutput tokens usually cost several times more and set how long the call takes. If you miss that, you miss the main lever, which is capping output with max_tokens.
Equates tokens with wordsA token is not a word, so budgets built on word counts drift. Say you would count tokens with the model's own tokenizer before relying on the number.
Explains the why but gives no actionIf you stop at why it matters, the answer sounds academic. End with what you do about it: trim old history and set max_tokens on every call.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
