LLM Fundamentals · question

Question 4 of 58

What is a context window?

beginner

Re-explain

The context window is the most tokens a model can handle in one request. The window covers the system prompt, the conversation history, and the tokens the model generates.

A 200K window with a 190K prompt leaves about 10K for the reply, so max_tokens must fit that leftover space.

WINDOW = 200_000
prompt_tokens = count_tokens(messages)   # measure it, do not guess
max_tokens = min(4096, WINDOW - prompt_tokens - 500)   # keep a safety margin

Models are stateless and keep nothing between calls, so each request resends the whole history.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for LLM Fundamentals One question at a time. Answer out loud, get graded, see what you missed.

See it - interactive diagram

Everything shares one 200K context window

Context window 200K tokens

First turn7.1K of 200K tokens

System prompt 2K tokens
History 1K tokens
Reply (max_tokens) 4.1K tokens

The system prompt, your first message and the reply budget all fit in one 200K window, with lots of room left.

Mid chat66K of 200K tokens

System prompt 2K tokens
History 60K tokens
Reply (max_tokens) 4.1K tokens

The model remembers nothing between calls. Each request resends the whole history, including the model's earlier replies, so the history keeps growing.

Long chat194K of 200K tokens

System prompt 2K tokens
History 188K tokens
Reply (max_tokens) 4.1K tokens

The prompt is now 190K tokens, so about 10K is left. A max_tokens of 4,096 plus a 500 token margin still fits.

Near the limit200K of 200K tokens

System prompt 2K tokens
History 195K tokens
Reply (max_tokens) 2.5K tokens

With a 197K prompt, the code sets max_tokens to 200K minus 197K minus the 500 margin, which is only 2,500.

Near the limit, no check201K tokens, over the limit

System prompt 2K tokens
History 195K tokens
Reply (max_tokens) 4.1K tokens

With a fixed max_tokens of 4,096, the request needs about 201K tokens, roughly 1,100 past the window. The call fails or the reply gets cut.

The system prompt, the history and the reply share one 200K window. As the history you resend each turn grows, the room left for max_tokens shrinks.

Other questions have diagrams like this too. Sign in to ask for one where it's missing - it's free.

Create a free account

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

The chat keeps growing. What do you do when history plus the reply no longer fits in the window? Trim or summarize old turns, keep the system prompt and recent turns, move reference docs to retrieval, and check the budget before sending.
Does answer quality hold up as the prompt gets close to the window limit? Name lost-in-the-middle and distraction from irrelevant text. A prompt that fits in the window is not always one the model uses well.
How do you count prompt tokens accurately before sending, and why not just estimate from characters? Use the model's own tokenizer or the provider's count endpoint. Ratios drift with language, code and formatting, and tokenizers differ by model.
Every turn resends the whole history. How do you keep cost and latency under control? You pay for the whole prompt on every call. Cache the stable prefix, summarize or drop old turns, and send only what this turn needs.
A model advertises a 1M-token window. How do you find out how much of it works for your task? Test on your own data at several lengths, with facts placed at different depths. The point where accuracy drops is your real limit.

What you can say

The context window is the most tokens a model can handle in one request, and it's one budget for everything going in and coming out.
That budget covers the system prompt, the whole conversation history, and the tokens the model generates in its reply.
So if the window is 200K and my prompt is already 190K, I have about 10K left, and max_tokens has to fit inside that.
In code I count the prompt tokens with a real tokenizer instead of guessing, then cap max_tokens at what's left, minus a safety margin.
Models are stateless, so nothing carries over between calls, and every request resends the whole history.
That's the catch in long chats. Each turn costs more and leaves less room for the reply, and once the history outgrows the window I have to trim or summarize it.

Weak answers to avoid

Counts only the input tokens The reply comes out of the same budget. Say that prompt plus max_tokens must fit, or the call fails or the reply gets cut off.
Thinks the model remembers earlier turns Models are stateless. The app resends the whole history on every call, which is why long chats cost more and eventually overflow.
Treats the window as words or characters The limit is in tokens, and the ratio to words varies by language and content. Count with the model's tokenizer instead of guessing.
Assumes a bigger window fixes everything A prompt that fits isn't always one the model uses well, and every token adds cost and latency. Mention trimming old turns and retrieval.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the LLM Fundamentals cheatsheet.

From the blog: Jev explained: the AI model that returns decisions, not text

← Back to all LLM Fundamentals questions

Pro · $10/mo

51 of 58 LLM Fundamentals answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →