Chunking exists because an embedding is one fixed-size vector, no matter how long the input. A whole document collapses into a single point that averages all its topics, so the one relevant paragraph loses its signal.
Three forces make this concrete:
Dilution. A 40-page manual covering billing, refunds, and shipping matches a refund question only weakly. Most of its text is about something else.
Model limits. Embedding models cap input tokens. Text past the cap is truncated, so it never gets indexed at all.
Prompt budget. What you retrieve is what you paste into the prompt. Whole documents burn tokens and bury the answer among unrelated text.
So retrieve the smallest unit that still stands on its own.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Small chunks retrieve well but give the LLM too little context. How do you get both?Match on small chunks, then give the model the parent section or neighboring text. Say what that costs in prompt tokens.
We chunk because an embedding is one fixed-size vector however long the input is, so a whole document gets squeezed into a single point.
That point averages every topic in the document, so the one paragraph that answers the question loses its signal. That's dilution.
Say a 40-page manual covers billing, refunds and shipping. A refund question matches it only weakly, because most of the text is about something else.
Embedding models also cap input tokens. Anything past the cap gets truncated, so it never gets indexed at all.
Whatever you retrieve also goes into the prompt, so whole documents burn tokens and bury the answer in unrelated text.
So I retrieve the smallest unit that still stands on its own. If you cut too small, a chunk loses the context it needs to make sense.
Weak answers to avoid
Only blames the token limitTruncation is real, but even a document that fits gets one vector that averages all its topics. Lead with dilution, then add the cap.
Says smaller chunks are always betterA chunk that is too small loses the context it needs and can split an answer in two. Aim for the smallest unit that still stands on its own.
Treats it as a retrieval-only problemWhat you retrieve goes into the prompt. Whole documents burn tokens and bury the answer, so chunking helps the generation step too.
Says chunking makes search fasterChunking gives you more vectors to search, not fewer. The reasons are retrieval quality and model limits, not index speed.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
