home
›
cheatsheets
›
ai_engineering
›
rag
Free to read Every card and diagram, and Quiz me
2
Free account Star cards. Your Quiz me marks are kept, so next time you drill just what you missed
3
Pro All 56 answers on this sheet, My sheet across every topic, and PDFs
A free account keeps your stars and Quiz me marks. Pro adds all 56 answers on this sheet, My sheet across every topic, and PDFs.
/
Read
Quiz me
Comfortable
Compact
0
Click any code to copy it
Done 0 /0
To review 0
On this sheet
01 How RAG works02 Chunk documents03 Embeddings & vector search04 Hybrid search & reranking05 Rewrite & route queries06 Prompt, context & citations07 Evaluate & debug08 Agentic, graph & SQL RAG09 Freshness, access & security10 Speed, caching & high stakes
The model RAG fetches relevant text at query time and adds it to the prompt. The model answers from that evidence, not only from what it learned in training. Two paths meet at the index . The indexing path chunks and embeds documents when they change. The query path embeds each question with the same model and finds the nearest chunks. Answer quality depends on retrieval . If the right chunk never reaches the prompt, the model is back to guessing. Retrieve wide, then narrow. Fetch 20 to 100 candidates cheaply, rerank them, and send the model the best 3 to 8.Measure retrieval and generation apart. RAG cuts hallucinations but doesn't end them, because the model can still misread good evidence. Indexing runs when documents change. Queries use the same embedding model, and a chunk the search misses never reaches the LLM.
How RAG works how an answer gets made, and what RAG fixes The pipeline, step by step List the indexing steps Parse, chunk, embed, then store the vectors with metadata.
List the query steps Rewrite, embed, retrieve, rerank, build the prompt, then generate.
Trap Embed queries and chunks with the same model, or their vectors won't compare.
What RAG fixes Name the problem it solves The model's knowledge ends at a cutoff and misses your private data.
Say why hallucinations remain Retrieval can miss, and the model can misread good passages.
Tip Edit a document and the next answer changes, with no retraining.
Trap If retrieval misses, the model usually answers from memory anyway.
RAG, fine-tuning or long context Pick RAG For large, changing or private data, citations and per-user access.
Pick fine-tuning To change behavior, like format, style or a narrow task.
Pick long context When a small, stable document set fits in the prompt.
Find when stuffing the whole corpus is cheaper Below breakeven, which is fixed cost over the saving per query.
Tip Prompt caching cuts repeat prefill to about a tenth. That raises the breakeven for stuffing more than tenfold.
Trap Fine-tuning is a weak way to add facts. The model blends them and can't cite a source.
Chunk documents split text into pieces that search well Chunk size and overlap Say why not embed whole documents One vector averages all its topics, so the answer gets lost.
Pick a chunk size The smallest unit that answers a typical question.
Set the overlap 10 to 20 percent, so an idea cut at an edge stays whole in a chunk.
Tip Small chunks match sharply but lose context. Large ones stand alone but blur the match.
Trap With overlap, search can return near-copies. Dedupe neighbors before you build the prompt.
Pick a chunking strategy Chunk uniform text, or set a baseline Fixed-size. Cut every N tokens, plus overlap.
Chunk messy prose Recursive. Split by paragraph, then sentence, then word.
Chunk Markdown, HTML or code Structural. Follow headings, tags, syntax trees or table rows.
Chunk text that changes topic with no markup Semantic. Cut where the embedding distance between sentences jumps.
Tip Start structural when the corpus has real markup. Headings double as metadata for filters and citations.
Trap Semantic chunking costs an extra embedding pass and often loses to recursive. Measure it first.
Keep context around small chunks Search small chunks, but send the model more Parent-document retrieval. Match small chunks, send their parents.
Fix chunks that lost their subject or date Contextual retrieval. Index each with an LLM note on where it sits.
Index tables and images Embed a text stand-in, but send the model the original.
Tip Contextual retrieval changes what you index. Parent-document retrieval changes what you send.
Trap One big chunk isn't the same. Its single vector blurs the match a small chunk keeps sharp.
Embeddings & vector search turn chunks into vectors you can search fast Embeddings and similarity Say what an embedding is A list of numbers from a model. Similar meanings sit close together.
Pick a similarity metric The one the model was trained with, usually cosine.
Make dot product equal cosine v = v / np.linalg.norm(v) Normalize once, at write time.
Choose an embedding model Test two or three on labeled pairs from your own data.
Tip Leaderboards like MTEB rank general text. Your domain's jargon can reorder them.
Trap Many models expect a query or passage prefix. Leave it off and you quietly lose recall.
Vector databases and ANN indexes Say why not compare against every vector Cost grows with the corpus. ANN trades a little recall for speed.
Pick an index for most corpora HNSW. Best recall at low latency, but it uses more memory.
Pick an index when memory is tight IVF with product quantization. It scans only the nearest cells.
Tip Quantization works on top of either index. Rescore the top candidates with full vectors.
Trap IVF's cells come from k-means, so retrain them as the data drifts.
Filter by metadata Store what you'll filter on Tenant, language, date and permissions, beside each vector.
Keep k results under a filter Pre-filter. Only matching vectors can enter the results.
Say what post-filtering risks It filters after the search, so you can get fewer than k, or none.
Tip With very few matches, a filter inside the HNSW walk may never reach them. Good engines then scan the filtered set.
Trap A post-filter on a tenant with 0.1 percent of the corpus will still empty your top k.
Hybrid search & reranking find the right chunks and put the best first Keyword, dense or hybrid search query BM25 dense RRF rerank prompt hybrid BM25 and dense search run at once. RRF merges the lists, then a reranker keeps the best.Find a pasted SKU or error code BM25. It matches exact terms and weights rare ones up.
Find a paraphrase with no shared words Dense search. It matches meaning, not wording.
Say why hybrid beats either alone They fail on different queries, so both together lift recall.
Pick SPLADE or ColBERT over hybrid Mainly out of domain, where single-vector models drift.
Tip ColBERT keeps one vector per token, so its index is about ten times larger.
Merge ranked lists with RRF score(d) = sum over lists of 1 / (k + rank) k = 60 A: keyword rank 2, dense rank 3 1/62 + 1/63 = 0.0320 B: keyword rank 1, not in dense 1/61 = 0.0164
Say what RRF uses Each document's rank in each list, never its raw score.
Say why not add raw scores BM25 is unbounded and cosine runs from -1 to 1. They don't compare.
Say what k does It softens the top ranks, so no single list dominates.
Trap RRF drops the margin, so a clear winner and a near-tie look alike. Add a reranker when that matters.
Choose k, and skip near-copies Pick k, the number of chunks kept Test several values. Pick where the hit rate stops rising.
Set k per stage Fetch 20 to 50 candidates, then send the model 3 to 8.
Keep near-copies out of the prompt MMR. Each pick loses points for repeating one already picked.
Tip Too small a k drops the right chunk. Too large adds noise, tokens and latency.
Trap Plain top-k often returns near-copies of one passage and pushes out a second source.
Rerank the shortlist # bi-encoder: two passes, compared later score = cosine(embed(query), embed(doc)) # cross-encoder: one pass over the pair score = model(query, doc)
Say why a cross-encoder is more accurate It reads query and chunk together, so every token sees every other.
Say why it only reranks a shortlist Nothing caches, so every candidate costs a full model pass.
Pick an LLM reranker instead When relevance needs instructions, or you have no labels.
Tip A common cascade has a cross-encoder cut 100 to 10, then an LLM judge those 10.
Trap A reranker can't bring back a chunk that retrieval missed.
Rewrite & route queries turn what the user typed into a good search Rewrite follow-up questions User: Do you refund annual plans? User: and after 30 days? Search: annual plan refund after 30 days
Say why not embed the latest message A follow-up often names no topic, so its vector lands nowhere useful.
Choose what the generator sees The original message. Only the search uses the rewrite.
Trap A bad rewrite fails quietly, since retrieval still returns confident chunks. Log both forms.
Expand, split or use HyDE Fix a short or vague query Expansion. Search several phrasings and fuse the lists.
Handle a question about two topics Decomposition. Search each part, since one vector matches neither.
Search with a drafted answer HyDE. Embed a drafted answer, which looks more like a passage.
Make HyDE safer Search with the question and the draft, then fuse with RRF.
Tip Each one adds retrieval or LLM calls, so latency rises. Split a query only when it really has two parts.
Trap A HyDE draft can invent names or numbers that pull the search toward the wrong documents.
Route to the right source Pick a router Rules for a few sources, embeddings for many, an LLM for overlaps.
Know when to skip retrieval For greetings, answered follow-ups, and rewrite or translate tasks.
Tip A missed search gives a wrong answer. An extra one costs only tokens and latency.
Prompt, context & citations fit the best chunks in and make answers checkable Fill the prompt template System: Answer only from the context. If it has no answer, say you don't know. Cite sources as [1], [2]. [1] (handbook.pdf, p.12) <chunk text> [2] (faq.md) <chunk text> Question: How many vacation days do I get?
Label every chunk Give each one an id and a source to cite.
Place the chat history In its own slot, never inside the context block.
Trap Retrieved text is untrusted data, not instructions. Mark the context block clearly.
Ground answers and check citations Say what a grounded answer is Each claim traces to a chunk. If none answers, the model says so.
Catch a made-up citation id Check each cited id against the ids you sent.
Catch a real id that doesn't back the claim Require a verbatim quote, match it, then check entailment.
Tip Track the share of citations you reject as a live metric.
Trap Citation ids are local to one prompt, not global document ids. Your app maps each back to its source.
Order and trim the context 2 3 4 5 6 7 8 9 10 11 12 1 start middle end Numbers are rerank positions, in prompt order. Rank 1 opens the context, rank 2 closes it, and the weakest chunks sit in the middle.
Say what lost in the middle means Models use a long prompt's start and end best, and its middle worst.
Fix lost in the middle Send fewer chunks, and put the best ones at the edges.
Assemble the final context Dedupe, fill a token budget by rerank score, then order.
Tip Count tokens with the model's tokenizer, and cap chunks per document.
Trap A bigger context window doesn't flatten the curve. Test by moving a known answer through every position.
Handle sources that disagree Handle an outdated source Prefer the current one. Mention the old version only if it helps.
Handle conflicting opinions or findings Summarize each view neutrally, each with its citation.
Handle a source that's likely false Ignore it and answer from the reliable sources.
Handle answers that fit together Merge them into one answer. Don't frame it as a debate.
Tip Store an effective date, version and authority on every chunk, so ranking can prefer current, trusted text.
Evaluate & debug measure retrieval and answers, then find what broke Retrieval metrics Tune a retriever that feeds a reranker Recall@k, the share of relevant chunks in the top k.
Score when only the top result gets read MRR, the mean of 1 / rank of the first relevant hit.
Score graded relevance where order matters nDCG, which discounts each hit by its position.
Tip MRR looks only at the first hit, so it misleads on questions that need several passages.
Trap Recall@k can score 1.0 while the answer sits at rank 20.
Grade the answers Check the answer sticks to its context Faithfulness. Supported claims divided by all claims.
Check it answers the question asked Answer relevancy. Rebuild the question from the answer and compare.
Grow evaluation from prototype to production Retrieval hit rate, answer checks per release, then live signals.
Tip Answer correctness needs a reference answer. Faithfulness and relevancy don't.
Trap LLM judge scores drift when the judge model changes. Pin its version and check it against human labels.
Build an eval set from scratch Seed questions from the corpus An LLM writes questions a chunk answers. That chunk is the label.
Add the cases that break systems Real user queries, multi-hop questions and unanswerable ones.
Tip 100 human-checked items beat 10,000 unchecked ones. Freeze and version the set.
Trap Questions made from a chunk copy its wording, so retrieval looks better than it will in production.
Find where a chunk got lost Check the document is indexed Fetch it by id. Missing means ingestion failed.
Check the chunk can be found at all Search for its own text. A miss means a stale vector or wrong namespace.
Check the filters Drop tenant, ACL and metadata filters. They hide valid hits.
Check its rank At rank 30 it's a ranking problem, often a wording mismatch.
Tip If the chunk did reach the prompt, retrieval worked and generation failed.
Trap Log the query that was actually embedded, not the one the user typed.
Where RAG fails Name the failures before generation Missing content, ranked too low, or trimmed out of the context.
Name the failures in the answer Not extracted, wrong format, wrong level of detail, incomplete.
Debug high recall with wrong answers Answer from the gold chunk alone. If that fails, fix generation.
Tip You can only validate a RAG system in operation, so log every stage.
Trap Recall labeled per document counts a hit even when the model saw only half the answer.
Agentic, graph & SQL RAG when one search pass isn't enough Agentic RAG and multi-hop questions Say what makes RAG agentic The model decides at run time to search, search again or stop.
Say what agentic RAG buys and costs It recovers from a weak first search, but cost and paths vary.
Answer a question that needs two lookups Loop. Plan each next query from what the last search found.
Tip The bridge entity that links the hops isn't in the question, so one search can't find it.
Trap Cap it at two or three hops. One bad hop poisons every later one.
GraphRAG Say what GraphRAG builds An LLM extracts entities and relations, then summarizes clusters.
Say when it's worth it When entities are stable and questions span many documents.
Trap Every update means re-extracting and re-summarizing. For plain lookups, hybrid search plus a reranker is cheaper.
Self-RAG and Corrective RAG search grade answer weak rewrite grade loop Good chunks go to the answer. Weak ones trigger a rewrite and one more search.Say what Self-RAG does It trains the model to emit reflection tokens that judge retrieval.
Say what Corrective RAG does A small grader rates the retrieved set. Bad grades trigger web search.
Trap Production copied the control flow, not the training. Cap the loops, or latency and cost double.
RAG over SQL tables and APIs Say what you retrieve The schema, sample values and verified queries, not the rows.
Run generated SQL safely A read-only role, a timeout and a row cap. Show the SQL too.
Tip Index the distinct values of low-cardinality columns, and resolve names before writing SQL.
Trap Entity linking fails most often. Users type Acme Corp while the column holds ACME CORPORATION.
Freshness, access & security keep the index current and each tenant's data apart Keep the index fresh Drive updates from change events CDC and webhooks, plus a scheduled reconcile for missed events.
Re-embed only what changed Key chunks by doc id and content hash, and re-embed only new hashes.
Hide deleted documents at once Filter them at query time too, since index deletes lag.
Swap embedding models without downtime Dual-write to a second index, backfill, test on a golden set, canary.
Tip Store raw chunk text next to each vector. A model migration has to re-embed it.
Trap After an edit, delete the document's chunks below the new version, or a shorter document leaves stale ones behind.
Tenants and access control Enforce who can see a chunk filter={"tenant_id": session.tenant_id, "acl": {"$in": principals}} Pre-filter inside the search, never in the prompt.
Get the caller's user and group ids Resolve them on the server from the session, never from the request.
Pick an index layout Shared for scale, namespaces to contain leaks, separate for isolation.
Trap The index lags behind permission changes. Re-check access in the source system before showing a citation.
Security threats Stop indirect prompt injection Treat chunks as data, and gate tool calls.
Stop corpus poisoning Vet sources, review before indexing, and keep provenance.
Stop an over-permissioned retriever Search as the person asking, not as the app.
Protect the vector store Vectors partly reveal their text. Guard them like the documents.
Tip Anyone who can write to the corpus can write to your prompt.
Trap Delimiting chunks helps, but it isn't a control. The real controls sit outside the model.
Speed, caching & high stakes cut latency, cache safely, and handle high-stakes answers Cut latency Say where the time goes Mostly the LLM. Search takes tens of ms, decode takes seconds.
Pick the latency target Time to first token at p95. Streaming hides decode.
Remove serial LLM calls Merge the rewrite and router calls, or search while they run.
Search 100M+ vectors under a latency target Shard, quantize, rescore, and partition on a filter every query has.
Tip Prefill grows with context, so cutting 12 chunks to 5 is faster and often more accurate.
Trap With scatter-gather, p99 is set by the slowest shard, not the average one.
Cache without leaking or going stale List the cache layers Embeddings, search results, rerank scores, prompt prefix, answers.
Build a cache key Include the model version, index version, tenant and access scope.
Use a semantic cache safely Keep the threshold high. Skip numeric and time-sensitive queries.
Tip Provider prompt caching only hits when the retrieved chunks come after the fixed prefix.
Trap Leave the tenant out of the key and one user's answer is served to another.
Design for legal or medical stakes Say what the goal becomes Minimize wrong answers. Refusing counts as success.
Filter on authority and date Make jurisdiction and effective date hard pre-filters.
Check every sentence Match each sentence to its cited span. Drop what doesn't match.
Tip Report the risk-coverage curve. At what answer rate does error stay under your limit?
Trap The best text match can still be legally wrong, like a superseded statute.
Nothing matches. Try a shorter word, or clear the search.
Practice Guided practice One question at a time, with the ones you miss coming back. Practice 56 interview questions Full answers, from beginner to expert.
Found this useful? Pass it on.
Share on LinkedIn
Share on X
Copy link
This sheet stays free. With Pro , you can:
Every answer on this sheet. Read the full answer to all 56 questions it links, not just the 6 free ones.My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.
