What is a vector database, and why not just compare the query against every stored vector?

beginner

← All RAG (Retrieval-Augmented Generation) questions

Re-explain

A vector database stores each chunk's embedding next to its text and metadata, and returns the chunks nearest to a query vector. Its core trick is an approximate nearest neighbor (ANN) index, which finds close matches without scanning every vector. In RAG, the chunks it returns become the LLM's context.

Comparing the query against every stored vector is a brute-force or flat scan. It is exact, and it is fine for a few thousand vectors. But the cost grows linearly with the corpus. Five million vectors at 1536 dimensions is about 7.7 billion multiply-adds per query.

An ANN index keeps the cost per query nearly flat as the corpus grows. It usually finds 95 percent or more of the true nearest neighbors. The database also handles filters, updates and persistence, which an in-memory array leaves to you. So you trade a little recall for a large speed win.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for RAG (Retrieval-Augmented Generation) One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

How does an ANN index like HNSW or IVF avoid comparing the query against every stored vector? HNSW walks a layered graph of neighbors, IVF probes only the closest clusters. A strong answer also names the knob that trades recall for speed.
The index should find about 95 percent of the true neighbors. How would you check that on your own data? Run a sample of real queries through a flat scan and through the index, compare the top-k sets, then tune search settings such as ef_search or nprobe.
You need results from one customer's documents only. How does that filter combine with the ANN search? Contrast filtering before the search with filtering after it. Post-filtering can return fewer than k results when the filter removes most matches.
When would you skip a dedicated vector database entirely? Small corpus, a need for exact results, or data already in Postgres. A flat scan or an extension like pgvector saves you from running another system.

What you can say

A vector database stores each chunk's embedding next to its text and metadata, and for a query vector it returns the nearest chunks.
In RAG, those chunks become the context the LLM answers from.
Comparing the query against every vector is a flat scan. It's exact, and for a few thousand vectors it works fine.
The problem is that cost grows linearly. Five million vectors at 1536 dimensions is about 7.7 billion multiply-adds for every query.
So the database builds an approximate nearest neighbor index, which keeps per-query cost nearly flat as the corpus grows. It also handles filters, updates and persistence.
The price is recall. You usually get 95 percent or more of the true neighbors, so you give up a little accuracy for a big speed win.

Weak answers to avoid

Calls it a database that just stores vectors Storing vectors is the easy part. Say what makes it useful, which is fast nearest-neighbor search through an ANN index, plus filters, updates and persistence.
Dismisses brute force outright A flat scan is exact and fine for a few thousand vectors. Say it breaks down because cost grows linearly with the corpus, not that it never works.
Says brute force is slow without saying why Show the math. Each query costs vectors times dimensions multiply-adds, so 5 million vectors at 1536 dimensions is about 7.7 billion.
Treats ANN results as exact An ANN index can miss true neighbors. Say it usually finds 95 percent or more of them, and that you accept that small recall loss for speed.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the RAG (Retrieval-Augmented Generation) cheatsheet.

← Back to all RAG (Retrieval-Augmented Generation) questions

Pro · $10/mo

50 of 56 RAG (Retrieval-Augmented Generation) answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →