How do you estimate how much VRAM a model needs just to load it?

beginner

← All Open-Source & Local LLMs questions

Re-explain

Take the parameter count in billions and multiply by the bytes per weight, which gives gigabytes. Then add about 20% for runtime overhead. The precision sets the bytes:

FP16 or BF16 takes 2 bytes, so an 8B model's weights take about 16 GB.
FP8 or INT8 takes 1 byte, so the same weights take about 8 GB.
4-bit Q4_K_M takes about 0.6 bytes, so the weights take about 4.8 GB.

8B at FP16:  8 * 2 = 16 GB weights
plus ~20% runtime overhead  ->  ~19 GB needed

4-bit is not exactly 0.5 bytes per weight. Quantized files also store scales and zero points, and often keep some tensors at higher precision.

The same math settles whether a 70B model fits on one 80 GB card. At FP16 its weights alone are 140 GB, so it does not. At 4-bit they drop to about 42 GB, about 50 GB with overhead, so it fits.

That is the load cost only. Serving adds the KV cache, which grows with context and batch size and can outgrow the weights.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for Open-Source & Local LLMs One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

How does the KV cache change that estimate once you serve real traffic? Says it scales with layers, KV heads, head dimension, context length and batch size, and can outgrow the weights at long context.
For an MoE model, do you size VRAM on total parameters or active parameters? Says every expert must be in memory, so total parameters set load size. Active parameters set compute per token and speed.
The model does not fit on your one GPU. What are your options? Covers lower precision, a smaller model, splitting across GPUs and CPU offload, and what each one costs in quality or speed.
How would you choose between 8-bit and 4-bit for that 70B model? Weighs the memory saved against quality loss measured on your own eval set, and notes the freed memory can go to KV cache and batch size.
What actually goes into that 20% overhead? Names the CUDA context, activation and scratch buffers, and allocator fragmentation, and says the real figure varies by runtime.

What you can say

The quick estimate is parameters in billions times bytes per weight, which gives you gigabytes, plus about 20% for runtime overhead.
Precision sets the bytes. FP16 or BF16 is 2, FP8 or INT8 is 1, and a 4-bit format like Q4_K_M is about 0.6.
So an 8B model at FP16 is 16 GB of weights, about 19 GB with overhead, and the same model at 4-bit is under 5 GB of weights.
4-bit isn't exactly half a byte, because quantized files also store scales and zero points and often keep some tensors at higher precision.
The same math says a 70B model won't fit on one 80 GB card at FP16, since the weights alone are 140 GB. At 4-bit it's about 50 GB with overhead, so it fits.
That's only the cost to load it, though. Serving adds the KV cache, which grows with context and batch size and can end up bigger than the weights.

Weak answers to avoid

Gives a size without naming the precision 'An 8B model needs 16 GB' holds only at FP16. Name the precision first, since it moves the same model from about 5 GB to 16 GB of weights.
Treats weight size as the whole budget Weights alone leave no room for runtime overhead, and serving adds a KV cache that can outgrow them. Add about 20% and call out the KV cache separately.
Assumes 4-bit means exactly half a byte Quantized files also store scales and zero points and keep some tensors at higher precision. Budget about 0.6 bytes per weight for Q4_K_M.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the Open-Source & Local LLMs cheatsheet.

← Back to all Open-Source & Local LLMs questions

Pro · $10/mo

48 of 55 Open-Source & Local LLMs answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →