Start with the weights, because they set the memory floor. Video memory (VRAM) is roughly the parameter count multiplied by bytes per parameter.
- 16-bit floating point: 2 bytes per parameter.
- 8-bit: about 1 byte per parameter.
- 4-bit: about 0.5 to 0.6 bytes per parameter (a bit over 4 bits once you include scales).
The quick estimate is about two gigabytes per billion parameters at 16-bit precision, or a little over half a gigabyte at 4-bit precision.
7B model: FP16 ~14 GB | 8-bit ~7 GB | 4-bit ~4 GB
13B model: FP16 ~26 GB | 8-bit ~13 GB | 4-bit ~7 GB
70B model: FP16 ~140 GB | 8-bit ~70 GB | 4-bit ~40 GB
That covers only weights. Add the key-value attention cache, activations, runtime memory, and unused gaps. The attention cache grows with context length and concurrent requests.
A safe rule for a rough estimate is to take the weight figure and add 20 to 40 percent of headroom for short contexts. More if you serve long contexts or many concurrent requests.
Weights set the floor, while the attention cache often causes surprises with long contexts or high concurrency.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓