Quantization stores model weights, activations, or the key-value attention cache with fewer bits. The attention cache holds information from earlier tokens during generation. Values trained at 16-bit precision may be stored with 8, 4, or fewer bits using scaling factors.
Weights usually dominate the model's memory footprint. Fewer bits reduce required video memory (VRAM) and can speed generation when memory bandwidth is the limit.
Why it matters locally: a 70B model at 16-bit needs around 140 GB, which is multiple data-center GPUs. The same model at 4-bit needs roughly 40 GB and fits on a single high-end card, or even a well-equipped workstation.
The tradeoff is quality. Aggressive quantization introduces rounding error that can degrade accuracy, and small models feel it more than large ones.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓