Quantization stores model weights at lower precision, with fewer bits per number than the original 16-bit floats. A 4-bit copy of a model is roughly a quarter of the size.
That size drop is the whole point. The weights have to fit in VRAM, and consumer GPUs have very little of it.
It also makes generation faster. Decoding is limited by memory bandwidth, so reading fewer bytes per token gives more tokens per second.
The cost is accuracy. 8-bit is close to lossless. 4-bit is usually fine for chat. Below that, quality drops fast.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Does quantization speed up prompt processing the same way it speeds up generation?Decode reads every weight for each token, so it is bandwidth-bound. Prefill is compute-bound, so weight-only quantization helps it much less.
Quantization means storing a model's weights with fewer bits per number, say 8 or 4 bits instead of the original 16-bit floats.
At 4 bits the model is roughly a quarter of its original size, and for local use that size drop is the main reason to do it.
The weights have to fit in VRAM, and a consumer GPU has very little of it, so without quantization many models won't load at all.
It also makes generation faster. Decoding is limited by memory bandwidth, so reading fewer bytes per token gives you more tokens per second.
The cost is accuracy. 8-bit is close to lossless and 4-bit is usually fine for chat, but below 4 bits quality drops fast.
Weak answers to avoid
Says it just makes the model smallerSize is only half of it. Say why size matters. Weights must fit in VRAM, and decode is bandwidth-bound, so fewer bytes also means more tokens per second.
Treats quantization as freeIt trades accuracy for memory. Give the rough curve. 8-bit is near lossless, 4-bit is usually fine for chat, and below 4 bits quality drops fast.
Mixes it up with distillation or pruningQuantization keeps every weight and stores each one in fewer bits. Distillation trains a smaller model and pruning drops weights. Keep them apart.
Credits the speedup to cheaper mathDecode is limited by memory bandwidth, not arithmetic. The speedup comes from reading fewer bytes of weights for every token generated.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
