What is GGUF, and where does it fit?

beginner

← All Open-Source & Local LLMs questions

Re-explain

GGUF is the model file format that llama.cpp reads. It replaced the older GGML format. One file holds the tensors plus the metadata: tokenizer, architecture, and the chat template.

That self-contained design is the point. You download one file and run it, with nothing else to assemble.

It fits the local, single-user lane. llama.cpp, Ollama and LM Studio all take GGUF, on CPU or a split between CPU and GPU. Server engines like vLLM read safetensors on the GPU instead.

The quantization level sits in the file name.

Llama-3.1-8B-Instruct-Q4_K_M.gguf
                      ^^^^^^^ 4-bit k-quant, medium size

GGUF is an inference format. You convert to it from Hugging Face weights, and you do not train in it.

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for Open-Source & Local LLMs One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

How do you get a GGUF file for a model that only ships as safetensors on Hugging Face? Run llama.cpp's converter on the Hugging Face weights, then quantize the result. The converter writes the tokenizer and template into the file.
How would you pick between Q4_K_M, Q5_K_M and Q8_0 for a given machine? Leave room for the KV cache, then take the most bits that still fit. Test quality on your own task before you settle.
If the file is bigger than your VRAM, how does llama.cpp split a GGUF between GPU and CPU? It offloads whole layers to the GPU until VRAM is full and runs the rest on CPU. The CPU layers set the pace, so tokens per second drop.
The GGUF carries a chat template. What breaks if that template is missing or wrong? The model gets a prompt layout it was not trained on. It rambles, mixes up roles or never stops. Check the template in the metadata or override it.
Why would you serve a multi-user API with vLLM on safetensors rather than a GGUF runtime? vLLM runs many concurrent requests on GPU with continuous batching and PagedAttention. GGUF runtimes target one user on modest hardware.

What you can say

GGUF is the model file format that llama.cpp reads, and it replaced the older GGML format.
One file holds the weights plus the metadata the runtime needs, meaning the tokenizer, the architecture and the chat template.
That self-contained design is the whole point, because you download one file and run it with nothing else to assemble.
It belongs to the local, single-user lane, where llama.cpp, Ollama and LM Studio all load it on CPU or on a CPU and GPU split.
The quantization level is in the file name, so Q4_K_M means a 4-bit k-quant at the medium size.
It is an inference format only, so you convert to it from Hugging Face weights and never train in it, and a GPU server like vLLM reads safetensors instead.

Weak answers to avoid

Calls GGUF a quantization method GGUF is a file format, and it can hold several quant levels or full precision. Call it the container and point to Q4_K_M in the name as the quant level.
Says you can train or fine-tune in GGUF It is an inference format. You train on Hugging Face weights and convert to GGUF at the end, when you want to run the model locally.
Pitches GGUF for a high-traffic GPU server That lane belongs to server engines like vLLM reading safetensors on GPU. GGUF fits local, single-user runs on CPU or a CPU and GPU split.
Leaves out the metadata in the file Skipping the tokenizer and chat template misses why one file is enough. Say the file carries them, so there is nothing else to fetch or wire up.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the Open-Source & Local LLMs cheatsheet.

← Back to all Open-Source & Local LLMs questions

Pro · $10/mo

48 of 55 Open-Source & Local LLMs answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →