Open-Source & Local LLMs.

An open-weight model ships its trained parameters as a download. You can run it, fine-tune it, and inspect the weights. What you usually do not get is the training data or the training code.

Open source, strictly speaking, also publishes the data, the code, and the recipe under a standard license. Proprietary models keep the weights in-house, so you only reach them over an API.

Most models people call "open source" are really open-weight. Llama weights are free to download, but the license adds use limits, so read it. Gemma moved to plain Apache 2.0 with Gemma 4.

Teams self-host when data cannot leave their network, or when steady volume makes owned hardware cheaper than per-token billing. The main reasons:

• Privacy. Prompts and documents never leave your network.
• Cost. You pay a fixed GPU bill instead of a per-token one.
• Stability. You pin the weights, so there is no surprise deprecation or silent update.
• Latency. There is no public internet hop, and a small local model answers fast.
• Control. You keep your own fine-tunes, sampling settings, and full logs.

Air-gapped sites, with no network path to the internet, have no other option.

The cost win only holds when the GPU stays busy. An idle GPU bills the same. You also inherit capacity planning, upgrades, and evaluation. For spiky or low traffic, a hosted API is usually cheaper and ships sooner. Frontier hosted models also still lead on the hardest tasks.

Around ten families from US, Chinese and French labs cover almost everything teams self-host. A family is one lab's ongoing series of models, not a single release.

From US labs:

• Llama by Meta, custom community license
• gpt-oss by OpenAI, Apache 2.0
• Gemma by Google, Apache 2.0 since Gemma 4, custom Gemma terms before it
• Phi by Microsoft, MIT
• Olmo by Ai2, Apache 2.0, with open training data too

From Chinese labs:

• Qwen by Alibaba, Apache 2.0
• DeepSeek by DeepSeek, MIT
• GLM by Z.ai, MIT. Z.ai was formerly Zhipu.
• Kimi by Moonshot AI, modified MIT

France's Mistral AI makes Mistral, Mixtral and Magistral, under Apache 2.0 or research-only terms. IBM Granite and NVIDIA Nemotron are also common. So are TII's Falcon and MiniMax.

Apache 2.0 and MIT are permissive, while custom terms like Llama's add use limits. Chinese labs now ship many of the strongest permissive weights, often as large mixture-of-experts models that run only part of the model per token.

Each family ships sizes from roughly 1B to over 500B parameters, plus instruct and reasoning variants. Pick on license and size first, then test.

Ollama is a tool that runs open-weight models on your own machine. It pulls a model with one command and serves it over a local HTTP API on port 11434.

It wraps llama.cpp and ships each model with its chat template, the exact turn markers the model was trained on, plus sane defaults. So you do not pick a quantization or format prompts by hand. It also exposes an OpenAI-compatible route, so most SDKs work after you change the base URL.

ollama run llama3.2

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "llama3.2",
  "messages": [{"role": "user", "content": "hello"}]
}'

Reach for it for local development, private prototypes, and desktop apps serving one user. It is not built for many concurrent users, which is a job for a throughput-focused server.

Quantization stores model weights at lower precision, with fewer bits per number than the original 16-bit floats. A 4-bit copy of a model is roughly a quarter of the size.

That size drop is the whole point. The weights have to fit in VRAM, and consumer GPUs have very little of it.

It also makes generation faster. Decoding is limited by memory bandwidth, so reading fewer bytes per token gives more tokens per second.

The cost is accuracy. 8-bit is close to lossless. 4-bit is usually fine for chat. Below that, quality drops fast.

GGUF is the model file format that llama.cpp reads. It replaced the older GGML format. One file holds the tensors plus the metadata: tokenizer, architecture, and the chat template.

That self-contained design is the point. You download one file and run it, with nothing else to assemble.

It fits the local, single-user lane. llama.cpp, Ollama and LM Studio all take GGUF, on CPU or a split between CPU and GPU. Server engines like vLLM read safetensors on the GPU instead.

The quantization level sits in the file name.

Llama-3.1-8B-Instruct-Q4_K_M.gguf
                      ^^^^^^^ 4-bit k-quant, medium size

GGUF is an inference format. You convert to it from Hugging Face weights, and you do not train in it.

Take the parameter count in billions and multiply by the bytes per weight, which gives gigabytes. Then add about 20% for runtime overhead. The precision sets the bytes:

• FP16 or BF16 takes 2 bytes, so an 8B model's weights take about 16 GB.
• FP8 or INT8 takes 1 byte, so the same weights take about 8 GB.
• 4-bit Q4_K_M takes about 0.6 bytes, so the weights take about 4.8 GB.

8B at FP16:  8 * 2 = 16 GB weights
plus ~20% runtime overhead  ->  ~19 GB needed

4-bit is not exactly 0.5 bytes per weight. Quantized files also store scales and zero points, and often keep some tensors at higher precision.

The same math settles whether a 70B model fits on one 80 GB card. At FP16 its weights alone are 140 GB, so it does not. At 4-bit they drop to about 42 GB, about 50 GB with overhead, so it fits.

That is the load cost only. Serving adds the KV cache, which grows with context and batch size and can outgrow the weights.