These labels describe different levels of access and freedom.

• Proprietary: a vendor keeps the weights and exposes the model through a service. Users cannot run or inspect the model independently.
• Open-weight: users can download trained parameters and run them on their own hardware. The release may still omit training data, training code, or detailed methods. Its license may restrict some uses or redistribution.
• Open-source AI: the release must provide the freedoms and materials needed to use, study, modify, and share the system under a qualifying license. Open-source software around a model does not automatically make the model itself open source.

Always read the exact model license and card. Check commercial use, redistribution, acceptable-use terms, included artifacts, and whether a modified model can be shared. Downloadable weights are useful, but they describe access, not the full legal or technical openness of the system.

Five reasons appear often. Each has a trade-off.

• Privacy and data control: prompts and outputs never leave your infrastructure, which matters for regulated data (health, finance, legal) or contractual restrictions. Nothing is logged by a third party.
• Cost at scale: hosted APIs are priced per token and are cheap until volume is enormous.
• Latency and locality: you can put the model next to your application, avoid network round-trips, and control tail latency instead of sharing a multi-tenant queue.
• Offline and air-gapped: on-prem, edge, or classified environments where calling the public internet is not an option.
• Control and stability: no surprise deprecations, no silent model updates changing behavior, full choice of model, quantization, and sampling.

The honest counterweight: self-hosting means you own capacity planning, GPU procurement, on-call, upgrades, and evals. A well-used API is often cheaper and better for spiky or low-volume traffic.

Several organizations publish widely used open-weight model families:

• Meta develops Llama, with general-purpose dense and mixture-of-experts models.
• Mistral AI develops Mistral and Mixtral, plus coding and smaller edge-focused variants.
• Alibaba develops Qwen, including general, coding, vision, and reasoning models.
• DeepSeek develops large mixture-of-experts and reasoning models.
• Google develops Gemma, including compact and multimodal variants.
• Microsoft develops Phi, a family focused on smaller models.

Model names, sizes, licenses, and supported context lengths change often. Check the official model card before choosing one. Compare candidates on your own task, hardware, language needs, safety tests, and license terms. Family reputation is only a shortlist; it does not replace evaluation of the exact checkpoint and quantization you plan to deploy.

Ollama packages the main steps needed to run open-weight models on a local computer. It includes a model library, an inference engine, automatic processor placement, and a local HTTP server with its own and OpenAI-compatible interfaces.

You do not manage quantization files or compile anything; you run one command.

# Install, then pull and chat with a model
ollama run llama3.1:8b
# It also exposes a server on localhost:11434
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain the attention cache in one sentence."
}'

Use Ollama for local development, prototyping, single-user desktop apps, quick model comparisons, and demos. It handles common quantized model files and runs on macOS, Linux, and Windows.

Where it fits less well is high-throughput production serving of many concurrent users. It is not optimized for continuous batching across large request volumes the way vLLM or SGLang are.

In practice: Ollama optimizes for developer experience and single-user simplicity; production multi-tenant serving is a different tool's job. Many teams prototype on Ollama, then graduate the winning model to vLLM for serving.

Quantization stores model weights, activations, or the key-value attention cache with fewer bits. The attention cache holds information from earlier tokens during generation. Values trained at 16-bit precision may be stored with 8, 4, or fewer bits using scaling factors.

Weights usually dominate the model's memory footprint. Fewer bits reduce required video memory (VRAM) and can speed generation when memory bandwidth is the limit.

Why it matters locally: a 70B model at 16-bit needs around 140 GB, which is multiple data-center GPUs. The same model at 4-bit needs roughly 40 GB and fits on a single high-end card, or even a well-equipped workstation.

The tradeoff is quality. Aggressive quantization introduces rounding error that can degrade accuracy, and small models feel it more than large ones.

GGUF is a model file format used by llama.cpp, Ollama, and the LM Studio app. One file can hold quantized weights, the tokenizer, the chat template, and other model details.

It replaced the older GGML format and is widely used for laptop and CPU inference.

GGUF comes in many quantization levels, and the naming tells you the tradeoff:

• Q2 and Q3: smallest files, with clear quality loss.
• Q4: a common balance of size and quality.
• Q5 and Q6: larger files with better quality.
• Q8: close to full precision but much larger.

# Illustrative: tags vary by model library
ollama pull <model>:<quantized-tag>

GGUF can run on a CPU, GPU, Apple Metal, or a mix. You can place some layers on the GPU and keep the rest in system memory. For high-throughput GPU servers, formats designed for GPU kernels may perform better.

Start with the weights, because they set the memory floor. Video memory (VRAM) is roughly the parameter count multiplied by bytes per parameter.

• 16-bit floating point: 2 bytes per parameter.
• 8-bit: about 1 byte per parameter.
• 4-bit: about 0.5 to 0.6 bytes per parameter (a bit over 4 bits once you include scales).

The quick estimate is about two gigabytes per billion parameters at 16-bit precision, or a little over half a gigabyte at 4-bit precision.

7B  model: FP16 ~14 GB   | 8-bit ~7 GB   | 4-bit ~4 GB
13B model: FP16 ~26 GB   | 8-bit ~13 GB  | 4-bit ~7 GB
70B model: FP16 ~140 GB  | 8-bit ~70 GB  | 4-bit ~40 GB

That covers only weights. Add the key-value attention cache, activations, runtime memory, and unused gaps. The attention cache grows with context length and concurrent requests.

A safe rule for a rough estimate is to take the weight figure and add 20 to 40 percent of headroom for short contexts. More if you serve long contexts or many concurrent requests.

Weights set the floor, while the attention cache often causes surprises with long contexts or high concurrency.