Teams self-host when data cannot leave their network, or when steady volume makes owned hardware cheaper than per-token billing. The main reasons:
Privacy. Prompts and documents never leave your network.
Cost. You pay a fixed GPU bill instead of a per-token one.
Stability. You pin the weights, so there is no surprise deprecation or silent update.
Latency. There is no public internet hop, and a small local model answers fast.
Control. You keep your own fine-tunes, sampling settings, and full logs.
Air-gapped sites, with no network path to the internet, have no other option.
The cost win only holds when the GPU stays busy. An idle GPU bills the same. You also inherit capacity planning, upgrades, and evaluation. For spiky or low traffic, a hosted API is usually cheaper and ships sooner. Frontier hosted models also still lead on the hardest tasks.
Rewriting in plainer words…
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Would you ever run local and hosted models side by side? How would you route between them?Route by data sensitivity and task difficulty. Private data stays local, and the hardest tasks go to a frontier API.
Teams self-host when their data can't leave the network, or when steady volume makes owned GPUs cheaper than paying per token.
Privacy usually comes first, because prompts and documents never leave your network, and an air-gapped site has no other option.
Cost changes from a per-token bill to a fixed GPU bill, and because you pin the weights, you get no surprise deprecation or silent update.
You also skip the public internet hop, so a small local model answers fast, and you keep your own fine-tunes, sampling settings and full logs.
The catch is that an idle GPU bills the same, and you now own capacity planning, upgrades and evaluation.
So for spiky or low traffic I'd start with a hosted API, since it's usually cheaper and ships sooner, and frontier models still lead on the hardest tasks.
Weak answers to avoid
Says self-hosting is always cheaperAn idle GPU bills the same. Say cost only wins at steady, high utilization, and that spiky or low traffic is usually cheaper on a hosted API.
Ignores the ops burdenSelf-hosting hands you capacity planning, upgrades and evaluation. Leaving that out sounds like you've never run one. Name it as the price of control.
Claims local models match frontier qualityFrontier hosted models still lead on the hardest tasks. Say you'd test a local model on your own workload before switching.
Gives privacy as the only reasonPrivacy is often first, but steady-volume cost, pinned weights, latency and control are real reasons too. Name them, then say which one drives your case.
Interview lens
Likely follow-ups, what you can say, and the weak answers to avoid.
