What is an LLM gateway (or AI proxy), and what belongs in it?

An LLM gateway is a single service that all model traffic flows through, the way an API gateway fronts microservices.

What belongs in it:

• Authentication and provider key management, so application code never holds raw provider keys.
• Per-tenant and per-user rate limiting, quotas, and spend caps.
• Routing: choosing a model, provider, or region per request, and failing over when one is down.
• Caching: exact-match and semantic caches checked before the model is called.
• Observability: structured logging of prompts, responses, token counts, latency, and cost, with a request ID that ties everything together.
• Guardrail hooks: calling input/output safety checks.
• Retries and timeouts with sensible backoff.

The benefit is leverage. When a new cheaper model ships, or a provider has an outage, or finance needs a per-team cost report, you change or read one component.

The tradeoff: a gateway is now a critical path dependency, so it must be highly available and low-overhead. Otherwise it becomes the bottleneck it was meant to prevent.