An LLM gateway, or AI proxy, is one internal service that every app calls instead of calling model providers directly. It is a reverse proxy for model traffic, so cross-cutting concerns live in one place.
// app code: one endpoint, provider-shaped body, no provider keys
const res = await fetch('https://gateway.internal/v1/messages', {
method: 'POST',
headers: { 'x-tenant-id': tenantId, 'x-trace-id': traceId },
body: JSON.stringify({ model: 'claude-sonnet-4-6', messages, stream: true })
});
The gateway owns:
- provider keys and caller auth, so no key sits in app code
- per-tenant rate limits and quotas
- retries, timeouts and failover
- trace IDs, token accounting and request logs, so usage is measured once
Prompt templates, retrieval and business logic stay in the app. So do conversation state and response rendering.
Keep it thin. It sits on the hot path of every request, so it must pass streamed chunks through untouched. It is also a single point of failure, so it needs health checks and a bypass path.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓