What are the core architecture layers of a production LLM application?

beginner

Re-explain

A production LLM app is a normal web app plus two layers, orchestration and model access. The model is one dependency, not the system.

The client renders streamed output and holds the UI session.
The application API owns auth and business rules, and persists conversation history.
Orchestration builds each request from retrieval, templates and tool calls, and runs multi-step flows.
The data layer holds the vector index, documents and application database.
Model access talks to providers and holds their keys. It runs retries and timeouts, and tracks spend.
Providers are hosted APIs or self-hosted inference.

Safety checks, evals and telemetry are not a layer. They wrap every call.

These are logical layers, so start in one process. From day one, nothing outside model access imports a provider SDK.

// the only module that imports a provider SDK
export async function complete({ model, messages }) {
  return client.messages.create({ model, messages, max_tokens: 1024 });
}

Rewriting in plainer words…

This answer doesn't lend itself to a diagram - it reads best . No credits were charged.

Why there's no diagram: “”

The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓

The diagram below the answer is the concept . Jump to it ↓

Tailored explanation · switch back to · ·

What should the new diagram focus on?

Guided practice for AI System Design One question at a time. Answer out loud, get graded, see what you missed.

How well did you know this?

AI:

Saved in this browser - sign in to keep your review list.

How should your speech become text?

Listening… your words appear above as you speak - tap Stop when you're done.

Recording · cr - tap Stop & transcribe when you're done.

Transcribing with AI…

Voice:

Keep going - a few more words and AI can grade it.

Interview lens How interviewers actually use this question

Likely follow-ups

When would you split these logical layers into separate services, and which would you pull out first? Split only on real pressure: separate scaling, several apps sharing model access, different deploy pace. Model access often moves out first as a gateway.
How does streaming change what each layer has to do? Every hop must forward tokens, not buffer them. The API saves the full reply once the stream ends, and output checks have to work on partial text.
The model API is stateless. Which layer owns conversation history, and how does it reach the model? The application API stores history. Orchestration rebuilds the prompt each turn and trims or summarizes old turns to fit the context window.
Your main provider starts timing out. Which layer handles it, and what does the user see? Model access runs bounded retries with timeouts, then falls back to another model or provider. The client shows a clear message, not a hang.
Where would you put guardrail checks, given they are not a layer of their own? Check input before orchestration, check tool calls before they run, check output before the client. Name the latency each blocking check adds.

What you can say

I think of a production LLM app as a normal web app plus two extra layers, orchestration and model access, with the model as one dependency, not the system.
The client renders streamed output and holds the UI session, and the application API owns auth, business rules and conversation history.
Orchestration builds each request from retrieval, templates and tool calls and runs multi-step flows, backed by a data layer with the vector index, documents and app database.
Model access is the only code that talks to providers, hosted APIs or self-hosted inference, so it holds the keys, runs retries and timeouts, and tracks spend.
Safety checks, evals and telemetry aren't a separate layer, since they wrap every call.
These are logical layers, not services, so I'd start in one process, and from day one keep every provider SDK import inside model access, so a provider swap touches one module.

Weak answers to avoid

Describes the model as the whole system Prompt in, text out skips auth, history, retrieval and provider failures. Say the model is one dependency and show the web app around it.
Lets every service call the provider SDK directly Keys, retries, timeouts and spend tracking then spread across the codebase and a provider swap touches everything. Put all calls behind one model access module.
Draws each layer as its own microservice on day one These are logical layers. Splitting before there's a scaling or team reason adds network hops and deploys. Start in one process and keep the module boundaries clean.
Makes safety or logging one box in the stack Safety checks, evals and telemetry wrap every call, including tool calls and retrieval. Treat them as cross-cutting, not a step that runs once per request.

Interview lens

Likely follow-ups, what you can say, and the weak answers to avoid.

Want a quick review of the fundamentals? See the AI System Design cheatsheet.

← Back to all AI System Design questions

Pro · $10/mo

47 of 54 AI System Design answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →