AI System Design.

The round tests whether you can build a reliable product around a model you did not train. You treat the model as a remote dependency. It is paid per token, sometimes slow, and non-deterministic, so the same input can give different output.

Model design asks how to make the model better. System design asks what to do when it is wrong, slow, or down.

Interviewers listen for numbers: cost per request, a latency target, and a plan for failure. Saying "just call the API" without those is the usual reason people fail.

A production LLM app is a normal web app plus two layers, orchestration and model access. The model is one dependency, not the system.

• The client renders streamed output and holds the UI session.
• The application API owns auth and business rules, and persists conversation history.
• Orchestration builds each request from retrieval, templates and tool calls, and runs multi-step flows.
• The data layer holds the vector index, documents and application database.
• Model access talks to providers and holds their keys. It runs retries and timeouts, and tracks spend.
• Providers are hosted APIs or self-hosted inference.

Safety checks, evals and telemetry are not a layer. They wrap every call.

These are logical layers, so start in one process. From day one, nothing outside model access imports a provider SDK.

// the only module that imports a provider SDK
export async function complete({ model, messages }) {
  return client.messages.create({ model, messages, max_tokens: 1024 });
}

Time-to-first-token (TTFT) is the wait from sending the request until the first token comes back. Total latency is the wait until the last token arrives.

Roughly, total = TTFT + output_tokens * time_per_output_token.

TTFT depends on prompt length, queue wait and prefill, the pass that reads the whole prompt before generating. It is what the user feels in streamed chat. Total latency depends on the number of output tokens and decode speed. It matters for JSON, tool calls and batch jobs, where nothing is usable early.

The two move independently. Trimming the prompt cuts TTFT. Trimming the answer cuts total latency.

So track both at p95, per route. A fast first token followed by a slow 800-token answer still fails a voice agent. One blended number hides which half is broken.

Streaming sends tokens as the model writes them, so the user reads the first words in under a second, not a spinner. Total time barely changes. The wait just feels shorter.

It changes the architecture because the reply stops being one JSON object. You hold a connection open, usually with Server-Sent Events (SSE), a one-way stream over HTTP. Every hop must forward chunks without buffering, and proxies, gzip and gateways buffer by default.

The HTTP status goes out before the answer exists. So a mid-stream failure needs an in-band error event, not a 500. Token counts also arrive last, so keep a copy of the stream on the server for logging, caching and output checks.

let full = '';
for await (const chunk of stream) {
  full += chunk.delta;                       // server-side copy
  res.write(`data: ${JSON.stringify(chunk)}\n\n`);
}
res.write(`event: done\ndata: ${JSON.stringify(usage)}\n\n`);
await logAndCache(full);                     // only possible after the last chunk

If the user stops a bad answer, abort the upstream call. Otherwise you pay for tokens nobody ever sees.

Prompt caching stores the model's internal state for a prompt prefix, its KV cache, so a later request with the same prefix skips recomputing it. The match is prefix-based from the first token, so put stable parts first and the changing user turn last. One edited character early invalidates everything after it.

// Anthropic-style: mark where the stable prefix ends
const req = {
  system: [
    { type: 'text', text: instructions + toolDocs,
      cache_control: { type: 'ephemeral' } }      // cache up to here
  ],
  messages: [{ role: 'user', content: userTurn }] // changes every call
};

Cost falls because cached tokens are read back at a steep discount. Writing the cache can cost more than plain input, so it pays off only on repeats. Providers also set a minimum prefix size, so short prompts never cache.

Latency falls because caching shortens prefill, the pass over the input before the first output token. Time-to-first-token drops, but generation speed is unchanged. Entries usually expire after minutes of inactivity, so repeated, bursty traffic gains most.

An LLM gateway, or AI proxy, is one internal service that every app calls instead of calling model providers directly. It is a reverse proxy for model traffic, so cross-cutting concerns live in one place.

// app code: one endpoint, provider-shaped body, no provider keys
const res = await fetch('https://gateway.internal/v1/messages', {
  method: 'POST',
  headers: { 'x-tenant-id': tenantId, 'x-trace-id': traceId },
  body: JSON.stringify({ model: 'claude-sonnet-4-6', messages, stream: true })
});

The gateway owns:

• provider keys and caller auth, so no key sits in app code
• per-tenant rate limits and quotas
• retries, timeouts and failover
• trace IDs, token accounting and request logs, so usage is measured once

Prompt templates, retrieval and business logic stay in the app. So do conversation state and response rendering.

Keep it thin. It sits on the hot path of every request, so it must pass streamed chunks through untouched. It is also a single point of failure, so it needs health checks and a bypass path.

Model routing sends each request to the model that fits what the request needs. Small, cheap models handle easy work, and large models handle the hard cases. The router decides before the call, using rules, request length or a small classifier.

A cheap-first cascade decides after the fact instead. You always call the cheap model first and check its answer. You escalate to the stronger model only when that check fails.

async function cascade(prompt) {
  const draft = await call('small-model', prompt);
  if (passesCheck(draft)) return draft;   // most traffic stops here
  return call('large-model', prompt);     // escalation pays twice
}

The tradeoff is that an escalated request pays for both calls and both latencies. A cascade saves money only while the escalation rate stays low. The check must also be cheap, because a check that calls a big model to grade defeats the point.