This round tests whether you can architect a production system around an LLM you neither train nor control. You treat the model as a probabilistic, high-latency, rate-limited, occasionally-unavailable dependency. You design everything else: request routing, retrieval, conversation state, caching, guardrails, fallbacks, cost controls, and observability.

It differs from two adjacent skills. Model/ML design is about architectures, training data, and weights. Classic system design assumes deterministic, cheap, millisecond services. AI system design sits between them and inherits the hard parts of both, plus constraints unique to LLMs:

• Outputs are non-deterministic and can be wrong, unsafe, or off-format.
• Cost is metered and varies with model choice, token volume, and call count.
• Latency is often much higher and less predictable than a normal service call.
• The model is a third-party black box with quotas, filters, and outages.

A complete design must therefore cover the system around the model, not only the model call.

Think of six layers, each with a clear job:

• Gateway/proxy: the single entry point in front of all model calls. Handles auth, per-tenant rate limiting and quotas, request/response logging, caching, key management, and provider routing. It is the control plane that keeps model access from sprawling across your codebase.
• Orchestration: the application logic that turns a user request into one or more model calls. Prompt assembly, retrieval, tool calling, multi-step workflows or agent loops, retries, and fallback logic live here.
• Model layer: the LLMs themselves plus any embedding and reranking models, possibly spanning multiple providers, regions, and tiers.
• Retrieval: vector stores, keyword indexes, and databases that inject relevant context (RAG) so the model answers from your data, not just its weights.
• Memory/state: storage for conversation history, user profiles, and long-term agent memory, since the API is stateless.
• Guardrails: input and output checks for safety, prompt injection, personal data, and format validation, placed around the model calls.

Cross-cutting concerns wrap all of it: observability (tracing, token accounting, cost), evals, and a feedback/data flywheel.

LLM latency has two distinct components, and conflating them is a common mistake.

Time-to-first-token (TTFT) is the wait before output begins. It includes prompt processing, network time, queueing, retrieval, and checks that run before generation. TTFT grows with input length.

Total latency (or end-to-end time) is TTFT plus the decode phase, where tokens are generated one at a time. Decode is memory-bandwidth-bound and roughly linear in the number of output tokens.

Why both matter: for interactive UX, TTFT is what determines whether the app feels responsive. With streaming, the user starts reading as soon as the first tokens arrive.

Design implication: to cut TTFT, shrink the prompt (trim retrieved context, use prompt caching), and stream. To cut total time, cap output length, use a faster model, or split work.

Streaming sends tokens to the client as they are generated rather than waiting for the full response. It matters because it collapses perceived latency. With a 300-token answer, the user sees words within a second instead of staring at a spinner for five.

Architecturally, streaming forces choices through the whole stack:

• Transport: Server-Sent Events (SSE) is the common choice for one-way token streams; WebSockets when you need bidirectional (voice, interrupts).
• Connection handling: each generation keeps a connection open, which affects connection limits, memory use, and autoscaling.
• Error handling gets harder: a failure mid-stream has already shown the user partial output.
• Post-processing tension: output guardrails and JSON validation want the whole response, but streaming shows tokens before you can validate them.

Streaming improves perceived speed, but it does not reduce total cost or generation time. It also makes output validation harder because users see partial text.

Prompt caching lets a model service reuse work for a prompt prefix that repeats across requests. Everything before the cache boundary must match exactly, including whitespace and tool definitions.

A cache hit avoids some prompt-processing work. This can lower input cost and time to first token, although billing rules, expiry, and minimum prefix lengths vary by provider.

Where it applies: a long, stable system prompt; few-shot examples; a large document or tool schema you ask many questions about. It also fits conversation history that grows by appending.

A common mistake is placing a timestamp or unique request identifier near the top. That changes the prefix on every request and prevents hits. Put stable instructions and examples first, followed by changing conversation or user content. Measure hit rate and savings using current provider rules.

An LLM gateway is a single service that all model traffic flows through, the way an API gateway fronts microservices.

What belongs in it:

• Authentication and provider key management, so application code never holds raw provider keys.
• Per-tenant and per-user rate limiting, quotas, and spend caps.
• Routing: choosing a model, provider, or region per request, and failing over when one is down.
• Caching: exact-match and semantic caches checked before the model is called.
• Observability: structured logging of prompts, responses, token counts, latency, and cost, with a request ID that ties everything together.
• Guardrail hooks: calling input/output safety checks.
• Retries and timeouts with sensible backoff.

The benefit is leverage. When a new cheaper model ships, or a provider has an outage, or finance needs a per-team cost report, you change or read one component.

The tradeoff: a gateway is now a critical path dependency, so it must be highly available and low-overhead. Otherwise it becomes the bottleneck it was meant to prevent.

Model routing is deciding, per request, which model handles it, instead of sending everything to one expensive frontier model. The goal is to match each request to the cheapest model that can do it well. That cuts cost and latency without hurting quality where it counts.

A cheap-first cascade is the simplest routing pattern. You try a small, fast, cheap model first. If its answer is good enough, you return it.

The savings depend on current provider prices and traffic mix. Measure how often the small model succeeds, then include the cost and delay of requests that must be retried on a larger model.

Two things to watch. First, the escalation signal is the hard part; a bad one either escalates everything (no savings) or nothing (quality drops).