This round tests whether you can architect a production system around an LLM you neither train nor control. You treat the model as a probabilistic, high-latency, rate-limited, occasionally-unavailable dependency. You design everything else: request routing, retrieval, conversation state, caching, guardrails, fallbacks, cost controls, and observability.
It differs from two adjacent skills. Model/ML design is about architectures, training data, and weights. Classic system design assumes deterministic, cheap, millisecond services. AI system design sits between them and inherits the hard parts of both, plus constraints unique to LLMs:
- Outputs are non-deterministic and can be wrong, unsafe, or off-format.
- Cost is metered and varies with model choice, token volume, and call count.
- Latency is often much higher and less predictable than a normal service call.
- The model is a third-party black box with quotas, filters, and outages.
A complete design must therefore cover the system around the model, not only the model call.
This answer doesn't lend itself to a diagram - it reads best . No credits were charged.
Why there's no diagram: “”
The interactive diagram is below the answer - jump to diagram ↓ · Below it, the related concept . Jump to it ↓
The diagram below the answer is the concept . Jump to it ↓