home
›
cheatsheets
›
ai_engineering
›
ai_system_design
Free to read Every card and diagram, and Quiz me
2
Free account Star cards. Your Quiz me marks are kept, so next time you drill just what you missed
3
Pro All 54 answers on this sheet, My sheet across every topic, and PDFs
A free account keeps your stars and Quiz me marks. Pro adds all 54 answers on this sheet, My sheet across every topic, and PDFs.
/
Read
Quiz me
Comfortable
Compact
0
Click any code to copy it
Done 0 /0
To review 0
On this sheet
01 Layers & the gateway02 Latency & streaming03 Context, state & memory04 Prompt & semantic caches05 Routing & cost06 Tenants, limits & queues07 Failures & fallbacks08 Guardrails & grounding09 Evals, tracing & rollout10 Designs interviewers ask for
The model The model is a remote dependency you didn't train. It's paid per token, rate-limited, sometimes slow or down, and non-deterministic . The design is everything you build around it. The API is stateless . Your server stores each conversation and resends all of it every turn, so every turn costs more than the one before. Latency is two numbers. Time-to-first-token is what a streaming user feels. Total latency adds every output token, and a JSON reply waits for all of it. You pay for tokens , not requests, and output costs several times input. The usual waste is resent context or a big model doing easy work. Failures are normal traffic. Every call needs a timeout and a fallback that fails differently. Evals tell you when quality slips. The provider remembers nothing, so your app resends the full history every turn. The gateway adds the key, fails over on errors, and streams tokens back.
Layers & the gateway the parts every production LLM app has The layers of an LLM app Name the layers Client, app API, orchestration, data, model access and the providers.
Keep providers swappable Let only the model access module import a provider SDK.
Tip Safety checks, evals and telemetry aren't a layer. They wrap every call.
LLM gateway Say what an LLM gateway does Every app calls it, not the provider. It owns keys, limits, failover and token counts.
List what stays in the app Prompts, retrieval, business logic and chat state.
Trap Every request goes through it, so it can take everything down. Keep it thin, with health checks and a bypass.
What the round tests Tell it apart from model design Model design makes the model better. System design plans for when it's wrong, slow or down.
Know what interviewers listen for Numbers. A cost per request, a latency target and a plan for failure.
Trap Saying "just call the API" without those numbers is the usual reason people fail.
Hosted API or your own GPUs Pick hosted or self-hosted Hosted APIs. Self-host only when a named constraint forces it.
Name the reasons to self-host Data that can't leave your network, steady volume, or weights you must pin or fine-tune.
Keep a self-hosted GPU busy Continuous batching. New requests join at every decode step.
Find a GPU's real capacity limit KV cache memory, not the weights. Long contexts shrink the batch.
Tip Utilization decides it. A reserved GPU bills all day, so spiky traffic loses even at high volume.
Latency & streaming the first token, the last token, and the stream Two latency numbers Define time-to-first-token The wait for the first token. Trim the prompt or cache its prefix to cut it.
Define total latency The wait for the last token. A shorter answer cuts it.
Say what limits prefill and decode Prefill is bound by compute. Decode is bound by memory bandwidth.
Trap One blended number hides which half is slow. Track both at p95, per route.
Stream the reply Say why streaming matters The first words show in under a second. Total time barely changes.
Report a failure mid-stream Send an in-band error event. The 200 already went out.
Log and cache a streamed answer Keep a copy on the server. Token counts arrive last.
Trap Proxies, gzip and gateways buffer by default. Every hop must pass chunks straight through.
Streaming at scale Survive a dropped connection Write tokens to a Redis run buffer. Replay from the browser's Last-Event-ID.
Set up the load balancer No buffering, an idle timeout longer than any reply, and a heartbeat every 15 seconds.
Tip Serve SSE over HTTP/2. On HTTP/1.1 a browser allows only six connections per origin.
Set a latency budget Split a 3-second p95 budget 100 ms for auth and input checks, 400 for retrieval, 1,200 to first token, 1,300 spare.
Enforce the budget Carry one deadline with the request. Stages drop optional work when time runs short.
Budget a voice assistant Under 800 ms, last word to first sound. Endpointing and first token take most of it.
Trap Stage p95s don't add up to the end-to-end p95, because slow stages rarely hit the same request. Leave slack.
Context, state & memory the API forgets, so you store, resend and trim The API is stateless Give the model memory of past turns Store every turn yourself, and resend the whole history each call.
Say why long chats get expensive Turn 10 pays again for turns 1 to 9. Input grows with the square of the turn count.
Keep prompt caching working in a chat Append only. Never rewrite earlier turns.
Fit the context window Define the context window The most tokens one request can hold, prompt and reply together.
Budget the window Count tokens first. Cap chunks at top-k, trim history, and leave room for the reply.
Decide what to drop The oldest turns or the lowest-ranked chunks. Pick the rule up front.
Say why not to fill it Facts buried in the middle get missed. Every token also adds cost and latency.
Trap Going over the window is a hard error, not a slower answer.
Store conversation state Keep the source of truth An append-only message log in Postgres, keyed by conversation and sequence.
Serve the next turn fast Redis holds recent turns plus a rolling summary, rebuilt from the log on a miss.
Trim history without losing cache hits Let the window grow past a budget, then fold old turns into the summary in one step.
Trap Never make Redis the source of truth. One eviction loses history.
Give the model your data Add facts the model lacks RAG, for a large corpus or one that changes often.
Fix a format or house tone Fine-tune. It teaches behavior, not facts.
Use a small, stable corpus Long-context prompting. It's the fastest to build.
Trap Fine-tuning doesn't add knowledge reliably. Baked-in facts go stale, and each fix needs a retrain.
Memory for a long-lived agent Split an agent's memory Working set, step log, facts and recipes, each with exactly one writer.
Write long-term facts A background job reads the step log and supersedes old facts.
Trap Treat stored memory as untrusted. A fact saved from tool output can carry an injection that fires days later.
Prompt & semantic caches reuse work the model already did Prompt caching Say what a prompt cache stores The KV cache for a prompt prefix, so a repeat skips recomputing it.
Order the prompt for hits Tools, system rules, shared documents, and the live turn last.
Know when it pays off On repeats. Reads are cheap, but a write can cost more than input.
Measure the hit rate Cached read tokens over all input tokens, per route.
Tip On Anthropic, the cache lasts 5 minutes and each read resets it. A prefix under the model's minimum, 512 to 4,096 tokens, never caches.
Trap One changed character early, like a timestamp, misses the cache for everything after it.
Exact-match and semantic caches Cache identical prompts Hash the prompt. It's never wrong, but hits are rare. Check it first.
Cache prompts that mean the same thing Reuse the closest past answer above a similarity threshold.
Key a semantic cache By tenant, permissions, model, prompt version and embedding model, not text alone.
Tune the threshold Sweep it on labeled data. Keep false hits under budget.
Trap "Cancel my order" and "can I cancel my order" embed close together. One is an action, one asks about policy.
Routing & cost the cheapest model that clears the bar Route or cascade Pick the model before the call That's routing. Rules, request length or a small classifier decide.
Try the cheap model first A cascade. Escalate to a large model only when its answer fails a check.
Know when a cascade saves money Only while few requests escalate. Each one pays for both calls.
Trap The check must be cheap. Grading with a big model defeats the point.
Design the routing layer Choose a model for a request Rule out models missing a hard need, like tools. Take the cheapest that meets the tier.
Store the policy As versioned config. Log the route and version on every request.
Fit routes to eval data Grade each model per request type. Each type gets the cheapest one that passes.
Tip Pin a conversation to one model. Switching mid-thread changes its voice and wastes the prefix cache.
Trap A cheap tier that escalates 40% of requests can cost more than the strong model alone.
What an LLM system costs Price one call Input tokens plus output tokens, each at its own rate.
Pick the number to track Cost per business outcome, like a resolved ticket. Not cost per call.
Estimate cost per conversation Sum every turn by token class. Prefix caching cuts the resent history, but only within its TTL.
Report the estimate As p50 and p90 from real traffic. Rare long chats dominate spend.
Tip Output tokens cost several times input, and a frontier model can cost ten times a small one.
Tenants, limits & queues share scarce capacity fairly under load Limit each tenant Meter tenant load In tokens per minute. One request can be 200 tokens or 200,000.
Stop noisy neighbors A reserved floor per tenant, plus a shared pool for bursts.
Count output you can't know yet Reserve input plus estimated output at admission. Refund the difference afterward.
Bill teams for their usage The gateway tags every call with a cost center and refuses untagged calls.
Trap Cap concurrency per tenant too. 500 slow requests starve everyone, even inside a token budget.
Enforce spend budgets Check the budget before a call Reserve the worst case atomically, then write back the real cost.
Act as spend climbs Alert at 80%. Downgrade the model at the soft limit. Reject with 402 at the hard cap.
Trap Streamed calls report usage only at the end. Settle when the stream closes, aborts included.
Handle 429s and backpressure Stay under provider limits A local token bucket that reserves estimated input plus output tokens.
Retry a 429 Honor retry-after, then back off with full jitter. Keep a retry budget.
Push back when your queue fills Keep the queue bounded. Fail fast with your own 429 and Retry-After.
Trap An unbounded queue only turns 429s into client timeouts.
Scale across providers and regions Plan capacity per region Quotas are fixed per key or region. Size each region for its load after a failure.
Balance the load By remaining token headroom, not round robin.
Keep prefix cache hits Hash the stable prefix to one provider. Spill over only when it's out of headroom.
Stop a failover retry billing twice Key results by the client's request id in a globally replicated store.
Trap Don't autoscale the app tier on CPU. CPU stays flat while calls pile up, so scale on in-flight requests.
Queue async and long jobs Accept a 500-page document 202 Accepted Return a job id. The client polls, or you call a webhook.
Resume a failed job Save each chunk's result as it lands. A retry resumes at chunk 340, not 1.
Drain tenants fairly Deficit round robin per tenant, counted in estimated output tokens.
Skip work nobody waits for Stamp each job with a deadline, and drop expired ones at dequeue.
Tip If results can wait hours, use the Batch API. It costs about half as much.
Trap Require an Idempotency-Key on submit. Otherwise a client retry starts a second job.
Failures & fallbacks stay up when the model isn't Match the fix to the failure Retry Fail over Repair
Retry once, then fail over Handle a timeout or 5xx
Back off, or use a smaller model Handle a 429 rate limit
Refuse plainly and log it Handle a content filter block
One repair try with the error Handle output that breaks the schema
Raise the token cap Handle a reply cut off at the token cap
Tip A fallback must fail differently. A retry repeats the same path, so it only fixes brief blips.
Trap Sending a blocked prompt to a looser model elsewhere is filter shopping. Auditors read it that way too.
SLOs and a degradation ladder Pick the SLOs Availability, p95 time-to-first-token and a sampled quality score.
Count a request as available Only for a valid answer that isn't a refusal. HTTP 200 isn't enough.
Degrade in rungs Drop the reranker, then the big model, the live call, the instant reply, and last the answer.
Choose the rung From the error budget's burn rate, not by hand.
Tip Tag every degraded reply, or quality metrics look healthy while users get less.
Reliable JSON and tool calls Enforce the output shape One JSON Schema per tool, sent in strict mode and reused in your validator.
Check what strict mode can't Meaning. Check business rules, like whether that order id exists.
Stop a retry running a tool twice Derive an idempotency key from the tool call id.
Trap Strict modes support only part of JSON Schema. Keep schemas flat.
Guardrails & grounding stop bad input, bad output and bad actions Where guardrails go Check input Before the call. It saves tokens but can't see what the model will say.
Check output After the call. It catches the real harm, but it fights streaming.
Check tool calls Before they run, with a policy per tool. It stops the side effect itself.
Tip Hold back a short tail of streamed text and release what's behind it. A check can then stop a reply in time.
Ground answers or abstain Abstain before generating When the top chunks score below a threshold.
Make claims checkable Return each claim with its chunk ids. Reject any citing text you never retrieved.
Verify each claim Score it against its cited text with an NLI model. Cut what isn't supported.
Cite web sources in an answer engine Answer only from numbered chunks, tagging each sentence with its chunk ids.
Trap Score coverage apart from accuracy. Otherwise guessing looks better than an honest abstention.
Defend against prompt injection Handle retrieved text Keep it in a data slot, never pasted into the system prompt.
Authorize a tool call Check the user's permissions in code, not the model's plan.
Break the lethal trifecta Never give one agent private data, untrusted content and a way to send data out.
Sandbox an agent's tools A microVM or container per run, with network egress denied by default.
Trap No filter reliably stops injection. A passing classifier score is never the reason an action runs.
Keep PII away from providers Decide what may leave Classify the data first. The highest classes never reach a hosted API.
Mask PII in what you send Swap sensitive spans for placeholders like PERSON_1. Restore them in the reply.
Cover what the detector misses Back the filter with zero-retention, no-training terms. Detector recall is never 100%.
Trap Treat embeddings as PII. If they leave the region, so did the data.
Human review and escalation Send a request to a human On low confidence, a guardrail hit, a risky tool call, or when the user asks.
Pause an action for approval Save it with an idempotency key and a timeout. Deny on timeout if it's irreversible.
Tip Store each decision as a label with a reason code. It feeds evals, so escalations fall.
Evals, tracing & rollout see when quality drops, and ship changes safely What to measure Split evals from monitoring Offline evals gate each change. Online monitoring watches live traffic.
List the signals to watch Schema failures, groundedness, refusals, regenerates, escalations, p95 latency and cost.
Keep an LLM judge honest Pin its version, and check it against human labels.
Tip Tag each request with its prompt and model version. Slice by tenant and intent too.
Trace a multi-step request Trace one request One trace id, one span per step. Tokens go only on spans that called a provider.
Record token usage From the provider's response. Split input, output and cached input.
Trap Anthropic's input_tokens counts only the uncached part. Add the cache read and write counts for the total.
Roll out a prompt or model change Shadow the change first Mirror sampled traffic and drop the replies. Tools dry-run and writes are blocked.
Canary the change Start near 1% and ramp, bucketed by user id. Roll back automatically on bad metrics.
Move off a retiring model Pin snapshot ids, then replay evals and recorded traffic through the new one.
Trap Output is non-deterministic, so exact diffs prove nothing. Compare paired outputs with a judge model.
Improve from production data Log a trace you can replay Include the retrieved chunk ids, prompt version and model version.
Stop overfitting your evals Keep a frozen holdout set you never tune on.
Trap Thumbs and clicks are biased, because unhappy users stay silent. Label a small random sample too.
Designs interviewers ask for the classic prompts, one card each ChatGPT-style assistant Trace one turn Gateway, then the conversation service, then the provider. Tokens stream back over SSE.
Save the reply After the stream ends, so a dropped socket never saves half an answer.
Tip Keep the transcript in Redis and Postgres, not on the server. Then any instance can serve any turn.
Support bot over company docs Answer from the docs Only from retrieved passages, with a source link per claim.
Escalate to a human When nothing relevant comes back, the user is angry or asks twice, or it's billing or refunds.
Hand off without a cold start Pass the transcript, the passages and why it escalated.
Trap Deflection rate alone rewards a bot that stonewalls. Track it against CSAT.
Enterprise RAG with access control Store permissions Put the allowed group and user ids on every chunk as ACL metadata.
Apply permissions at query time As a filter during the search, never after it.
Handle an index that lags Re-check permissions on cited chunks, and tombstone deletes.
Trap A post-filter fetches ten hits, then drops most. Recall collapses for users with narrow access.
AI coding assistant Serve autocomplete A small fill-in-the-middle model, 200 to 500 ms at p95.
Keep autocomplete cheap Cancel the in-flight request on every keystroke.
Serve agent mode A large tool-calling model. It proposes diffs and runs commands in a sandbox.
Tip Measure autocomplete by acceptance rate, and whether a suggestion is still there 30 seconds later.
Nothing matches. Try a shorter word, or clear the search.
Practice Guided practice One question at a time, with the ones you miss coming back. Practice 54 interview questions Full answers, from beginner to expert.
Found this useful? Pass it on.
Share on LinkedIn
Share on X
Copy link
This sheet stays free. With Pro , you can:
Every answer on this sheet. Read the full answer to all 54 questions it links, not just the 7 free ones.My sheet. The cards you star and the ones you miss in Quiz me, from every sheet, on one page to quiz and print.PDFs. Download this sheet and My sheet, laid out for A4, with every question a link to its answer.The rest of Pro. Every answer in the question library, and 1,000 AI credits a month.
