How do you design APIs for scalability?

expert Pro

The full answer covers the core mechanics, when this pattern wins, and the trade-offs interviewers most often probe. It includes a code sample. Read it once at your own pace, then try to recall the structure from memory before the interview. Pro also includes this question's interview lens - the likely follow-up probes and what you can say in the room.

Full answer + code samples + AI explanations

Unlock to read the complete answer for this premium question.

Browse free API Design questions → Unlock all · $10/mo

Guided practice for API Design One question at a time. Answer out loud, get graded, see what you missed.

Related concept

Scale and availability: what statelessness makes possible, and what still fails

Holding no session is what lets the servers scale out. For availability, the other half, assume parts fail and stop one failure from spreading.

For this question · chapters 1 and 2 of 3

1/3 Session in memory breaks the balancer

Session in memory breaks the balancer

Most systems start with one server, box A, that keeps each session in its own memory. Box A holds sid 7f3a, the session ID for user 124, plus 811 other sessions. Nothing is wrong yet, and every request from that browser comes back with status 200. But watch two counts: how many servers are running, and how many servers can answer you.
The whole chapter is about one gap: the servers that are running, and the servers that can answer you. Traffic grows, so you add a second server and put a load balancer in front of both. Now two servers are running, but only the first server, box A, can answer you. The gap opened as soon as there were two of something.
The request was correct, and it still came back 401. The load balancer sent the request to box B, a server that had never seen session sid 7f3a. The user did nothing, but the load balancer signed them out. The balancer will sign them out again on roughly half of their requests.
Sticky sessions are popular because the cookie pins each user to box A, the server holding the session, so the error goes away. Then box A is deployed and loses everything in its memory, so 812 people are signed out at once. Sticky routing still sends those 812 people to box A, the server that lost their sessions. So the sticky-session patch fails at exactly the moment you needed it.
In practice, stateless does not mean there is no state. It means no single server owns the state. The session moves out of the server process into Redis, which keeps sid 7f3a with a thirty minute expiry. Both servers read the session from Redis, so both return 200.
The other place a session can live is the client itself. The client holds a signed token that carries user 124 and an expiry time. Each server checks the signature on its own, so there is no session store to run. The trade-off is revocation, which means cancelling a session early. A shared store lets you delete a session right away. But a token works until it expires, so you keep the expiry short.
This is what you mean when you say a service scales horizontally. None of the four servers holds a session, so any request can go to any server. Four servers are running, and at last all four can answer requests. A fifth server needs only a new number in a config file, because the fourth server holds nothing worth copying.

Use these in order

The servers are now replaceable, so you could add one, but try these options first. Their order matters more than the list itself. At three thousand requests a second, checkout answers in 950 ms at the 95th percentile. Each row names one real part of that wait, and the rows add up to 950 ms. A bigger server is already used up, because doubling the machine is a real first move that stops at one machine.
Caching the hot reads comes first, because an afternoon of work removes 560 ms, more than half the wait. The same product page is requested 82 times a second, and a cache answers that page once instead of 82 times. The cost is stale data: a read can return a page up to a minute old. You also have to handle invalidation now, which means removing a cached page when the product data changes.
The primary database is answering reads that it never had to answer. Reads outnumber writes about ten to one here, so sending the reads to replicas (copies of the database) saves 80 ms. The cost has a name that interviewers listen for: replication lag, the delay before a write reaches the replicas. A read straight after a write can miss that write, so any path that must see its own write stays on the primary.
Connection pooling needs no new machines and no new service, only one setting. Opening a new database connection for every request costs more than most of the queries run on that connection. Reusing open connections from a pool takes 27 ms off the wait. But pool size becomes a limit you have to tune, and one slow query now holds a connection that another request wanted.
Moving work off the request saves time by changing the contract with the client. The confirmation email goes out inside the request, so the request waits for the email. Put the email on a queue instead, and the request stops waiting, which cuts 60 ms from the response time. An asynchronous API is this trade: the client gets a 202 now and finds out the result later.
Sharding (splitting the data across databases) comes last, and the numbers show why. Sharding cuts 25 ms from this p95 latency, the smallest saving of any option here. And sharding adds costs: every query now needs a shard key, you give up cross-shard joins, and you need a plan to rebalance the data. So shard when one database runs out of capacity, not to make a request faster.
Most of performance work is knowing which part of the wait nothing can remove. Field selection and gzip shrink the JSON from 180 kB to 24 kB, so the wait drops by 90 ms here. Sending less saves more over mobile, because bandwidth and battery both cost the user something there. The 108 ms left after this last optimization is the endpoint doing its work, and no optimization removes that time.

Chapter 1 · step 1 of 7

Most systems start with one server, box A, that keeps each session in its own memory. Box A holds sid 7f3a, the session ID for user 124, plus 811 other sessions. Nothing is wrong yet, and every request from that browser comes back with status 200. But watch two counts: how many servers are running, and how many servers can answer you.

1 of 7 Use ← → or swipe See the concept: all 3 chapters in it

Want a quick review of the fundamentals? See the API Design cheatsheet.

← Back to all API Design questions

Pro · $10/mo

74 of 92 API Design answers are in Pro.

Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.

Full answers + code
AI explanations, simpler or deeper
1,000 AI credits / month
Cancel anytime

Get Pro → See plans & pricing →