How do you design APIs for high availability?
Scale and availability: what statelessness makes possible, and what still fails
Holding no session is what lets the servers scale out. For availability, the other half, assume parts fail and stop one failure from spreading.
For this question · chapter 3 of 3
Cascade and circuit breaker
- All of that assumed the parts work, but next, one of them gets slow. Checkout calls pricing, pricing calls inventory, and the page returns in 60 ms. Each service has a pool of forty threads, with no more than three busy. Remember that all three health checks return 200 up.
- The sentence to say out loud here is that slow is worse than down. Inventory is not down. But its database is saturated, so each call takes 9 seconds instead of 25 milliseconds. Every pricing thread that calls inventory waits, so pricing's pool of forty threads fills up. A call to a service that is down fails at once, but a slow service holds your threads.
- The failure now spreads up the chain of calls because each caller waits. Checkout's threads wait on pricing, and pricing waits on inventory, so all forty of checkout's threads are busy too. The user gets a 503 after thirty seconds of waiting. Three services are unusable, but not one process crashed, so the CPU graphs look idle through the whole incident.
- The one thing to say about availability is that every call over a network has a deadline. A missing timeout is the usual root cause. With a deadline on the call, the pricing service stops waiting for the inventory service after 800 milliseconds instead of 9 seconds. Each thread returns to the pool eleven times sooner, so the pricing service's thread pool drops from forty busy threads to twelve.
- Twenty failures in ten seconds move the breaker on pricing's calls to inventory from closed to open. A call to inventory now fails in 1 millisecond without leaving pricing's process, so pricing holds four threads instead of twelve. In half-open, the breaker's third and last state, one probe call every thirty seconds checks whether inventory has recovered. Inventory falls to six busy threads, because you stopped sending calls to a service that was already struggling.
- The pricing service answers from its last good price, which is three minutes old, so the page still renders. Accepting a three minute old price is a product question, not an engineering decision. Ask that question now rather than during the outage. Decide in advance what a degraded answer looks like on each screen.
- The Inventory service answered 200 through this whole outage. A health check that only proves the process is alive gives a false answer, and the platform trusts that answer. So the platform keeps sending traffic to an instance that cannot do its job. Check the dependency and let the readiness check fail, so the platform stops sending traffic to that instance.
All of that assumed the parts work, but next, one of them gets slow. Checkout calls pricing, pricing calls inventory, and the page returns in 60 ms. Each service has a pool of forty threads, with no more than three busy. Remember that all three health checks return 200 up.
Answer in two halves. Holding no session is what makes scaling out possible. Session state lives in a shared store or a signed token, so any server can handle any request. Then scale in this order: cache hot reads, add read replicas, and pool connections. Next, move slow work out of the request. Shard last, because sharding saves the least and costs the most. Availability is the second half. Every call gets a deadline. A circuit breaker stops calls to a service that is already failing. Slow is worse than down.
© LearnThatStack - diagrams may not be republished without permission.
Want a quick review of the fundamentals? See the API Design cheatsheet.
74 of 92 API Design answers are in Pro.
Full answers, code samples, and AI explanations that go simpler or deeper. Cancel anytime.
- Full answers + code
- AI explanations, simpler or deeper
- 1,000 AI credits / month
- Cancel anytime