Rate limiting shares capacity fairly, and the algorithm decides how

A limit exists for four reasons: it stops abuse and holds back a client stuck in a retry loop. The limit also caps what one caller can cost you and keeps one tenant from taking the capacity the others need. This limiter allows 100 requests per clock minute, and at 0:00 its counter starts at 0. The limit is a promise about a shared resource, not a punishment.

Most traffic looks like this: a client sends three requests in the first 40 seconds. The limiter counts them as 1, 2 and 3 and lets each one through. A client that behaves never reaches the limit at all. For that client, the counter is bookkeeping that the limiter never acts on.

The limiter refuses no request here, because the window is the budget for this minute and this burst spends that budget. A burst of 97 requests arrives at 0:59, the last second of the minute. The counter reaches 100 of 100, so this minute's window is now full. But the minute is about to end.

The reset at the window boundary is the whole flaw of a fixed window. At 1:00 the limiter sets its request counter from 100 back to 0. So the limiter keeps no record of the old window, not even the 97 requests from one second ago. A retry storm (a surge of retried requests) is exactly the traffic that triggers this flaw.

Counting requests per clock window leaves a loophole at the window boundary. A second burst of 100 requests arrives at 1:01, just after the boundary. The new window's counter reaches 100 again. So 197 requests pass in two seconds, under a limit of 100 per minute. The total is nearly twice the limit. But the limiter never refused a single request.

A sliding window closes the loophole and refuses the same second burst with 429. At 1:01, the last 60 seconds still include the 97 requests from 0:59 and the 3 from earlier, so the counter stays at 100. This fix is accurate but costs memory, because each caller needs a log of 100 timestamps instead of one integer. Most limiters use a compromise: a sliding counter that weights the previous window's count.

The fixed window reset its count at every minute boundary. A token bucket remembers the tokens already spent, and it allows a burst on purpose. Here the gateway's bucket has a capacity of 20 tokens and refills at 2 tokens per second. Capacity and refill are two product decisions: capacity sets the burst allowed at once, and refill sets the rate allowed over time.

The limiter allows a burst of 20 on purpose. A page load is a burst of requests, so a limiter that refused that burst would block normal use. Each token in the bucket lets one request through. Twenty requests take one token each, so the API serves all twenty and the bucket drops from 20 to 0.

The limit exists for exactly this request, the 21st. The capacity is 20 tokens, one per request, so a 21st request in the same instant has no token to take. The gateway refuses that request and sends a 429 back to the client. The client has to wait until a token is available again.

The refill rate is the long-run limit, and the limiter enforces it one token at a time. The limiter adds back 2 tokens per second, and four requests arrive at that same pace. Each request uses one token as it arrives, so the bucket of unused tokens never holds more than 1. The limiter never refuses a caller who stays at this rate.

The caller gets the same budget, bursty or smooth: an average of 2 requests per second. The bucket gains three tokens while no requests arrive, so it holds 4. A burst of four requests then uses all four tokens, one token per request, so the API serves every one. Four more tokens later refill the bucket to 4. A leaky bucket instead queues requests and passes them on at a fixed rate.

The API's three instances have to share one bucket, because separate buckets would allow 60 requests per burst, three times the limit. The gateway's Redis holds that one bucket, so three requests to three instances take three tokens from the same bucket. Per key, per IP and per endpoint limits are different buckets, and a global limit is another. You usually need more than one.

The limiter protects you, but the other half of rate limiting keeps your callers well behaved. Client A gets a 200 response with X-RateLimit-Limit 100, X-RateLimit-Remaining 1 and X-RateLimit-Reset 30 seconds. A client that can see its budget can stay inside it: Client A knows it has one request left until the reset in 30 seconds. The draft standard's RateLimit headers carry the same three numbers.

This refusal is a helpful error, because it tells each client when to come back. Both clients get a 429 Too Many Requests with Retry-After: 30, so each one knows to wait 30 seconds. The Retry-After value can be a number of seconds or an HTTP date. A 429 means this one caller is over its limit, but a 503 would mean the whole service is down.

Every instant retry costs the server work and achieves nothing. Client A ignores Retry-After, the header that says when to try again, and sends five retries in two seconds. The server answers all five with a 429 right away, so its refused count reaches 7. Retrying instantly is how one slow minute turns into an outage.

Client B shows the client half of rate limiting: read the header, wait, then succeed. Client B obeys Retry-After and waits the full 30 seconds. Then Client B sends just one request. That request gets a 200 and a fresh budget of 99. This loop is the half of rate limiting that interviewers ask about.

Retry-After tells each client how long to wait, so clients that all obey the same value come back at the same moment. Fifty clients got Retry-After: 30, so all fifty retry in the same second. The server receives 50 requests in 1 s. Fifty polite clients with the same wait are a burst, and the limiter was built to refuse a burst.

The fix is to make each client wait a different time. Jitter adds a random extra wait, so the server receives the same fifty retries spread over 30 seconds, from second 30 to second 60. Exponential backoff doubles every client's wait after each failure, but it doubles all the waits together. So without jitter, the clients all retry together again in every round.

I keep a token bucket of 20 in Redis, so all three instances share one count. The 20 is the burst I allow on purpose, and the refill of 2 per second is the rate. My 429 carries Retry-After, so a good client waits. My clients also back off exponentially with jitter, so they never retry at the same moment.