Rate Limiting at Scale

Concept#

A system with no limit on how much any single client can request is one bad actor (or one buggy retry loop) away from a self-inflicted outage. Rate limiting is the deliberate mechanism for capping how many requests a given client, identified by API key, user ID, or IP address, can make in a given window of time, protecting shared capacity from any one consumer monopolizing it.

The four standard algorithms#

FIXED WINDOW:
  Count requests in fixed, non-overlapping time buckets (e.g. per
  calendar minute). Limit: 100 requests per minute.
  Simple to implement (increment a counter, reset it when the
  window rolls over) but has a real edge-case flaw: a client can
  send 100 requests in the LAST second of one window and another
  100 in the FIRST second of the next window, 200 requests in
  under 2 seconds, despite a "100/minute" limit, because the two
  windows don't overlap.
 
SLIDING WINDOW (log or counter):
  Instead of resetting at fixed boundaries, count requests in a
  CONTINUOUSLY MOVING window of the last N seconds, recalculated
  on every request. Fixes fixed window's boundary-burst flaw at
  the cost of needing to track more state (a log of recent request
  timestamps, or a weighted blend of the current and previous
  fixed windows as an approximation).
 
TOKEN BUCKET:
  A bucket holds up to N tokens, refilling at a steady rate (e.g.
  +10 tokens/second, capped at 100). Each request consumes 1 token;
  if the bucket is empty, the request is rejected/delayed.
  Naturally allows BURSTS up to the bucket's capacity (a client that
  hasn't made requests in a while has a full bucket and can burst),
  while still enforcing a steady-state average rate over time, a
  genuinely useful property fixed/sliding windows don't have.
 
LEAKY BUCKET:
  Requests enter a queue (the "bucket") and are processed ("leak
  out") at a FIXED, constant rate, regardless of how bursty the
  input is. Excess requests beyond queue capacity are dropped.
  Smooths bursty input into a steady output rate, useful when the
  downstream system genuinely can't handle ANY burst at all, at the
  cost of added latency for requests waiting in the queue.

Token bucket is the most commonly used algorithm in practice specifically because "allow reasonable bursts, enforce a steady average" matches how real client traffic actually behaves, a user opening several tabs at once shouldn't be instantly rate-limited the way a sustained flood should be.

Why rate limiting must live in shared state, not per-server memory#

This is the point where rate limiting directly depends on the scalability topic's core lesson: once a service is horizontally scaled behind a load balancer, a rate limiter that tracks request counts in each server's own local memory is fundamentally broken. If a client's requests get round-robined across 3 servers, each server sees only roughly a third of that client's actual traffic, and each one independently thinks the client is well under the limit, the effective limit the client experiences becomes roughly 3x the intended limit, silently, with no error or warning.

❌ Per-server in-memory rate limiter (3 servers, round robin):
   Client sends 300 requests, limit is "100/minute":
   Server A sees ~100 requests → thinks client is exactly at limit
   Server B sees ~100 requests → thinks client is exactly at limit
   Server C sees ~100 requests → thinks client is exactly at limit
   ACTUAL total: 300 requests got through, the true limit was
   silently tripled by horizontal scaling.
 
✅ Shared-state rate limiter (e.g. Redis, checked by every server):
   Every server increments/checks the SAME counter for this client,
   stored in Redis, regardless of which server the request landed
   on, the count reflects the client's TRUE total across the fleet.

This is exactly the same "externalize state instead of relying on server-local memory" principle from the Scalability topic, applied to a different kind of state (request counts instead of sessions), and it's a genuinely easy trap to fall into, because a naive rate limiter often works correctly in local development (one process, one instance) and only reveals the bug once deployed behind a real load balancer with multiple instances.

The HTTP contract: 429 and Retry-After#

When a client exceeds its rate limit, the standard, well-behaved response is HTTP status 429 Too Many Requests, typically paired with a Retry-After header telling the client concretely how long to wait before trying again:

HTTP/1.1 429 Too Many Requests
Retry-After: 30
Content-Type: application/json
 
{ "error": "rate_limit_exceeded", "retryAfterSeconds": 30 }

Well-behaved clients (and most modern HTTP client libraries) will honor Retry-After automatically, backing off for the specified duration before retrying, this is a real, load-bearing part of the contract, not just a nicety, because a client that retries immediately on a 429 without honoring Retry-After just adds to the load the rate limiter was trying to protect against in the first place.

Try It#

Your API uses fixed-window rate limiting at 100 requests/minute, with windows aligned to the top of each minute (:00 to :59). A client sends 100 requests at 12:00:59 and another 100 requests at 12:01:00. Does this violate the "100 requests per minute" limit, and why or why not according to the algorithm as implemented?

Implement It Yourself#

A minimal token bucket rate limiter, the actual mechanism behind the most commonly used real-world rate-limiting algorithm:

class TokenBucket {
  constructor({ capacity, refillRatePerSecond }) {
    this.capacity = capacity;
    this.tokens = capacity; // start full
    this.refillRatePerSecond = refillRatePerSecond;
    this.lastRefill = Date.now();
  }
 
  _refill() {
    const now = Date.now();
    const elapsedSeconds = (now - this.lastRefill) / 1000;
    const tokensToAdd = elapsedSeconds * this.refillRatePerSecond;
    this.tokens =

This is genuinely how production token-bucket limiters work, just backed by shared state (Redis, typically using INCR plus EXPIRE, or a Lua script for atomicity) instead of a single in-process object, the refill-based-on-elapsed-time math shown here is the real core logic, not a simplification of it.

Under the Hood#

A shared, atomic counter check under concurrent access, the exact correctness requirement a Redis-backed rate limiter has to satisfy, is the same class of problem as coordinating concurrent access to a single-threaded event queue, covered in The Node.js Event Loop: the rate limiter's atomicity requirement exists precisely because, without it, concurrent requests can interleave in ways a single-threaded mental model wouldn't predict.

Common Mistakes#

1. Implementing rate limiting with per-process in-memory state behind a load balancer#

// ❌ Works fine locally (1 process); silently multiplies the effective
// limit by the number of server instances in production
const requestCounts = new Map(); // lives only in THIS process's memory
function isAllowed(clientId) {
  const count = requestCounts.get(clientId) ?? 0;
  if (count >= 100) return false;
  requestCounts.set(clientId, count + 1);
  return true;
}

As covered in Concept, this silently multiplies the true effective limit by the number of server instances, since each instance only sees a fraction of a given client's total traffic. The fix is tracking counts in shared state (Redis or similar) that every instance reads from and writes to.

2. Returning a bare 429 with no Retry-After header#

HTTP/1.1 429 Too Many Requests
// ❌ no Retry-After, client has no idea how long to wait, and may
// retry immediately, adding more load to an already-overloaded limiter

Without Retry-After, well-behaved clients are left guessing how long to back off, and poorly-behaved ones may retry immediately, defeating much of the point of rate limiting in the first place. Always pair a 429 with a concrete Retry-After value.

3. Using fixed-window limiting where burst-at-the-boundary behavior actually matters#

"Our limit is 1000 requests/minute, fixed window, aligned to the clock." // ❌ if bursts near window boundaries are a real concern

As shown in Try It, fixed window allows up to 2x the stated limit in a worst-case boundary burst. If genuinely even pacing matters (e.g. protecting a fragile downstream system), sliding window or token/leaky bucket are more accurate choices, fixed window's simplicity comes at the cost of this specific, well-known gap.

Best Practices#

• Enforce rate limits in shared state (Redis or equivalent) reachable by every server instance, never per-process memory, once horizontally scaled.
• Prefer token bucket for client-facing APIs where occasional legitimate bursts (a user opening multiple tabs, a client retrying after a network blip) shouldn't be punished the same as sustained abuse.
• Always return 429 with a Retry-After header, this is a real contract well-behaved clients depend on, not an optional nicety.
• Rate limit at multiple granularities where appropriate, per-IP (catches unauthenticated abuse), per-API-key/user (catches abuse from an authenticated but misbehaving client), and sometimes per-endpoint (a cheap health-check endpoint can tolerate a much higher limit than an expensive search endpoint).
• Enforce rate limits as close to the edge as practical (CDN/API gateway before origin servers), rejecting an over-limit request before it consumes real backend capacity is strictly cheaper than rejecting it deeper in the stack.

Performance Tips#

• Token bucket and leaky bucket both require only a small, fixed amount of state per client (a token count and a timestamp, or a queue depth), this is cheap to store and check even at very high request volumes, unlike a sliding-window log approach that tracks every individual request timestamp.
• Checking and decrementing a rate limit counter in Redis should be done atomically (e.g. via a Lua script or Redis's INCR+EXPIRE combination), a naive read-then-write from application code is a race condition under concurrent requests, letting more requests through than the limit intends.
• Rejecting requests early (at the edge, before touching a database or doing expensive computation) means the cost of an over-limit request is just the rate-limit check itself, nearly free compared to letting it partially execute before being rejected downstream.