Rate Limiting

Concept#

The beginner framing: an API needs a way to stop any single client from making so many requests that they degrade service for everyone else, rate limiting caps how many requests a client can make in a given time period, rejecting the excess with a 429 Too Many Requests.

Fixed window, simple, but with a real boundary bug#

Limit: 100 requests / 60-second window
Window 1: [00:00, 00:59]   Window 2: [01:00, 01:59]
 
Client sends 100 requests at 00:59:59 (all counted in Window 1, allowed)
Client sends 100 MORE requests at 01:00:01 (Window 2 just reset, ALSO allowed)
 
→ 200 requests within a 2-SECOND span, against a "100/minute" limit

Fixed-window counting is the simplest implementation, increment a counter, reset it every N seconds, but has a genuine, demonstrable burst problem exactly at window boundaries: nothing stops a client from exhausting the ENTIRE limit in the last instant of one window and the entire limit again in the first instant of the next, since the two windows are counted completely independently. The advertised "100/minute" limit provides no actual guarantee about request density in any arbitrary 60-second span, only about each fixed, clock-aligned window individually.

Token bucket, smooths bursts, allows some burstiness by design#

Bucket capacity: 10 tokens
Refill rate: 1 token / second
 
Each request costs 1 token. No tokens available → reject (429).
Tokens refill continuously, up to the capacity cap.

A token bucket holds up to capacity tokens, refilling continuously at refill rate. A request is allowed only if a token is available (and consumes one); otherwise it's rejected. This naturally allows a genuine BURST up to the bucket's capacity (useful, a client that's been idle for a while has accumulated tokens and can legitimately send a quick burst), while the steady-state refill rate enforces the actual long-run average limit, and critically, because tokens refill continuously rather than resetting at fixed clock boundaries, there's no equivalent of the fixed-window doubling bug; the bucket's state at any instant reflects genuinely how much "budget" has accumulated, not which arbitrary window the clock happens to be in.

Sliding window, the most precise, at a higher cost#

Track EVERY request timestamp for this client (or an approximation).
On each new request: count timestamps within the last 60 seconds (exactly).

A true sliding-window-log tracks individual request timestamps and counts exactly how many fall within the trailing N-second window at the moment of EACH new request, this has no boundary bug at all, since "the last 60 seconds" is always computed relative to the current moment, not a fixed clock-aligned window. The cost is real: storing every individual timestamp (or a bucketed approximation, "sliding window counter," which interpolates between two fixed windows to approximate this without storing every timestamp) uses meaningfully more memory than a single counter or token count.

Try It#

Predict the outcome before checking the solution.

Rate limit: "1000 requests per hour", implemented as a FIXED window
  resetting at the top of every clock hour (e.g. 2:00:00, 3:00:00...).
 
A client sends 1000 requests at 2:59:58, 2:59:59, then another 1000
requests at 3:00:01, 3:00:02.

Is this client in violation of the "1000 requests per hour" limit as advertised?

Implement It Yourself#

Build a minimal, runnable token bucket rate limiter, the actual mechanism, not pseudocode:

class TokenBucket {
  constructor({ capacity, refillRatePerSecond }) {
    this.capacity = capacity;
    this.refillRate = refillRatePerSecond;
    this.tokens = capacity; // start full
    this.lastRefill = Date.now();
  }
 
  #refill() {
    const now = Date.now();
    const elapsedSeconds = (now - this.lastRefill) / 1000;
    const tokensToAdd = elapsedSeconds * this.refillRate;
    this.tokens 

The mechanism: #refill() computes tokens to add based on ELAPSED TIME since the last check, not a fixed clock boundary, this continuous, time-proportional refill is exactly what eliminates the fixed-window boundary bug demonstrated in Try It, since there's no "reset instant" for two clients (or one client) to exploit from opposite sides.

Under the Hood#

Rate limiting and pagination's limit parameter (covered in Pagination, Filtering & Sorting Design) solve related resource-protection problems from different angles, one bounds request FREQUENCY, the other bounds response SIZE per request, and a well-defended API needs both, since an unbounded limit combined with an otherwise-reasonable rate limit still allows a client to request enormous payloads at an "acceptable" request rate. Rate limiting is also a core defense against a specific class of dependency/supply-chain-adjacent abuse covered in Secrets Management & Dependency Security, an exposed or leaked API key without a rate limit backstop turns a credential leak into effectively unlimited resource consumption by whoever obtained it.

Common Mistakes#

1. Sizing a token bucket's capacity equal to the full intended rate#

new TokenBucket({ capacity: 1000, refillRatePerSecond: 1000/3600 }); // ❌ for a "1000/hour" limit

This permits the EXACT same burst problem as the fixed-window bug, a client that's been idle can burst all 1000 tokens instantly, then (combined with poor timing relative to another client or a retry) approach double the intended rate in a short window. Bucket capacity should generally be set well below the full period's total, sized to the burst tolerance actually intended, not the full long-run quota.

2. Rate-limiting only by IP address#

const key = req.ip; // ❌ breaks down behind shared NAT/corporate proxies, trivially bypassed via IP rotation

Many legitimate users can share a single IP (corporate NAT, mobile carrier NAT), causing false-positive throttling of unrelated users; conversely, an attacker with access to many IPs trivially bypasses IP-based limiting entirely. Rate limiting by authenticated identity (API key, user ID, session) when available is generally more accurate and harder to evade.

3. Returning a bare 429 with no indication of when to retry#

HTTP/1.1 429 Too Many Requests
(no Retry-After header) // ❌ client has to guess when it's safe to try again

Without a Retry-After header (seconds, or an HTTP date), well-behaved clients have no principled way to know how long to back off, leading to either overly-conservative retry delays (poor UX) or overly-aggressive immediate retries (worsening the exact problem the rate limit exists to prevent).

Best Practices#

• Use token bucket (or true sliding window) over naive fixed-window counting, fixed windows have a genuine, demonstrable boundary-doubling bug that scales with window size.
• Size bucket capacity deliberately below the full period's quota, capacity should reflect intended BURST tolerance, not just equal the long-run rate, or the same boundary-adjacent burst problem resurfaces.
• Rate-limit by authenticated identity when available, not bare IP address, more accurate, and much harder to trivially evade than IP-based limiting alone.
• Always return Retry-After on a 429 response, gives well-behaved clients a principled backoff signal instead of guessing.
• Layer rate limiting at multiple levels when it matters (per-IP as a coarse backstop, per-API-key/user for the primary limit), defense in depth against different evasion strategies.

Performance Tips#

• Token bucket state is O(1) per client (just a token count and a last-refill timestamp), dramatically cheaper to store and check than a true sliding-window-log's per-timestamp storage, while avoiding the fixed-window boundary bug entirely.
• Rate-limiting checks need to be fast and low-latency themselves (they run on the hot path of EVERY request), an in-memory or Redis-backed counter (for multi-instance consistency) is the standard choice over anything requiring a full database round-trip per check.