Design a Rate Limiter

A rate limiter sets a cap on how many requests one client may send in a set amount of time — for example "no more than 2 posts per second" or "10 new accounts per day from one IP address". It protects your APIs from abuse (like a DoS attack, where someone floods you with traffic to knock you offline), keeps your bills down when you pay per call to a third-party service, and stops a sudden traffic spike from crashing your servers. The hard part is not deciding whether to add one — it is choosing which algorithm to use and where to put it.

Step 1: Clarify requirements and estimate scope

Before you design anything, agree on the scope with the interviewer. Here are the usual questions and the answers we will go with:

So the requirements are: block excess requests accurately, add almost no delay (never slow down a normal response), use little memory, work across many servers, return clear errors, and keep working even if the rate limiter itself breaks (this is called being fault tolerant).

Step 2: Where does the rate limiter live?

There are three places you could put it. Here they are, best last:

The middleware (or a managed API gateway, which often also handles SSL, login checks, and IP allow-lists) catches every request before it reaches an API server. Here is the idea: if the API allows 2 requests per second and a client sends 3 within one second, the first two go through and the third is blocked with an HTTP 429 Too Many Requests error.

There is no single right answer on placement — it depends on your tech stack, your team size, and whether you already run an API gateway. If you are on microservices and already have a gateway doing login checks, add the limiter there.

Step 3: Pick an algorithm

Five algorithms come up often. Know the trade-offs cold — interviewers love to poke at the edge cases.

A closer look: the token bucket

It needs just two settings:

• Bucket size — the most tokens the bucket can hold.
• Refill rate — how many tokens are added each second.

In the animation the bucket holds 4 tokens and the refiller adds +2 per second. A burst can spend all 4 tokens at once (that is the "allows bursts" trait), but over time traffic is capped at the refill rate.

How many buckets do you need? It depends on your rules:

• Different API endpoints → one bucket each (for example: 1 post/sec, 150 friend-adds/day, 5 likes/sec = 3 buckets per user).
• Limiting by IP → one bucket per IP address.
• A single global limit (for example 10,000 requests/sec) → one global bucket shared by everyone.

Step 4: High-level architecture

The counter has to live somewhere. A database is too slow — reading from disk on every single request would break the low-latency goal. So we use an in-memory cache (data kept in fast RAM, not on disk). Redis is the standard pick because it is fast and has two handy commands:

• INCR — add 1 to the stored counter.
• EXPIRE — set a timer after which the counter deletes itself.

Here is the flow: the middleware reads the counter for the right bucket from Redis. If the limit is reached, it rejects the request with a 429. Otherwise it forwards the request, adds 1 to the counter, and saves it back.

Rules and headers

You write the rules in config files on disk (Lyft open-sourced a popular format for this), and background workers load them into the cache:

domain: auth
descriptors:
  - key: auth_type
    value: login
    rate_limit:
      unit: minute
      requests_per_unit: 5    # no more than 5 logins per minute

When a client is blocked, send back a 429 plus some headers so it knows how to fix itself:

As an option, instead of dropping a blocked request you can add it to a queue to handle later (for example, orders that get rate-limited during a heavy spike).

Step 5: Rate limiting across many servers

Once you run more than one rate-limiter server, two tricky problems show up.

Race conditions

The "read the counter, check it, add 1" sequence does not happen all at once. If two requests both read counter = 3 at the same moment, they each compute 4 and save it — but the right answer was 5. Locks would fix this, but they are too slow. The standard fixes:

• Lua scripts — run the read-modify-write as one single, uninterruptible step inside Redis.
• Redis sorted sets — used by the sliding-window-log approach.

Synchronization

The web tier is stateless (no server remembers anything between requests), so client 2 might hit rate-limiter 1 on one request and rate-limiter 2 on the next — and limiter 2 has no idea what limiter 1 has already counted. Sticky sessions (always pin one client to one limiter) seem to fix this, but they do not scale well and are not flexible. The right answer is one shared central store like Redis that every limiter reads from and writes to.

Performance and fault tolerance

• Multiple data centers / edge — users far from the data center see delay, so route them to the nearest edge server (Cloudflare ran 194 edge locations as of 2020).
• Eventual consistency — let the counters across servers sync up over a short time (eventual consistency) rather than forcing them to match exactly on every request (strong consistency), which would be too slow.
• Fail open — if Redis goes down, the limiter must not take the whole API down with it. It should let traffic through instead of blocking everything.

Wrap-up

A rate limiter is a deceptively deep 45-minute problem. The arc: clarify the scope (server-side, distributed, rule-driven) → place it as middleware → pick the token bucket for its burst tolerance → store counters in Redis with INCR/EXPIRE → handle race conditions (Lua) and synchronization (a central store) at scale → return 429 with Retry-After headers.

A few extra talking points if you have time:

• Hard vs soft limits — a hard limit never goes over the cap; a soft limit allows a brief overshoot.
• Layer 7 vs layer 3 — we limited at the application layer (HTTP); you can also limit by IP at the network layer with iptables.
• Client best practices — cache responses, respect the limit, catch 429 errors, and add back-off (wait a bit longer each retry) to your retry logic.

Practice

Check your understanding of rate limiters.