Once you run more than one server (Chapter 2), something has to pick which server gets each request. That something is a load balancer — a traffic cop sitting in front of your servers. The rule it uses to choose matters more than you'd think. It decides how evenly the work is shared, how the system copes with slow requests, and whether your caches stay useful.
The three canvases on the right show the three rules you must be able to compare in an interview. Step through each one with Play or Next / Back.
Round-robin
Round-robin
Give each new request to the next server in a fixed loop: 1, 2, 3, 1, 2, 3… A pointer moves one step forward after every request.
It's the default for good reason. It's easy to build, it keeps no notes about each server, and it shares work perfectly evenly when every request costs the same. Its weak spot is exactly that "when." It has no idea that the request it just sent to Server 1 will take 10 seconds while the others finish in 10 milliseconds.
Weighted round-robin gives each server a weight. A big 16-core machine might get weight 4 while a small 4-core machine gets weight 1, so the big one takes proportionally more traffic. Same simple idea, better when your servers aren't all the same size.
Least-connections
Least-connections
Count how many requests each server is handling right now, and send the next request to the server with the fewest.
This fixes round-robin's weak spot directly. A server stuck on slow requests piles up open connections, so the balancer naturally sends new traffic away from it and toward the idle servers. The price you pay: the load balancer now has to keep a live count for every server.
Least response time is a sharper version. It picks the server with the best mix of fewest active connections and lowest average response time — handy when speed really matters.
Round-robin and least-connections both assume servers are interchangeable — any server can handle any request. The moment that stops being true (for example, you want the same user to always hit the same cache), you need the third rule.
Consistent hashing
Consistent hashing
Place both the servers and the keys (a key is just an id, like a user id) on a circle, or "ring." Each key belongs to the first server you hit walking clockwise from it. Adding or removing a server only moves the keys in one slice of the ring — about 1 out of every N keys.
The simple way to hash — server = hash(key) % N — looks fine until N (the number of servers) changes. Add one server and almost every key suddenly points to a different server. That stampedes your databases and wipes out every cache at once. Consistent hashing exists to make changing N cheap: only a thin slice of keys moves, and everything else stays put.
Watch the third canvas: when Server D is added, only key 3 moves. When Server B is removed, only key 1 moves. Everything else stays exactly where it was.
Virtual nodes
The basic ring has a fairness problem. With only a few servers, their spots on the ring can be spread unevenly, and one server ends up owning a much bigger slice than the others.
Virtual nodes (vnodes)
Put each real server at many spots around the ring instead of just one. Each spot is a "virtual node." A server with 4 virtual nodes owns 4 small slices instead of 1 big one, so the load spreads out more evenly.
Systems like Apache Cassandra and DynamoDB give each real machine 64–256 virtual nodes. This also makes adding a server smoother. Instead of taking one big slice from a single neighbor, the new server takes small slices from many nodes at once.
Expect the follow-up: "What if one server gets a hot slice and overloads?" The senior answer is virtual nodes — put each real server at many spots around the ring so the load smooths out. Bringing up virtual nodes before you're asked signals you've actually used consistent hashing, not just read about it.
Consistent hashing isn't just for routing — it also splits up data
The same ring is how distributed databases split data across machines (called sharding), not just how a balancer picks a server. Put the storage machines on the ring. A row's key is hashed onto the ring and stored on the first machine you hit walking clockwise. This is exactly how Amazon Dynamo and Apache Cassandra spread their data — and it's why those systems can grow without one central node having to reshuffle everything.
Splitting data this way gives a database two useful properties:
| Property | What it means |
|---|---|
| Automatic scaling | You can add or remove machines as load changes, and only the keys in the affected slice move — roughly k/n keys (with k keys and n machines). No global re-hash, no flood of cache misses. |
| Different-sized machines | A machine's share of the data is set by how many virtual nodes it gets. A box with twice the disk and CPU gets twice the virtual nodes, so it owns more of the ring and stores more data. |
Notice the difference from before: here, virtual nodes aren't only smoothing out a hot slice — they're the dial for giving a bigger machine more data on purpose. Same trick, two jobs.
Round-robin and least-connections answer "which server handles this request." Ring-based splitting answers "which machine owns this data." Both are load balancing — one balances work, the other balances stored data. Interviewers love it when you connect the two with the phrase consistent hashing.
Replication on the ring
If one machine owns a key, losing that machine loses the data. So databases keep extra copies. The rule is a small add-on to the lookup:
Clockwise replica placement
After a key lands on its spot on the ring, keep walking clockwise and put copies on the next N
machines — where N is a number you choose (often N = 3).
Virtual nodes add one catch. Because each real machine sits at many spots on the ring, the next N spots clockwise might belong to fewer than N real machines — so you'd copy the data onto the same box twice and gain no safety. The fix: skip spots that lead to a machine you already picked, and count only N different real machines. For real safety, those copies are then spread across separate data centers (linked by fast connections), so one power cut or network failure can't take out every copy at once.
When asked how Cassandra or Dynamo stays available, say: "The key maps to a node on the hash ring, then I walk clockwise to the next N different real machines for the copies — skipping any virtual node that points to a box I already picked — and place them in separate data centers." Calling out the different-machine skip is the detail that shows you've thought past the textbook picture.
Layer 4 vs. Layer 7 load balancing
| L4 (Transport) | L7 (Application) | |
|---|---|---|
| What it sees | Just IP + TCP/UDP headers | The full HTTP request: URL, headers, cookies, body |
| Speed | Very fast (nothing to read) | A bit slower (must read the HTTP request) |
| Routing decisions | IP, port, protocol | Path, hostname, cookie, content type |
| TLS (encryption) | Passes it straight through | Decrypts it (so it can read the content) |
| Typical use | High-speed TCP routing | HTTP APIs, A/B testing, auth headers |
Most web apps use L7 (for example AWS ALB, nginx, or HAProxy in HTTP mode). L4 balancers (like AWS NLB) are used when you want maximum speed and don't need to look inside the request.
Health checks and failover
A load balancer that sends traffic to a dead server just causes errors. Health checks prevent that:
- Active health check: the balancer pokes each server (for example
GET /health) every 5 seconds. Two failures in a row → mark the server "unhealthy" and stop sending it traffic. - Passive health check: the balancer watches real traffic. If a server returns too many 500 errors or times out → pull it out of rotation.
When a server recovers, it's eased back in — often with a slow start that ramps its share of traffic from 0% up to 100% over about 60 seconds, so it isn't flooded the instant it restarts.
Sticky sessions
Some apps keep per-user data in a single server's local memory (usually for legacy reasons). Consistent hashing is one way to handle that; another is sticky sessions (also called session affinity):
The load balancer reads a cookie from the request and always sends that user back to the same server. It works, but it breaks easy scaling: you can't safely shut a server down without losing those users' sessions. That's why stateless servers plus an outside session store (Chapter 2) is the better long-term design.
Public IP in, private IPs out
A load balancer isn't just a routing rule — it's also the system's front door, and that has a security upside worth naming. Clients look up your domain (for example api.mysite.com) and get one public IP, which belongs to the load balancer. The web servers behind it sit on private IPs that work only inside your private network and can't be reached from the internet. The balancer is the one thing that connects the two.
This means you can scale, replace, or reboot the whole fleet of servers without any client ever learning a server's address — clients only ever know the balancer's public IP. It also makes you safer: an attacker can reach the load balancer but can't reach a web server directly. Saying "clients hit the LB's public IP; the LB talks to web servers over private IPs" is a one-line signal that you understand network isolation, not just request routing.
How to choose, out loud
| Situation | Pick |
|---|---|
| Uniform, stateless requests | Round-robin |
| Mixed or long request times | Least-connections |
| Same user must hit the same cache/DB | Consistent hashing |
| Servers of different sizes | Weighted round-robin |
| Maximum speed, no need to read HTTP | L4 (TCP) load balancer |
| Path-based routing, TLS termination | L7 (HTTP) load balancer |
Saying why you'd pick one — and naming the case where it falls down — is what separates a memorized answer from a designed one.
If the interviewer asks "what happens if the load balancer itself fails?", the answer is an active-passive pair: a standby balancer watches the main one, and a floating IP (or DNS failover) switches over to it within seconds. Cloud providers do this for you automatically (AWS ALB/NLB are redundant inside), but naming the pattern shows you think about failure at every layer.
Practice
Answer these to check you understood how a load balancer chooses a server.
1. What does a load balancer do?
2. What is round-robin's main weak spot?
3. Why use consistent hashing instead of plain hash(key) % N?
4. What problem do virtual nodes (vnodes) solve?
5. Why are web servers usually given private IPs behind the load balancer?