In Chapter 1, one server did all the work. That is fine until too many people show up at once. Then that one machine runs out of CPU, memory, or open connections, and starts to choke. There are only two ways to fix this, and being able to explain the trade-off between them is a classic interview moment.
Step through the animation on the right. Watch one server's load climb until it is overwhelmed (it sends back a 503 error). Then more servers are added behind a load balancer (a traffic director), and requests get spread out evenly. One server even crashes to show why having extra boxes keeps you running.
Scale up (vertical)
Vertical scaling
Make the one machine more powerful: more CPU cores, more memory (RAM), faster disks. The design stays the same — the box just gets bigger.
This is the easy path. You change no code and deal with no extra complexity. But it has two hard limits. First, there is a ceiling — you can only buy a machine so big. Second, it is still just one machine: if it dies, everything goes down with it. There is no backup.
A typical "buy a bigger box" path on AWS (Amazon's cloud):
- t3.micro: 2 vCPUs, 1 GB RAM — a tiny starter app
- c5.2xlarge: 8 vCPUs, 16 GB RAM — early growth
- c5.18xlarge: 72 vCPUs, 144 GB RAM — a large single app
- At some point, the next size up costs 4× more for only 2× the speed
The price stops being worth it long before you hit the real technical limit.
Scale out (horizontal)
Horizontal scaling
Add more machines and put a load balancer in front of them. You grow by adding boxes, and the system keeps working even if one box fails.
This is how big systems really scale. The catch: your servers must be stateless (they must not store important data only in their own memory), so any server can answer any request. Anything that needs to be remembered — like login sessions or uploaded files — has to move to a shared place all servers can reach.
Use this line: "Vertical scaling is the simplest, so I'd start there, but it has a ceiling and no backup. To scale further and stay available, I'd switch to horizontal scaling behind a load balancer." That one sentence shows you weigh trade-offs instead of just naming buzzwords.
Making servers stateless
For horizontal scaling to work, every server must be interchangeable — any one of them can handle any request. That means moving "memory" out of the servers:
| What needs remembering | Where it should live instead |
|---|---|
| Login sessions / auth tokens | Redis (a fast in-memory data store) |
| Uploaded files (images, videos) | Object storage (S3, GCS) |
| Info about the current request | Carried inside the request itself (a JWT or cookie) |
| App settings | Environment variables / a config service |
Once sessions live in Redis instead of on a single server, any server can serve any user. A request that started on Server A can finish on Server B after a deploy or crash, and the user never notices.
Auto-scaling: adding servers on demand
Instead of adding servers by hand every time traffic spikes, modern systems auto-scale:
Auto-scaling
A system that watches numbers like CPU use, queue length, and response time, and automatically starts up new servers or shuts down extra ones to match how much traffic there is.
Add servers when: CPU is above 70% for 5 minutes → start 2 more servers Remove servers when: CPU is below 30% for 15 minutes → shut down 1 server
This is exactly why statelessness matters: the auto-scaler does not know which server a user was on before — that server may have just been shut down.
The 80% rule: set up enough capacity so that handling 100% of your load only uses about 80% of it. That spare 20% soaks up sudden traffic spikes while new servers boot (booting takes 2–5 minutes). The headroom also keeps responses fast — servers slow down when they are nearly maxed out.
Redundancy and availability
Horizontal scaling is what makes high availability possible. The key ideas:
High Availability (HA)
A system that keeps running even when some parts fail. It is measured as uptime: 99.9% ("three nines") means about 8.7 hours of downtime per year; 99.99% ("four nines") means about 52 minutes per year.
To get high availability you need:
- No single point of failure — every part has at least one backup ready to take over.
- Health checks — the load balancer keeps testing servers and stops sending traffic to any that have failed.
- Multiple availability zones — spread servers across different datacenters so one power outage does not take them all down.
Multiple data centers
Availability zones protect you inside one region. Once you have users in other countries, you go one level higher: run the whole system in several data centers in different parts of the world. This both speeds things up (users reach a nearby data center) and survives an entire region going dark.
geoDNS routing
A DNS service (the system that turns a domain name into an address) that hands back a different
address depending on where the user is. Normally traffic is split — say x% to US-East and
(100 − x)% to US-West — so each user reaches the closest data center. If one data center fails,
you point geoDNS at the healthy one and send 100% of traffic there.
Going multi-data-center is not free. Three problems have to be solved:
| Challenge | What it means | How it's solved |
|---|---|---|
| Traffic redirection | Send each user to the right data center | geoDNS routes by user location; send everything to the healthy data center during an outage |
| Data synchronization | A failover might send a user to a data center that does not have their data | Copy data to all data centers — e.g. Netflix's asynchronous multi-data-center replication |
| Test & deployment | Every data center must behave the same | Automated deployment that ships the same build to every data center and tests in many places |
Keeping data in sync is the hard part. With async replication (copying data with a small delay), there is a short window where a change saved in US-East has not reached US-West yet. If a failover happens during that window, the user can see old data or a "lost" save. This is the price of going multi-region — name it before the interviewer does.
Say: "I'd run multiple data centers with geoDNS sending users to the nearest one. The hard problem isn't routing traffic — geoDNS handles that — it's keeping data in sync across regions so a failover doesn't serve stale or missing data." That moves the conversation from plumbing to the real distributed-systems trade-off, which is the senior signal.
Scaling the database tier
Everything above scales the web tier (the servers). The database scales the same two ways — but the trade-offs are sharper, because the database is where the real data lives.
Scaling the database up (a bigger box) goes surprisingly far. AWS RDS offers machines with up to 24 TB of RAM — enough that StackOverflow served over 10 million monthly visitors in 2013 on a single main database. If your data fits on one big box, do it: there is no splitting logic, no cross-piece joins, no re-splitting later. But the ceiling is real:
- Hardware limits — there is a biggest machine money can buy, and a large user base eventually outgrows it.
- Single point of failure — one box holding all your data is one crash away from total outage.
- Cost — the most powerful machines cost far more than their extra power is worth (the same 4×-cost-for-2×-gain curve as the web tier).
When you outgrow the biggest box, the database scales out through sharding.
Sharding
Splitting one big database into smaller pieces called shards. Every shard has the same table
layout, but each one holds a different slice of the data. To find a row, you run a simple function
on the sharding key — e.g. user_id % 4 → shard 0–3 — and send the query to that shard.
Sharding key (partition key)
The column (or columns) that decides which shard a row lives on. This is the single most important choice in sharding: the key must spread data evenly, or some shards get slammed while others sit idle.
Sharding works, but it is the most complex move in this chapter, and it brings problems a single box never had:
| Problem | What goes wrong |
|---|---|
| Resharding | A shard fills up, or fills faster than the others, so you have to change the function and move data around (consistent hashing — Chapter 5 — tames this) |
| Celebrity / hotspot key | Put Katy Perry, Justin Bieber, and Lady Gaga on the same shard and the reads overwhelm it; very popular keys may each need their own shard |
| Joins & de-normalization | You can't easily JOIN across shards, so you copy related data into one table to keep each query on a single shard |
Sharding is exactly why "the database needs more memory → scale it up" (see the table below) is the default, not a contradiction. You scale the database up as far as the hardware allows precisely because sharding is painful to live with. You shard only when you have no other choice.
When asked to scale a database, lead with: "I'd scale up first — a single 24 TB box handles more than people expect — and only shard when I hit the ceiling, because sharding costs me joins and forces me to design around the celebrity-hotspot problem." Choosing the order of moves, and naming the costs, is what separates a senior answer from "just shard it."
When to use each approach
| Scenario | Right move |
|---|---|
| Database needs more memory | Scale up (memory does not shard easily) |
| Web/API servers under load | Scale out (stateless, easy to copy) |
| Need zero-downtime deploys | Scale out (update servers one at a time) |
| Low-traffic internal tool | Keep it a single box — simplicity wins |
The moment you choose horizontal scaling, a new question appears: which server should each request go to? That is Chapter 3.
Back-of-the-envelope estimation
Before choosing horizontal or vertical scaling — or anything else — you need rough numbers. A quick estimate tells you whether this is a 10-requests-per-second problem or a 100,000-requests-per-second problem. The answer completely changes the design.
A simple framework for estimating capacity:
- Daily Active Users (DAU) — how many different people use it each day?
- Request rate — DAU × requests per user ÷ 86,400 seconds = QPS (queries per second). Then multiply by 2–3 for peak times.
- Storage — size of one record × records per day × how long you keep them.
- Memory — if you cache: cache the top 20% most-used data (it usually covers about 80% of traffic).
Example — Twitter scale:
- 300M monthly users, 50% active daily = 150M DAU
- Each user posts 2 tweets/day → write QPS = 150M × 2 ÷ 86,400 = ~3,500 writes/sec
- 10% of tweets have media (avg 1 MB) → media storage = 150M × 2 × 10% × 1 MB = 30 TB/day
- Keeping 5 years: 30 TB × 365 × 5 = ~55 PB
Tips:
- Round hard:
99,987 / 9.1→100,000 / 10. Being exact is not the point. - Write down your assumptions. The interviewer can correct them, and you can point back to them later.
- Always label your units. "5" is unclear. "5 MB" is clear.
- Doing this without being asked is one of the strongest signals you can send in an interview.
When you talk about scaling in interviews, always bring up failure modes, not just throughput. An interviewer who hears "I'd use horizontal scaling" will ask "what happens if one instance crashes?" and "what happens during a deploy?" Having answers ready — health checks, rolling deploys, circuit breakers — moves you from someone who knows the words to someone who has thought through real production life.
How many servers do I need?
The Twitter example sized storage and write QPS. The other estimate interviewers love is fleet sizing: given that traffic and a per-server capacity, how many boxes does the web tier actually need? This is where the 80% rule and backups stop being abstract.
Work it in three moves — raw QPS, then headroom, then backups:
Step 1 — average load
1M DAU × 20 requests/day = 20M requests/day
20M / 86,400 s ≈ 230 req/s (average)
Step 2 — peak load
230 req/s × 3 (peak ≈ 3× average)
≈ 700 req/s (design target)
Step 3 — raw fleet from per-server capacity
one server handles ~1,000 req/s
700 / 1,000 ≈ 1 server of compute
Step 4 — apply the 80% rule (run at 80% capacity)
700 / (1,000 × 0.8) ≈ 0.9 → still 1 server's worth, but now ~90% full
Step 5 — add a backup (survive one box dying)
N + 1, with N = 1 → provision 2 servers
So the math says one server could carry the load — but you ship two. The second is not for throughput; it is so a single crash, deploy, or zone blip does not take you to zero. Always size the fleet for both "the load fits at 80%" and "we survive losing one," then take the bigger number.
| Quantity | Value | Where it came from |
|---|---|---|
| Average QPS | ~230/s | DAU × req/day ÷ 86,400 |
| Peak QPS (design target) | ~700/s | average × 3 |
| Servers for raw load | 1 | peak ÷ per-server capacity |
| Servers provisioned | 2 | N + 1 backup, headroom for spikes |
The same recipe scales up cleanly: at 50M DAU the peak is ~35,000 req/s, raw load is 35 servers, and after the 80% rule and N + 1 you would provision ~45. The numbers change; the three moves — load, headroom, backup — do not.
The junior answer divides QPS by server capacity and stops. The senior signal is provisioning more servers than the load strictly needs and saying why out loud: "One box covers the traffic, but I'd run two — the 80% rule keeps responses fast during spikes while auto-scaling boots, and N + 1 means a single failure or rolling deploy never drops me to zero capacity." Sizing for failure, not just throughput, is what they are listening for.
The scaling checklist
Scaling to millions of users is step by step — you apply these moves in roughly this order as load grows. Memorize the list; it is a ready-made backbone for "how would you scale this?"
- Keep the web tier stateless — push sessions and uploads to a shared store so any server can serve any request.
- Build backups at every tier — no single point of failure, from load balancer to database.
- Cache aggressively — cache the hot data (the top ~20% by usage) to take read load off the database.
- Support multiple data centers — geoDNS routing for speed and regional failover.
- Host static files in a CDN — serve JS/CSS/images from edge servers near the user, not from your web tier.
- Scale the database by sharding — once the biggest single box is not enough.
- Split into separate services — decouple parts (often via a message queue) so each scales on its own.
- Monitor and automate — central logging, host/tier/business metrics, and automated build-test-deploy.
These are not a strict sequence — you will revisit them. A message queue (item 7) decouples slow work like photo processing so the producers and consumers can scale on their own schedules. Monitoring (item 8) is optional on a few servers but essential once you are a real business.
Close a scaling discussion with the checklist as a backbone, not a recitation: "My order is stateless web tier, backups everywhere, cache the hot path, CDN for static files, multi-DC for geo and failover, shard the database last because it's the most expensive, then split into services and wrap it in monitoring and automation." Showing that you order the moves by cost and pain — cheap and reversible first, sharding last — is the senior signal.
Practice
Answer these to check that the two scaling paths and their trade-offs clicked.
1. What is vertical scaling?
2. Why must servers be stateless for horizontal scaling to work?
3. What does a load balancer do in horizontal scaling?
4. When scaling a database, which move should you usually try first?
5. Following the 80% rule and N + 1, if the math says one server can carry the load, how many do you provision?