A web crawler — also called a robot or a spider — is the program that search engines use to find new and changed pages on the web. The basic idea sounds simple: download a page, pull out its links, then do the same for each of those links, again and again. That simplicity is exactly why interviewers love it. Run that little loop over billions of pages and every hard distributed-systems problem shows up inside it: being polite to websites, not visiting the same page twice, keeping pages fresh, and avoiding traps.
The animation on the right walks through the core crawl loop one step at a time: seed URLs (the starting pages) go into the URL Frontier (a waiting line of pages to download), the HTML Downloader fetches a page, the Content Parser pulls out the links, the "URL Seen?" check throws away links we already visited, and the new ones go back into the frontier — closing the loop and spreading out one level at a time.
Step 1: Ask questions and estimate the scale
A crawler can be a tiny weekend script or a huge system run by many teams. So before you design anything, ask what you are building. Here are the questions to ask, and the answers we will design for:
| Question | Our answer |
|---|---|
| What is it for? | Indexing pages for a search engine |
| How many pages per month? | 1 billion |
| What kind of content? | HTML only (we can add more later) |
| Re-crawl pages that changed? | Yes — keep the data fresh |
| Save the page HTML? | Yes, for up to 5 years |
| Worry about duplicate content? | No — just skip it |
Beyond the features, name the four qualities of a good crawler. Saying these out loud is itself a strong signal:
The four qualities of a good crawler
Scalability — the web has billions of pages, so do many at once. Robustness — the web is full of traps: broken HTML, dead servers, bad links, so don't crash. Politeness — don't hammer one website with too many requests. Extensibility — you can add new content types later without rewriting everything.
Doing the math (back-of-the-envelope)
Pages/month = 1,000,000,000
Downloads/sec = 1B / 30 days / 24h / 3600s ≈ 400 pages/sec
Peak downloads/sec = 2 × 400 = 800 pages/sec
Average page size = 500 KB
Storage/month = 1B × 500 KB = 500 TB / month
Storage over 5 yrs = 500 TB × 12 × 5 = 30 PB
These numbers drive every later choice. 400 pages per second means we need many downloaders running in parallel. 30 petabytes means the pages live on disk (with the popular ones kept in memory), not all in RAM.
Do this math without being asked. Going from "1 billion pages a month" to "about 400 pages per second and 30 PB over five years" shows the interviewer that you think about scale before you draw any boxes. It is the strongest early signal in this problem.
Step 2: The high-level design
Picture the web as a graph (a network of dots connected by lines): each page is a dot, and each link is a line pointing to another page. Crawling is just walking that graph. Here are the parts that do the walking:
| Part | What it does |
|---|---|
| Seed URLs | The pages we start from; picked to reach as much of the web as possible |
| URL Frontier | A waiting line (FIFO queue) of pages still to download — the heart of the system |
| HTML Downloader | Fetches the pages that the frontier hands it |
| DNS Resolver | Turns a website name into an IP address (e.g. wikipedia.org → 198.35.26.96) |
| Content Parser | Reads and checks the HTML; runs on its own so one bad page can't jam the downloaders |
| Content Seen? | Compares pages to skip duplicates (about 29% of the web is duplicate content) |
| Content Storage | Where pages are saved — mostly on disk, popular pages in memory |
| Link Extractor | Pulls links out of the HTML; turns short relative paths into full URLs |
| URL Filter | Removes blocked sites, bad file types, and dead links |
| URL Seen? | Remembers URLs we already visited or queued — stops repeats and endless loops |
How the loop runs
The crawl is a loop, and the steps in the animation match it one-to-one:
- Put the seed URLs into the URL Frontier.
- The HTML Downloader takes the next URL out of the frontier.
- The downloader looks up the host with the DNS Resolver, then downloads the page.
- The Content Parser reads and checks the HTML.
- "Content Seen?" checks if we already have this page. Already saved → throw it away. New → keep going.
- New pages go to Content Storage; the links go to the Link Extractor.
- The links pass through the URL Filter.
- The survivors hit "URL Seen?" — already seen → drop it; new → put it back into the URL Frontier.
Lead with BFS, not DFS. DFS (depth-first) can dive very deep into one website and get stuck, so crawlers walk the web one level at a time — BFS (breadth-first) — using a simple waiting line (FIFO queue). That waiting line is exactly what the URL Frontier is. Saying this before you draw the queue shows you know why it has that shape.
Step 3: Deep dive — the URL Frontier
A plain waiting line does BFS, but it has two problems. The URL Frontier exists to fix both.
Politeness
Politeness
A crawler must not flood one website. Thousands of requests per second to a single site looks like an attack that tries to knock the site offline (a denial-of-service attack). The rule: download one page at a time per website, and wait a little between requests.
The frontier does this with back queues. A router and a lookup table make sure each back queue (b1…bn) holds URLs from one website only, and a selector ties each worker thread to one queue. So a given thread only ever talks to one website, one page at a time.
Priority
Not every page matters the same. The Apple home page is more important than a random forum post that happens to mention "Apple." A Prioritizer scores each URL by how useful it is (using things like PageRank, traffic, and how often it changes) and sends it into front queues (f1…fn), one per priority level. The selector picks from the high-priority queues more often.
So the full frontier has two stages: front queues handle priority, back queues handle politeness.
Freshness and storage
Pages change, so the crawler comes back and re-crawls them now and then. But re-crawling everything is wasteful. Instead, re-crawl based on how often a page has changed before, and re-crawl important pages first. For storage, the frontier can hold hundreds of millions of URLs, so it uses a mix: most of them sit on disk, with small in-memory buffers for adding and removing URLs that flush to disk every so often.
Most candidates stop at "it's a queue." Saying "front queues for priority, back queues for politeness, and a mix of disk plus memory for storage" is the answer that stands out — it shows you know the frontier does three jobs, not one.
Step 4: Deep dive — downloader, dedupe, and robustness
Making the HTML Downloader fast
- Robots.txt: before crawling a site, download its
robots.txtfile and follow its rules (this file tells crawlers what they may and may not visit). Cache the file so you don't re-download it for every page. - Spread the work: split the URLs across many servers, each running many threads.
- DNS cache: looking up a name can be slow (10–200 ms) and often blocks the thread. Keep your own name→IP cache, refreshed on a schedule, so threads don't wait.
- Be close: put crawl servers near the sites they crawl.
- Short timeout: cap how long you wait on a host; if it doesn't answer, move on.
Spotting duplicates and traps
| Problem | How we handle it |
|---|---|
| Repeated content (~29% of the web) | Compare page hashes (short fingerprints), not the whole page |
| Duplicate URLs / endless loops | The "URL Seen?" check (Bloom filter + hash table) |
Spider traps (/foo/bar/foo/bar/…) | Cap the URL length; flag sites with a strange number of pages |
| Junk (ads, spam) | Filter out low-value content |
Bloom filter (URL Seen?)
A small, memory-cheap way to answer "have we seen this URL before?" almost instantly. It can sometimes wrongly say yes, but it will never wrongly say no — so a duplicate URL is never crawled twice. This is what stops the frontier from looping forever.
Staying robust and easy to extend
- Consistent hashing spreads the load across downloaders and lets you add or remove servers without reshuffling everything.
- Save the crawl's progress to storage, so if it stops it can pick up where it left off instead of starting over.
- Handle errors gracefully and check the data, so one bad page can't crash the whole system.
- Easy to extend: the parser and downloader are plug-in pieces — you can add a PNG Downloader or a Web Monitor without rebuilding the core loop.
The dedupe story is your chance to connect chapters: "URL Seen?" is a Bloom filter, and the downloader load is spread with consistent hashing (Chapter 5). Reusing earlier building blocks instead of inventing new ones reads as senior — you're putting a system together, not improvising parts.
Wrap-up
Work through this and you've covered the whole arc: ask about scope, size it (400 pages/sec, 30 PB), draw the BFS loop, then go deep on the frontier (politeness + priority), the downloader (DNS cache, robots.txt), and dedupe (Bloom filter, page hashes, traps).
Extra points to mention if you have time — naming them unprompted shows breadth:
- Server-side rendering for links created by JavaScript (download and run the page before parsing).
- Anti-spam filtering to skip low-quality pages when storage is limited.
- Replication and sharding of the data layer for availability and scale.
- Horizontal scaling to hundreds of stateless download servers.
A web crawler in 45 minutes: 5 min requirements + estimation → 10 min the BFS loop and components → 20 min deep dives (frontier politeness/priority, downloader, dedupe + traps) → 10 min robustness and the "what we left out" list. The interviewer grades your process — narrate the loop, then back up each box with a number or a failure mode.
Practice
Answer these to check you understood the crawl loop.
1. What is the URL Frontier?
2. Why do crawlers use BFS (breadth-first) instead of DFS (depth-first)?
3. What does politeness mean for a crawler?
4. What is the job of the 'URL Seen?' Bloom filter?
5. Inside the frontier, what do the front queues and back queues each handle?