Design a Web Crawler (Googlebot) — System Design

🟠 High-Level Design · Senior → 🔴 the frontier & politeness deep dive

"Design a web crawler" (like Googlebot) means building a system that starts from a few URLs and systematically downloads a huge chunk of the web — billions of pages — to feed a search index. It sounds like a simple loop ("fetch page, find links, repeat"), but at web scale it hits fascinating problems: not re-crawling the same page forever, not hammering any single website, and avoiding infinite traps. Let's design it.

Step 1 — Requirements

• Functional: start from seed URLs; download pages; extract and follow links; store page content for indexing; re-crawl periodically for freshness.
• Non-functional: massive scale (billions of pages), polite (don't overload any site, obey robots.txt), robust (handle bad HTML, timeouts, traps), and efficient (don't waste effort on duplicates).

Step 2 — It's a graph traversal

🟠 The web is a giant graph: pages are nodes, links are edges. Crawling is a BFS (breadth-first traversal) starting from seeds. That framing instantly tells you what you need: a queue of nodes to visit, and a set of nodes already visited — the two data structures at the crawler's core.

Step 3 — The components

• URL Frontier: the queue of URLs waiting to be crawled — the crawler's heart (more below).
• Fetchers: many workers that pull URLs and download the pages over HTTP.
• Parser: extracts links (and content) from the downloaded HTML.
• Dedup / "seen" set: checks whether a URL (or its content) has already been crawled, so we don't loop forever.
• Content store: saves page content (to object storage) for the indexing pipeline.

Step 4 — Deduplication (don't crawl the same thing twice)

🔴 The web is full of links pointing back to pages you've seen. Checking "have I crawled this URL?" against a set of billions of URLs is a memory problem. The classic answer: a Bloom filter — it tells you "definitely not seen" in tiny memory, so you skip the expensive real check for the vast majority. You also dedupe content (many URLs serve identical pages) by hashing the page and skipping duplicates. This is exactly why bloom filters exist.

Step 5 — Politeness (the part people forget)

🔴 A naive crawler with thousands of fetchers would hammer a single small website with a flood of requests — effectively a DoS attack — and get you banned. A good crawler is polite:

• Obey robots.txt: each site publishes rules for what crawlers may access and how often. Respect them.
• Rate-limit per domain: never send many concurrent requests to the same host; add a delay between hits to a domain.

🔴 This shapes the frontier design: it's not one simple queue. A common approach is a two-level frontier — a front set of queues for priority (important pages crawled more often) and a back set of queues per domain, so a worker takes from a domain's queue only when that domain's politeness delay has elapsed. Naming "per-host queues for politeness + priority queues for freshness" is the senior insight this question is really probing.

Step 6 — Avoiding traps & robustness

🟠 The web is hostile: infinite calendars, session-id URLs that never repeat, cyclic links, and enormous pages. Defences: cap crawl depth and pages per domain, set download size/time limits, normalise URLs (strip fragments, sort params) so trivially-different URLs dedupe, and detect spider traps. Fetchers also need timeouts and retries because much of the web is slow or broken.

Step 7 — Scale & freshness

The whole thing is distributed: thousands of fetchers across many machines, the frontier and seen-set sharded (often by domain hash, which also naturally groups a domain's URLs for politeness). For freshness, you re-crawl pages on a schedule based on how often they change — news sites hourly, static pages rarely — so the index doesn't go stale without wasting effort re-fetching things that never change.

The interview-ready walkthrough

"A crawler is a distributed BFS over the web graph. The core is the URL frontier (queue of URLs); fetchers download pages, a parser extracts links, and I dedupe with a Bloom filter for seen URLs plus content hashing. The subtle part is politeness — obey robots.txt and rate-limit per domain — so I'd use a frontier with per-host queues for politeness and priority queues for freshness. I'd guard against spider traps with depth/size limits and URL normalisation, store content in object storage, and re-crawl on a change-frequency schedule. It's all sharded by domain for scale." That covers the structure, the dedup trick, and the politeness insight — the whole thing."

What to read next

• Design Typeahead Autocomplete — the search suggestions built on this crawl data
• Bloom Filters — the dedup structure at the crawler's heart
• Message Queues & Kafka — the frontier as a distributed queue
• ← The complete System Design guide (hub)

← Design typeahead autocomplete · Design a distributed cache →