How to design a web crawler

Requirements

Ask what the crawl is for. A search engine index, an archive and a price monitor need different freshness and coverage. Assume a general-purpose crawler feeding a search index.

• Functional: start from seed URLs; fetch pages over HTTP; extract links and add new ones to the crawl; store the raw pages for the indexer.
• Functional: obey robots.txt; recrawl pages so the stored copy stays reasonably fresh.
• Non-functional: politeness - never send more than about one request at a time to a single host, and back off when a site slows down.
• Non-functional: scale to billions of pages by adding workers; survive worker crashes without losing or repeating large parts of the crawl.
• Out of scope: rendering JavaScript-heavy pages, ranking and the search index itself.

Capacity estimates

These are assumptions, stated out loud. The point is the order of magnitude, which decides the architecture.

Three conclusions. The fetch rate is modest; the hard limit is politeness, not raw throughput. Pages go to object storage, not a database - 100 TB a month is a storage bill, not a query problem. And the URL-seen check runs 50 times per page, so it must be an in-memory lookup, not a database round trip.

API design

A crawler has no public API. The interfaces that matter are internal: the frontier hands out URLs, and fetcher workers report back what they found.

frontier.next(worker_id) -> { url, host, priority, lease_id }
frontier.add(urls[], source_url) -> { accepted, skipped_seen }
frontier.complete(lease_id, status, next_crawl_at)

fetcher.fetch(url) -> {
  status: 200,
  final_url: "https://example.com/a",
  content_hash: "sha256:9f2c...",
  simhash: "0x8a3f1c...",
  s3_key: "pages/2026/10/06/9f2c....html.gz",
  links: [ ... ]
}

The lease matters: a worker that takes a URL and then crashes never calls complete, the lease expires, and the URL goes back into the frontier. This is the same visibility-timeout idea SQS uses.

Data model

Two stores. Page bodies are large and written once, so they go to S3. Per-URL metadata is small and looked up by key, so it fits a key-value store such as DynamoDB.

url_state            key: url_hash
  url                text
  host               text
  last_crawled_at    timestamp
  next_crawl_at      timestamp
  last_status        int
  content_hash       text       exact-duplicate check
  simhash            bigint     near-duplicate check
  change_count       int        how often the page actually changed
  s3_key             text       latest stored copy

host_state           key: host
  robots_txt         text
  robots_fetched_at  timestamp
  crawl_delay_ms     int
  ip                 text       cached DNS answer
  next_allowed_at    timestamp

Key everything by a hash of the normalized URL. Normalize first - lowercase the host, drop default ports, fragments and tracking parameters - or the same page gets crawled under ten different URLs.

High-level design

The crawl is a loop through a few stages, each scaled on its own.

• The URL frontier decides what to fetch next. It holds priority queues (important or frequently changing pages first) feeding per-host queues that enforce politeness.
• Fetcher workers on EC2 take URLs from the frontier, check robots.txt and the DNS cache, download the page, and write the body to S3.
• A parser stage extracts links and computes content fingerprints. It can run in the same worker or behind an SQS queue so slow parsing doesn't hold up fetching.
• The dedup stage drops links already seen (Bloom filter, then url_state) and pages whose content is a duplicate. New URLs go back into the frontier.
• A recrawl scheduler sets next_crawl_at for every fetched URL and re-adds URLs to the frontier when they come due.

Where it breaks

Put every discovered URL on one big FIFO queue and let workers pull from it. Pages link mostly to their own site, so the queue fills with long runs of URLs from one host. A hundred workers then hit the same server at once - you are effectively attacking it, it starts returning 429s or blocks your IP range, and meanwhile the rest of the web waits.

The fix is the two-level frontier. Front queues sort URLs by priority. Back queues each hold URLs for a single host, and a heap keyed on next_allowed_at decides which host is ready. A worker only gets a URL from a host whose delay has passed. Throughput now comes from crawling many hosts in parallel, not one host fast. The new risk is skew: a few huge sites have millions of URLs queued, so give them their own back queues and let small hosts share.

URL frontier and politeness

• Front queues by priority: score URLs on things like link popularity, domain importance and how often the page changes, then pull from higher queues more often.
• Back queues by host: one host maps to one queue, and each queue has a next_allowed_at time. Respect Crawl-delay from robots.txt and stretch the delay when a host's responses slow down.
• Cache robots.txt per host for about a day, and cache DNS answers too. A DNS lookup can take longer than the fetch itself, and resolving the same host for every URL wastes both.
• Keep the frontier mostly on disk or in a durable store, with only the heads of queues in memory. Billions of pending URLs do not fit in RAM.

Duplicate URLs and duplicate content

• URL dedup: check each normalized URL against a Bloom filter first. A "no" is always right, so most new links are confirmed new in memory. A "maybe" goes to url_state for the exact answer.
• Exact content dedup: hash the page body. Mirrors and copies of the same page share a hash, so store and index them once.
• Near-duplicate dedup: pages that differ only by a timestamp or ad block have different hashes. Simhash maps similar text to fingerprints that differ in only a few bits, so near-duplicates can be found by comparing bits.

Recrawl scheduling and crawler traps

Pages change at very different rates. A news homepage changes every few minutes; an old blog post may never change. Track change_count over past crawls and adapt: recrawl sooner when the content hash changed, later when it didn't. Traps are the opposite problem - sites that generate endless URLs, such as calendars with a "next month" link forever, or session IDs in every link. Cap URL length and path depth, cap pages per host per crawl cycle, and flag hosts whose new URLs keep producing near-duplicate content.

What interviewers look for

• Politeness designed in from the start, with per-host queues - not added as a rate limit afterwards.
• A clear dedup story at both levels: Bloom filter for URLs, hashes and simhash for content, with the false-positive trade-off explained.
• Separating page storage (S3) from URL metadata (a key-value store), with numbers to back it.
• Handling failure and the endless web: leases for crashed workers, adaptive recrawls, and concrete limits against traps.