How to Design a URL Crawler Frontier
Learn how to design a URL crawler frontier: per-host politeness, Bloom filter dedup, and priority scheduling explained.
Expected Interview Answer
A URL crawler frontier is the prioritized, politeness-aware queue that decides which URL a crawler fetches next, combining per-host rate limiting, priority scoring, and a seen-URL filter so crawling stays efficient without hammering any single site or refetching duplicates.
New URLs discovered from parsing fetched pages are checked against a seen-URL set (often a Bloom filter for memory efficiency at web scale) so already-crawled or already-queued URLs are dropped before entering the frontier. Surviving URLs are placed into per-host queues rather than one global queue, because politeness requires spacing out requests to any single domain regardless of how many URLs from that domain are pending overall. A scheduler selects the next host to crawl using a priority signal (page importance, freshness need, or round-robin fairness across hosts) and pulls the next URL from that host’s queue only if enough time has passed since the last request to it, enforcing the crawl-delay/robots.txt politeness policy. This two-level structure — global priority across hosts, FIFO or priority within a host — prevents both duplicate work and accidental denial-of-service on any one site.
- Per-host queues plus rate limiting enforce politeness and avoid overloading any single site
- A seen-URL filter (Bloom filter) prevents redundant refetching at web scale with low memory
- Priority scoring lets important or frequently changing pages get crawled sooner
- The two-level design (host selection, then URL within host) balances fairness and efficiency across millions of domains
AI Mentor Explanation
A URL crawler frontier is like a talent scout’s master list of grounds to visit, organized so no single ground gets three scouts showing up in the same afternoon. New grounds discovered through word of mouth are checked against a list of already-visited grounds so the scout does not waste a trip revisiting one. Grounds are grouped by region, and the scout enforces a minimum gap between visits to any one region so local organizers are not overwhelmed with scouts at once. Higher-priority grounds — those hosting a big upcoming match — get visited sooner than lower-priority ones waiting their turn.
Step-by-Step Explanation
Step 1
Discover and dedupe
New URLs parsed from fetched pages are checked against a seen-URL set (Bloom filter) and dropped if already crawled or queued.
Step 2
Enqueue per host
Surviving URLs are placed into a per-host queue rather than one global queue to enable politeness control.
Step 3
Select next host by priority
A scheduler picks the next host to crawl based on priority and fairness, respecting each host’s minimum crawl-delay.
Step 4
Fetch and re-enqueue discoveries
The URL is fetched, new links are extracted, and the cycle repeats with newly discovered URLs re-entering the dedupe step.
What Interviewer Expects
- Explains per-host queues and why politeness (crawl-delay/robots.txt) requires them
- Names a seen-URL filter (Bloom filter) and why it matters at web scale for memory efficiency
- Describes priority scoring for important/frequently changing pages
- Addresses the two-level scheduling: which host next, then which URL within that host
Common Mistakes
- Using a single global FIFO queue with no per-host politeness control
- Storing all seen URLs in an exact set without considering memory cost at billions of URLs
- Ignoring robots.txt and crawl-delay, risking overloading target sites
- Not prioritizing important or fast-changing pages over low-value ones
Best Answer (HR Friendly)
“A URL crawler frontier is the smart to-do list that tells a web crawler which page to fetch next. It keeps the crawler polite by spacing out requests to any one website, avoids wasting effort refetching pages it has already seen, and makes sure important pages get crawled sooner than low-priority ones.”
Code Example
def get_next_url(frontier, min_delay_seconds=1.0):
now = time.time()
for host in frontier.hosts_by_priority():
last_fetch = frontier.last_fetch_time(host)
if last_fetch and now - last_fetch < min_delay_seconds:
continue # too soon, respect crawl-delay for this host
url = frontier.pop_from_host_queue(host)
if url is None:
continue
frontier.mark_fetch_time(host, now)
return url
return None # nothing ready to fetch right now
def enqueue_discovered(frontier, url, seen_filter):
if seen_filter.might_contain(url):
return # likely already seen, skip
seen_filter.add(url)
frontier.push_to_host_queue(host_of(url), url)Follow-up Questions
- Why use a Bloom filter for the seen-URL set instead of an exact hash set, and what is the trade-off?
- How would you prioritize URLs so important pages (like a homepage) get crawled more often than deep pages?
- How do you avoid crawler traps like infinite calendar pages generating endless unique URLs?
- How would you shard the frontier across multiple crawler machines while keeping per-host politeness?
MCQ Practice
1. Why does a URL crawler frontier use per-host queues instead of one global queue?
Per-host queues let the scheduler enforce a minimum delay between requests to any single site, regardless of overall queue size.
2. Why is a Bloom filter commonly used for the seen-URL set in large-scale crawlers?
Bloom filters trade a small false-positive rate for dramatically lower memory usage than an exact set at web scale.
3. What does “politeness” mean in the context of a web crawler frontier?
Politeness means pacing requests to any one host to avoid overwhelming it, typically per robots.txt crawl-delay directives.
Flash Cards
Why per-host queues in a crawler frontier? — To enforce politeness by rate-limiting requests to each host independently of overall queue size.
Why use a Bloom filter for seen URLs? — Compact, memory-efficient approximate membership testing at web scale, trading a small false-positive rate.
What decides crawl priority? — Signals like page importance, freshness needs, and fairness across hosts.
What is a crawler trap? — A site structure (like infinite calendar pages) generating endless unique URLs that can waste crawler resources.