Hard Design: Web Crawler

What does robots.txt's 'Crawl-delay: 10' directive mean?

— Tests your understanding of this concept.

Answer Options

A Maximum 10 concurrent connections to this domain
B Wait 10 seconds between consecutive requests to this domain
C Only crawl for 10 minutes per day
D Crawl only 10 pages from this domain

Want to see the correct answer?

Get the answer with a detailed explanation, plus practice 22+ more Design: Web Crawler questions with adaptive quizzes and timed interviews.

See the Answer on Guru Sishya →

This question is from the Design: Web Crawler topic (System Design Cases).

Learn Design: Web Crawler → Browse All Questions

More Design: Web Crawler Questions

Why is BFS (Breadth-First Search) preferred over DFS for web crawling?

Hard

What is the purpose of a Bloom filter in a web crawler?

Hard

A website generates infinite unique URLs like /products?page=1, /products?page=2 ... /products?page=1000000. How do you handle this?

Hard

How do you crawl JavaScript-rendered Single Page Applications (SPAs)?

Hard

A fetcher node crashes after popping a URL from the queue but before completing the crawl. How do you prevent URL loss?

Hard