HardDesign: Web Crawler

What is the 'politeness' policy in web crawling?

Tests your understanding of this concept.

Answer Options

AUsing polite language in crawler User-Agent
BLimiting crawl rate per domain to avoid overloading servers
CAsking website owners for permission before crawling
DOnly crawling during off-peak hours

Want to see the correct answer?

Get the answer with a detailed explanation, plus practice 22+ more Design: Web Crawler questions with adaptive quizzes and timed interviews.

See the Answer on Guru Sishya →

This question is from the Design: Web Crawler topic (System Design Cases).

More Design: Web Crawler Questions