HardDesign: Web Crawler

What User-Agent header should a well-behaved crawler send?

Tests your understanding of this concept.

Answer Options

AHide crawler identity to avoid being blocked
BA descriptive User-Agent with crawler name, URL to info page, and contact email
CMimic a browser User-Agent (Mozilla/5.0...)
DSend no User-Agent header

Want to see the correct answer?

Get the answer with a detailed explanation, plus practice 22+ more Design: Web Crawler questions with adaptive quizzes and timed interviews.

See the Answer on Guru Sishya →

This question is from the Design: Web Crawler topic (System Design Cases).

More Design: Web Crawler Questions