HardDesign: Web Crawler

How should a crawler handle a 404 Not Found response?

Tests your understanding of this concept.

Answer Options

AImmediately retry the URL
BRemove from queue, never retry
CMark as failed, retry with exponential backoff, eventually abandon
DCrawl parent directory instead

Want to see the correct answer?

Get the answer with a detailed explanation, plus practice 22+ more Design: Web Crawler questions with adaptive quizzes and timed interviews.

See the Answer on Guru Sishya →

This question is from the Design: Web Crawler topic (System Design Cases).

More Design: Web Crawler Questions