Design: Web Crawler

10-session plan from basic BFS crawling to distributed petabyte-scale web crawling. Design: Web Crawler: Requirements & Scope Functional Requirements Before designing anything, clarify what the system needs to do: Core features — What are the must-have features? User types — Who uses the system? End users, admins, API consumers? Scale — How many users? What's the read/write ratio? Non-Functional Requirements Availability: 99.9% (8.7 hours downtime/year) or 99.99%? Latency: < 100ms for reads? < 500ms for writes? Consistency: Strong or eventual? CAP trade-off? Durability: Can we lose data? How much? Clarifying Questions to Ask What's the expected DAU (Daily Active Users)? What's the read-to-write ratio? Do we need real-time updates? What's our budget constraint? What geography do we serve? Interview Tip "I always spend the first 5 minutes clarifying requirements. This shows structured thinking and prevents designing the wrong system. I explicitly separate functional from non-functional requirements." Java Implementation Python Implementation