📹 Live: System Design for a Web Crawler
In our Live Session today, we covered system design for a web crawler. We started off with a mock interview with a candidate. We did a feedback session and then did a long discussion on the fundamentals of crawlers. We designed a crawler and then addressed some more advanced concepts:
- Recrawling
- Page Freshness
- Fault Tolerance
- Social Politeness
- Redirects
For each concept, we went over how we can modify our design and add those features. We have attached the notes below:
