arrow-left arrow-right brightness-2 chevron-left chevron-right facebook-box facebook loader magnify menu-down rss-box star twitter-box twitter white-balance-sunny window-close
📹 Live: System Design for a Web Crawler
1 min read

📹 Live: System Design for a Web Crawler

In our Live Session today, we covered system design for a web crawler. We started off with a mock interview with a candidate. We did a feedback session and then did a long discussion on the fundamentals of crawlers. We designed a crawler and then addressed some more advanced concepts:

  1. Recrawling
  2. Page Freshness
  3. Fault Tolerance
  4. Social Politeness
  5. Redirects

For each concept, we went over how we can modify our design and add those features. We have attached the notes below: