![robots.txt doesn't block index.html even if with Disallow rule is present in it. - Google Search Central Community robots.txt doesn't block index.html even if with Disallow rule is present in it. - Google Search Central Community](https://storage.googleapis.com/support-forums-api/attachment/message-31652070-11771310525571336832.png)
robots.txt doesn't block index.html even if with Disallow rule is present in it. - Google Search Central Community
Data Science Pipeline — Part 1: Obtaining data from web using Scrapy | by Sagun Shrestha | Towards Data Science
![Python Scrapy Tutorial - 3 - Robots.txt and Web Scraping Rules Video Lecture | Study Python Web Scraping Tutorial - Back-End Programming | Best Video for Back-End Programming Python Scrapy Tutorial - 3 - Robots.txt and Web Scraping Rules Video Lecture | Study Python Web Scraping Tutorial - Back-End Programming | Best Video for Back-End Programming](https://img.youtube.com/vi/w_jUkFFaobQ/0.jpg)
Python Scrapy Tutorial - 3 - Robots.txt and Web Scraping Rules Video Lecture | Study Python Web Scraping Tutorial - Back-End Programming | Best Video for Back-End Programming
![Scraping reddit with Scrapy.. New to scrapy? Build a spider that… | by Brendan Ferris | Analytics Vidhya | Medium Scraping reddit with Scrapy.. New to scrapy? Build a spider that… | by Brendan Ferris | Analytics Vidhya | Medium](https://miro.medium.com/max/1400/1*4gfFzcrPVo7xwj1o50xmXw.png)