Can You Scrape a Protected Site？ Testing Cloudflare’s New ⧸crawl API

We made scraping easy for you, it's just one endpoint that you have to use to scrape any website on the Internet. Wait, wait, wait, but you might be thinking that, isn't Cloudflare helping me protect my website against scraping? And you are not wrong. The new crawl endpoint respects the robots.txt file of your website, so if you have a rule disallowing bots to scrape the content of your website, we will respect that. Moreover, if you have configured any bot management rules or have configured WAF or even turnstile, we gonna respect that. Which means that if your site is behind Cloudflare, has a robots.txt file which disallows a crawling or if you are using WAF, bot management or turnstile to block bots, the crawl endpoint is not gonna scrape your site. Let me show you with a quick demo. I have created a webpage crawlme. Now this webpage has a robots.txt file which disallows any kind of bots. Let's see in action. Now I have also added an interactive playground which allows you to use the crawl endpoint to crawl the website. Now you need your Cloudflare account id and your API token to use this. I have already configured my account id and the API token and for now, I am gonna just crawl my personal website. So if I hit run analysis, it has started the job as we can see over here and once the content is ready, it will show us the report. And as you can see, my personal website is partially blocked because I want the content of my website to be accessible to bots out there. Not all the pages but most of the pages. And as we can see, it was able to crawl a ton of pages on my website. Let's try the same result with the crawlme.net website. I have refreshed the page and updated the URL in here and now let's run the analysis again. As we can see, it has started a new job and once it is done trying to crawl this site, we are gonna see the result as well. And as you can see, it is not able to crawl crawlme.net because we have the robots.txt file which has the disallow rule in there. If you are interested in seeing the exact rule that I have, this is what I have in my robots.txt file. Again, because we are blocking all sorts of bots, the crawl endpoint is gonna respect that and is not gonna scrape this website. So, the company you trust for protecting your site from being scraped is making sure that it does that job for you. Even though we have a crawl endpoint that you can use to scrape websites, if your website does not wanna get scraped and have any sorts of rules configured to prevent scraping, the crawl endpoint is gonna respect that. Well, this was a quick demo on crawlme.net and the crawl endpoint. I am working on some more fun demos on using the crawl endpoint. But if you are also building something with the crawl endpoint, do let me know. I would love to see what you are building. See you in the next one.