Let's //Get\ Technical: Blocking AI bots and crawlers for free to make a safer Internet
In this week’s episode, we go a bit technical in our segment “Let’s //Get\ Technical,” featuring our CTO, John Graham-Cumming. We discuss AI bot and crawlers, elections and more.
Host João Tomé and John Graham-Cumming begin by explaining how Cloudflare recently launched a new “easy button” to block all AI bots, scrapers, and crawlers to help maintain a safe Internet for content creators. It’s available to all customers, including those on our free tier. We demonstrate how to activate this button on the Cloudflare dashboard. We also explain the purpose of the robots.txt file—it resides at the root of your domain and contains directives for search engine crawlers on how to index pages on a website.
Next, we discuss how we recently replaced links to polyfill.io with Cloudflare’s mirror to ensure a safer Internet—polyfill.io is a popular JavaScript library service that can no longer be trusted and should be removed from websites.
We also analyze recent election trends in the UK and France, as well as the Biden vs. Trump US election debate. We were surprised to find that so many websites related to political parties were the targets of DDoS attacks blocked by Cloudflare around the election (or debate) days.
Also mentioned is Cloudflare’s 1.1.1.1 incident on June 27, 2024. The root cause was a combination of BGP (Border Gateway Protocol) hijacking and a route leak.
Mentioned topics:
- Declare your AIndependence: block AI bots, scrapers and crawlers with a single click
- Automatically replacing polyfill.io links with Cloudflare’s mirror for a safer Internet
- French elections: political cyber attacks and Internet traffic shifts
- UK election day 2024: traffic trends and attacks on political parties
- How the first 2024 US presidential debate influenced Internet traffic and security trends
- Exam-ining recent Internet shutdowns in Syria, Iraq, and Algeria
- Cloudflare 1.1.1.1 incident on June 27, 2024
