Cloudflare TV

Let's //Get\ Technical: Blocking AI bots and crawlers for free to make a safer Internet

Presented by: John Graham-Cumming, João Tomé
Originally aired on November 1 @ 2:00 AM - 2:30 AM EDT

In this week’s episode, we go a bit technical in our segment “Let’s //Get\ Technical,” featuring our CTO, John Graham-Cumming. We discuss AI bot and crawlers, elections and more.

Host João Tomé and John Graham-Cumming begin by explaining how Cloudflare recently launched a new “easy button” to block all AI bots, scrapers, and crawlers to help maintain a safe Internet for content creators. It’s available to all customers, including those on our free tier. We demonstrate how to activate this button on the Cloudflare dashboard. We also explain the purpose of the robots.txt file—it resides at the root of your domain and contains directives for search engine crawlers on how to index pages on a website.

Next, we discuss how we recently replaced links to polyfill.io with Cloudflare’s mirror to ensure a safer Internet—polyfill.io is a popular JavaScript library service that can no longer be trusted and should be removed from websites.

We also analyze recent election trends in the UK and France, as well as the Biden vs. Trump US election debate. We were surprised to find that so many websites related to political parties were the targets of DDoS attacks blocked by Cloudflare around the election (or debate) days.

Also mentioned is Cloudflare’s 1.1.1.1 incident on June 27, 2024. The root cause was a combination of BGP (Border Gateway Protocol) hijacking and a route leak.

Mentioned topics:

English
News

Transcript (Beta)

Thumbnail image for video "This Week in NET"
This Week in NET
Tune in for weekly updates on the latest news at Cloudflare and across the Internet. Check back regularly for updates. Also available as an audio podcast!
Watch more episodes 
background