The Future of Content and AI: Pay per Crawl and What’s Next

In this episode, host João Tomé is joined by Will Allen, Cloudflare’s VP of Product Management, to discuss Pay-Per-Crawl and our new permission-based model for AI bots. These updates, launched on July 1, 2025 — what we call Content Independence Day — aim to reshape how AI models access and reward content, shifting from an opt-out to an opt-in approach. Cloudflare’s CEO, Matthew Prince, also appears in a video at the start, reflecting on Content Independence Day.

We explore how AI Overviews are changing the old “traffic for content” model and how Cloudflare is helping creators take control through tools like AI Audit. Plus: the future of trustworthy content, bot authentication, and the rise of a fairer content economy.

Send us your questions for a future episode on these topics at ThisWeekinNET@cloudflare.com.

Check the mentioned blog posts here: https://blog.cloudflare.com/tag/pay-per-crawl/