The Future of Content and AI: Pay per Crawl and What’s Next

I have this, you know, this idea and I would like to think that we can sort of make it a reality is imagine firing up your favorite deep research, you know, product from any one of these amazing sort of foundation model or sort of AI driven companies. And you say, I want to get I want to educate me about this particular topic. Maybe it's a health topic. Maybe it's a legal brief analysis. Maybe it's about gardening, whatever the whatever the concept is. And you say, here's my budget, you know, here is $10, $20, $200, whatever the amount is, here's some budget to go out there. And that deep research agent can go out and find you the best, most interesting, highly relevant, curated content for you and your particular need. Like I just I keep going back to this example when I think about it, because that for me as a consumer, that'd be amazing. Hello, everyone, and welcome to This Week in Net. It's the July the 10th, 2025 edition. We didn't have an episode last week. July the 4th was being celebrated in the U.S. And we wanted this episode to focus entirely on content independence. I'm your host, João Tomei, coming to you from our Klauffler's Lisbon office here in Portugal, recording from our cozy library. Joining me today is Will Allen, VP of Product Management. Klauffler recently announced Paper Crawl and a new permission -based model for AI bots. These changes could fundamentally reshape how content creators are compensated and how AI models are trained. It really is content independence for many. Today, we'll talk about what these changes mean for publishers, AI companies, and the Internet at large. So here's my take about this topic. This is a topic close to my heart. Content creators producing good and meaningful work is essential to the kind of world we want to leave to our kids. I was a journalist for several years in Europe, and it's both exciting and worrying to see how technology is impacting that. Getting information today has never been easier, thanks to the Internet. Amazing tools came with it. Search engines, Wikipedia, but also social media and smartphones brought another side to the coin. Flood of information, constant urgency, clickbait, flashy titles instead of nuance and quality. With AI, from LLMs to chatbots, and now agents, we might have a real chance to change that. These tools can help make trustworthy content more accessible and useful. The incentive isn't just to grab attention anymore, it's to provide real value so users actually pay for a subscription. Generative AI can prioritize accuracy over engagement, for example, which is a big shift from how social media works. That's why starting a real conversation between content creators and AI companies, including Google, which is now heavily pushing AI overviews in search, is essential. Clouder is helping lead that conversation, sitting right at the crossroads between content creators and AI platforms. And let's be clear, AI companies need high-quality content to improve their models and meet user expectations. Creating a fair marketplace where content is respected and rewarded is a step in the right direction. That's what I think, and Clouder as well. It's early days and this will evolve, but the direction matters. Before we go to my conversation with Will Allo, one of the leaders behind Pay-Per-Call and the tools being built for this new model, a quick note. We're thinking of doing a live Q&A episode about this topic. If you have questions, send them to thisweekinnet at clouder.com. There's a lot to discuss. And before my conversation with Will, here's our CEO and co-founder, Matthew Prince, announcing Content Independence Day, a moment that came with six blog posts on July the 1st, 2025. And this is a video I recorded with Matthew right here in our Lisbon library. Almost 30 years ago, two graduate students at Stanford University, Larry Page and Sergey Brin, began working on a research project they called Backrub. That, of course, was the project that resulted in Google. But also something more. It created the business model for the web. The deal that Google made with content creators was simple. Let us copy your content for search and we'll send you traffic. You, as a content creator, could then derive value from that traffic in one of three ways, running ads against it, or making a video about it. The idea was simple. Let's Google facilitated all of this. Search generated traffic, they acquired DoubleClick and built AdSense to help content creators serve ads, and acquired Urchin to launch Google Analytics and let you measure just who is viewing your content at any given moment in time. For nearly 30 years, that relationship was what defined the web and allowed it to flourish. But that relationship is changing. For the first time in its history, the number of searches run on Google is declining. What's taking its place? AI. If you're like me, you've been amazed at the new AI systems that have launched over the last two years and find yourself turning to them to answer questions that in the past you may have looked to Google. While it's still early, it seems clear that the interface of the future of the web will look more like chat GPT than a Spartan search box and 10 blue links. Google itself has changed. While 10 years ago, they presented a list of links and said that success was getting you off their site as quickly as possible. Today, they've added an answer box. And more recently, AI overviews, which answer users questions without them having to leave google.com. With the answer box, they reported that 75% of queries were answered without users leaving Google. With the more recent launch of AI overviews, it's even higher. While Google's users may like that it's hurting content creators, Google still copies creators content. But over the last 10 years, because of these changes to the UI of search, it's gotten almost 10 times more difficult for a content creator to get the same volume of traffic. That means it's 10 times more difficult to generate value from ads, subscriptions, or the ego of knowing someone cares about what you created. And that's the good news. It's even worse with today's AI tools. With open AI, it's 750 times more difficult to get traffic than it was with the Google of old. With Anthropic, it's 30,000 times more difficult. The reason is simple. Increasingly, we aren't consuming originals or consuming derivatives. The problem is whether you create content to sell ads, sell subscriptions, or just know that people value what you've created. An AI-driven web doesn't reward content creators the way the old search-driven web did. And that means the deal that Google made to take content in exchange for sending you traffic just doesn't make sense anymore. Instead of being a fair trade, the web is being strip-mined by AI crawlers with content creators seeing almost no traffic and therefore almost no value. That changes today, July 1st, what we're calling Content Independence Day. Cloudflare, along with the majority of the world's leading publishers, is changing the default to block AI crawlers unless they pay content creators for their content. That content is the fuel that powers AI engines, and so it's only fair that content creators are compensated directly for it. But that's just the beginning. Next, we're beginning work on a marketplace where content creators and AI companies, large and small, can come together. Traffic was always a poor proxy for value. We think we can do better. Let me explain. Imagine an AI engine is like a block of Swiss cheese. New, original content that fills one of the holes in the AI engine's block of cheese is more valuable than repetitive, low-value content that unfortunately dominates much of the web today. We believe that if we can score and value content not on how much traffic it generates, but on how much it furthers knowledge, measured by how much it fills the current holes in the AI engine's Swiss cheese, we not only will help AI engines get better faster, but also potentially facilitate a new golden age of high-value content creation. We don't know all the answers yet, but we're working with some of the leading economists and computer scientists to figure them out. The web is changing. Its business model will change. And in the process, we have an opportunity to learn from what was great of the web of the last 30 years and what we can make better for the web of the future. Cloudflare's mission is to help build a better Internet, and I'm proud of the role we're playing in doing exactly that as the web evolves. I'm also proud that we're helping content creators stick up and demand value for the content they work hard to create. Happy Content Independence Day. Hello, Will. Welcome to This Week In It. How are you? I am good. Thanks for having me here. So, for those who don't know, where are you based? I'm in the Bay Area in California, though I feel like I've spent a lot of my time recently on airplanes back and forth between different locations, but very happy to be home. I bet busy days before and a few more ahead, I bet. For those who don't know, can you explain to us a bit of what you do at Cloudflare? Yeah, I lead product for a couple of initiatives, and in particular, been thinking a lot about the intersection of content creators, people who create for a living, whether they're designers or news organizations or publishers, and AI companies who have a lot of needs for interesting, original, high-quality content, and how do we build a system to bring those two things together? To be honest, I think you do one of the most coolest jobs at Cloudflare. Cloudflare is a very cool company. It touches many different types of companies, as we'll discuss today, from publishers and content creators to AI chatbots and companies. So, it's really a big spectrum, and you are working in a bit of those, too, in this AI age. So, a very cool area to work on, right? I mean, I feel very lucky, and I mean, we have so many, better than anybody, so many incredible colleagues who do so many cool things, like our post -quantum cryptography group is incredibly fascinating. I get to work with some of the research folks who have helped propose the new standards for web bot auth, people who do image transformations at global scale, the developer platform with incredibly cool things like durable objects. So, I do feel like every day, anytime I turn my attention away to a meeting and I come out of it, we've announced something new and interesting, which is both incredible and overwhelming, but not only the breadth of what we do, but the depth in each of these one areas. So, I don't know. It's the fun part about working here, is you get to have so many interesting experts in so many areas that every day you get to learn from, like, oh my gosh, I had no idea that we did this cool stuff. The topic that brings us this episode, really, it's related to, it was almost an innovation week. Cloudflare has this amazing innovation weeks, and last week it was all about content independence. Six blogs we had last week about this topic. You were involved in all of them, mostly, but for those who don't know, those who didn't check the news last week, it was all over New York Times. Everyone was talking about this topic. What should they know about content independence day, about introducing paper crawl? What should they know? The core concept, the way the Internet has worked for a long time is that you put your content on the web, people ingest it and crawl it and do all sorts of things with it, and then you have to ask them after the fact, hey, please don't do that. And the way you would do that is through largely a voluntary system of robots.txt. Now, that can work pretty well for some people, but it was really challenging for a lot of content creators and news organizations and publishers. And what we heard from so many conversations with folks who create original content and put it out in the world was that their content was being used and they didn't know it, and they didn't know who was accessing it or how it was being accessed. So we said, what if we tried to invert the model and moved from a world of opting out after the fact to opting in to a permission-based model for crawling? So what we announced last week was truly our day one, our first initiative in this movement towards more of a consent-to-crawl or a permission -based system for crawling, where by default, these things don't even have access to your website or your domain or your content or your news organization. And then you get to decide as the content creator, as the publisher, hey, great, I want to allow these things or not. The idea is that when you put your stuff out in the world, you should be in the driver's seat. You should be able to decide, great, I want all of my stuff to be fully ingested. I want it to be trained on. I want it to be searched on. I want it to be incorporated into every AI model out there. Great, that's your prerogative. If you want to allow some and not others, great, you should be able to do that. And if you want to block all of them, great, you should be able to do that. The idea really centered around that concept of control and permission, and that you should, when you create content and put it in the world, you should be able to say, great, I want these things to be accessed by these crawlers and not these other ones, and having you have that ability to control it. It's quite interesting. I was a journalist for a number of years, so this is a matter close to heart. In the past few years, I did actually a podcast called Made in Tech, where one of the recurrent topics was the business model for journalism. So that was a topic that I know well and I spoke about many times. One of the things I find really interesting in this day and age is that AI, I also see as an opportunity, because the old model was also not working properly for not only news organizations, but content creators, every type of content creators. Even recently, a few months ago, Wikipedia was saying that their bandwidth costs were to the roof because of different crawlers crawling their websites, so traffic is increasing. That also had a cost. So there's this area, quite interesting, where AI comes and it changes the landscape, it changes the business model that was also having some difficulty before. It was not perfect at all. Content creators were having difficulties before. So in what way do you see also this as an opportunity? First, of course, this will make sure that crawlers, may that be AI or not, respect the content creators' wishes. But in what way do you see this going? Yeah, I have this idea and I would like to think that we can make it a reality. Imagine firing up your favorite deep research product from any one of these amazing foundation models or AI -driven companies, and you say, I want to educate me about this particular topic. Maybe it's a health topic, maybe it's a legal brief analysis, maybe it's about gardening, whatever the concept is. You say, here's my budget. Here is $10, $20, $200, whatever the amount is, here is some budget to go out there. That deep research agent can go out and find you the best, most interesting, highly relevant, curated content for you and your particular need. I keep going back to this example when I think about it, because for me as a consumer, that'd be amazing to be able to empower an agent to go out and find this great content. And the key there is that the content creators could get paid. They could say, here's our great content, and maybe we have this genetic paywall, and they have the ability to say, yes, we can charge for it. I really want us to move towards a world where something like that is possible. Because I would like to live in that world to where the content creators can say, here's our great stuff, and here's what we charge for it, and people can buy it or not. And the agents and the crawlers out there can say, great, we can go out and find amazing content, the most relevant, interesting, timely, best analysis possible, and bring those together in an incredible user experience at the end of the day. And I think that's a lot of what's motivated me personally, is I think it's not an either or. It doesn't have to be adversarial. It really is a yes and. It is like thinking about what would I like to see in my individual consumer experiences, and how do we make that incredible at scale for AI companies, for search companies, for content creators, for researchers, for news organizations? How can we build a new model that works for many people? It makes sense. One of the things I find interesting is that you don't need to block it every time. I've seen, even as a feedback from the blogs that we published last week and what we announced last week, some content creators saying, hey, I prefer that AI crawlers crawl my websites because I want to be featured there in their chatbots. So some want to share their information. Some don't want to share all the information. They want to share parts of the information. That decision process is enabled here as well, right? The specific specifics of this. Yeah. I mean, the way we think about it is first you have to understand what's going on. When you have a website, a content creator, you're a publisher, a news organization, first just get a lay of the land. What's the data? Who's crawling? Where is this content being ingested? Where is it being used? And so we built our product called AI Audit, a name every accountant would love, just to understand what's going on there. And we think it's really important. Just get your bearings on the data side. The second phase is deciding what are your business priorities? And those are going to be different for everyone, right? Again, some folks might say, I want to be absorbed into the super intelligence, right? I'm writing for the future AIs, as some well -known writers have said. Great. If that's what they want, I think it's amazing they should be able to do that. Some were like, I want to write for this AI and not this foundation model. Great. You should be able to select that as well. And some were saying, I want this content, but not that content. Great. Or I want to block everyone. Great. Again, the core concept is you having the tools as the publisher, the news organization, the content creator, the website owner to selectively allow and disallow what you want to have happen. And then that, I think when you take that level of control as the publisher, that then empowers the AI companies and news search companies and startups and agents to be able to go out and build incredible experiences on a new permission-based model, which we think will work really... I do think it's very pro-innovation in that sense. And so the granular level of control for your content and what you want to have open to the world when you want to have more permission, really important foundational technical and product experience. And then on the sort of innovation AI side, really lean into that as well, of seeing how do you enable them to have access to great high-quality content, great data at scale, great experiences and making it work so their crawlers can be more efficient and get access to more things more quickly. Makes sense. You were mentioning the AI audit tool specifically, and here is an example of that specifically. Allow, change, block from the blog you wrote actually. So here's a visualization that shows that. I advise anyone that wants to learn more about this topic to check the blog post we published, this one in particular. The paper crawl is very specific on this matter. One of the things I think I've seen people also discuss online about our announcements is that the level of participation from publishers, from even AI companies. How much can we say in terms of the participation in this process from publishers, content creators, but also from AI companies? If you look at the press release we announced last week, it had an incredible group of folks that joined us. What was important for this is that some of these folks are customers and some are personal customers, but many are not as well. It was really about catalyzing a movement of saying, let's move towards this world of permission -based crawling and consent and permission to crawl at scale. That was a reason why a lot of these folks joined in the announcement, is they believed in that, even if they weren't currently doing business with Cloudflare. I think that was the fact that we had such breadth and depth from amazing technology companies, amazing publishers, amazing news organizations. It really showed that this was obviously an initiative that's near and dear to us, but it's a bigger thing. It's an industry-wide, multiple industry -wide initiative that we wanted to really lean in on and catalyze the conversation. We think of this, again, this is day one. There's so much more work to be done here. We took some immediate steps in terms of changing our onboarding, giving content creators more tools, making it easier for really interesting cryptographic verification of crawlers. I'm incredibly excited about that work that's happening. There's tons more to do. We announced this last week and I feel like everyone said, congratulations, you're going to take a bunch of time off. I'm like, no, now the work begins. Now we're really digging in to figure out how to make this work and what are the parts of the product that work really well and what didn't work as well, and how can we iterate rapidly and continue to build incredible experiences and great technology for our customers across the world? One of the things that I'm also surprised, first, it's the amount, and I just showed the press release that you were mentioning, the amount of publishers that are participating from the get-go, and this is quite amazing. Congrats to your team doing those partnerships as well. I think there's even more than what we announced specifically, but one of the questions I got many times about this topic is, the AI companies should be mad about this, right? That is not necessarily the case, right? Because they want good content. AI companies that have chatbots, they want to have access to good content for their LLMs. Yes. Whether that's for training or testing compute inference, RAG, whatever you call it, or search, incredible content makes incredible experiences, right? Again, if I just put in my individual consumer ad, I pay for the pro version of every large foundation model company out there because I love them. It's incredible individual customer experiences. I go back to that same deep research analogy there. If I can bring in really high-quality content about whatever field that I'm interested in as an individual, that makes that experience in that AI application or the thing being built that much better. I really think that this is helping move the industry forward in a really positive direction that's truly pro-innovation because great consumer experiences and great business experiences as well, for that matter, that are AI-centric and AI -first rely upon incredible content. How do we incentivize that incredible content and compensate folks for making that content and then make it easy to bring into your experience? That, I think, is a lot of the stuff that we're continuing to push forward on and to make incredible across the board. One of the things that I also think is interesting is a matter of opportunity in terms of there's many talks about misinformation, people trusting information. AI chatbots, I think, could be helpful there, but also the source of the content could also be helpful there. What is the feedback that you've been getting in terms of good content, in terms of trust in the content that is shared? I think it's particularly true for news organizations. If you're reporting on particular events, you want to have things that are trustworthy and grounded in some form of reality. In a prior career, I spent a lot of time thinking about digital content provenance. With things like the Content Authenticity Initiative and the C2PA, which is open digital provenance standards, if you're looking at an image in the world or a video, knowing where that image came from, did it come from a Nikon camera that was then pulled into Lightroom and did some small edits and then before it was displayed on your website, that's incredibly important information. You want to try to ground yourself in the reality of what's happening out there. Just this shared basis of facts, I think, is incredibly important. I think that goes into the same announcement. We announced with Cloudflare Images our support for the C2PA metadata standards for content provenance earlier this year. I think it's incredibly important stuff that helps ground you in what's real versus not real. I was reading some interesting reports about new deep fakes for politicians that are out there. This has been a growing concern for folks from, I remember, I was for sure thinking about in 2018, 2019. The technology is just phenomenal today. We've known it's been coming for a long time. I feel like that world is definitely here. More ways to bring in trustworthy content, not saying it's good or bad, but at least here is the facts of what happened. This was actually a recording. This was actually a real human. This was actually the words that they said. Then bringing that into both AI-centric experiences, but also just into the news and publications in general. I think those are really important foundational truths that we need to continue to push on, which is why I'm just personally still very interested and excited about digital content provenance and how can we continue to push the boundaries there. Makes sense. In this day and age, one of the things we've been noticing, even, for example, Twitter has Grok. People ask Grok, hey, is this accurate? Grok, is this real? Yeah, exactly. Exactly. The same thing for ChatGBD. I just saw this. Is this accurate? Is this like a real news? It's kind of interesting to see AI actually helping chatbots, AI chatbots actually helping to provide the context sometimes and information, even for the immediacy of social media. Yes. That is one of the main ways people get news these days, but having AI could be actually helpful there. How do you see specifically this relation between AI companies and publishers? Can we help here in this relationship specifically? I mean, I certainly hope so. I think a lot of these publishers and AI companies are doing their own things already, and that's amazing. What we can help with in that spectrum is how do you make sure that the company gets the content that you want them to have access to as quickly as possible in the right format? Maybe they want it in a markdown version. Maybe they want it sort of cached in these areas. Maybe it's programmatic access. Maybe it's API access. You want to secure that. All things that we can do. But I do just think there's just a new... We're building incredible new experiences across the board, and it's really the global royal we about how we think about information, how we consume information, how we consume content, how we create content. I mean, it's one of just the most exciting times to be working in tech ever. Certainly for me, I feel like the pace of innovation or the pace of change is... Everyone says this. It's like, try to say it now, but it's true. It's just faster than ever, and it really feels like we're hitting this takeoff trajectory, which is exciting and also disorienting. I do think just the role that we can continue to play is building great products and great experiences and helping folks, whether you're building a new AI application and you want to build on top of Cloudflare, amazing. We want to push forward aggressively there. If you're thinking and you're a publisher or a content creator and you want to put your site up and keep it free from DDoS attacks and other bots that are out there, we help you there as well. So I think it's such an exciting time to be building across all of these areas. Absolutely. One of the things I find interesting is that robots.txt that we were mentioning before, it's not a new thing. It's actually around for several years, but it has become more important these days because of AI, of course. But the fact that... And I didn't know this when I started writing about these topics a few months ago. I didn't know that robots.txt was not enforced. Crawlers did it if they wanted, completely if they wanted. It was not enforced. But with our new tools specifically here, robots.txt is also a way of content creators thinking about what they want to protect from what. So this announcement, this blog that I'm sharing about control content used for AI training with Cloudflare's manage robots .txt and blocking for monetized content. It's a way also to help content creators manage that file, understand what that means specifically, have that tool in place if they don't, right? Yes. What we found, and I think it's in this blog post, the numbers aren't exactly correct up top of my head, but it was like the majority of some of the largest sites that are on Cloudflare don't have any robots.txt file, none whatsoever. And maybe it's because they don't care about it, but I don't think it's the case. From every conversation that we had, I've actually heard folks saying, yeah, 37% of the top 1 ,000 domains currently have a robots.txt file, which means the majority do not. Most folks just didn't know what to do, right? You can think of this as a no trespassing sign or a code of conduct you put up in front of your restaurant or your house. And that code of conduct, you say, please don't step on the grass and please be respectful and everything else. Those are good rules of the road, but they're not a good actor will follow them, but the bad actors will not, right? And so first and foremost, we said, how do we help you if you don't want to create your own or you haven't created one, you're weren't sure what to do? How can we make it incredibly simple for you to do, like literally one click toggle, and we'll do it on your behalf, right? We'll tell you what's in here. You get to opt into this and we can run this file on your behalf. And it's much easier for you because most people weren't doing it and they weren't aware that you needed to do it. So that was step one. I think it's really important. That then helps take care of a lot of the good actors, but then you have to think of what about the bad actors that are out there that look at this code of conduct and like, nah, it doesn't apply to me. And that's where I do think you want to look at more network level protection and bot mitigation. And so it's really important, we think, to have both of these options. One is to manage robots.txt, to lay out your code of conduct as a website owner or publisher. Here's what we want the world to see for what we do. And then also be able to actually enforce it, right? So think of it as like, it's not only the no trespassing sign in front of your house, but then there's actually, you could have a lock on your front door and you could open up that, you know, unlock that door for your friends and family and people you want to invite over. But for everyone else, you can just say, nah, this door is actually locked. And it makes all the difference because if something is not enforced, bots are not friendly several times. So it's a matter of trust. So it's quite important to enforce those type of things. We also announced this new addition to Cloudflare radar that is very specific to this area about the referrers specifically. And it shows a tendency, not only that, of course, search gives more clicks per crawl to typical content creators, but AI chatbots don't. Entropic being the one that had lowest results here. So more crawls per click specifically, right? Why we create out in the world and we do it for, you know, people create for to make money, of course, to tell a story. And it's true if you're a graphic designer, if you're a photographer, if you're a news organization, a publisher. And so there's the monetary aspect that's incredibly important. But there's also just the credit side, the attribution. Does the world know you did this thing? You know, I spent a lot of my career working on a platform for creatives. And so I just thought a lot about attribution and how do you both give people the exposure for the work that they deserve and then make sure the world knows that this was you who created it. Because our ego matters, right? It's about, you know, not only our livelihoods in terms of compensation, but also just in terms of like having the world know that it was me and people saying like, wow, great job. That was amazing. Thanks for writing that piece or thanks for creating, you know, this new thing or thanks for reporting on this thing over here. Just that level of attribution is really important. That is much harder in the world that we're in today, which is different than even a couple years ago, where so many of our consumer experiences are really driven by, you know, incredible chat experiences, but where the attribution out to the original sources is much more muted. And, you know, that can be fine for even trying to do synthesis, but it's harder, you know, for a content creator in a lot of cases. So there's just real challenges here of like, how do you still create and how do you get credit for the work that you've done? Leaving aside like, you know, the compensation part, which is incredibly important, but just the credit, just the attribution side. How do you continue to sort of carry that through for creatives and artists and news organizations and publishers and content creators across the globe? Makes sense. Actually, it's not only, of course, there's the business side incentive, but if you don't have a credit incentive, you'll do less content or less quality content. You'll be less driven to do better work maybe. And that impacts society at large in terms of humans not having that drive to do more, to do better, better content, to investigate more. So the incentives are quite important in this area for sure, not only on the business side, which is important, but also on the credit side. So it makes perfect sense. In terms of what's to come, we launched the first step, as you mentioned here. We got so many great feedback I've seen online. How would first you summarize the feedback that we got? Were you surprised by the feedback that we got? And then next steps? Yeah. Not surprised, but overwhelmed by like the sheer volume of it in a great way. Like we, you know, every conversation we've had across the board, people are like, great, this feels like the right step. Thank you for doing this. And, you know, when you finally were able to sort of get out and talk about things more publicly, it's a relief because you've been trying to sort of work behind the scenes for a long time. But just hearing that sort of from across the board, more folks really recognize what we're trying to do here and trying to, you know, push forward new open standards for other folks to adopt and think about. It's been incredibly rewarding, but we really, I just anchor on and, you know, our whole team here anchors on this is day two of this initiative, right? You know, like the announcement was like day one and that now begins a lot of the really interesting, more complex, more interesting work of like, how do we continue to build great experiences for content creators and news organizations and publishers and AI companies and people innovating in search and the world of agents? So I love the feedback, anyone out there, keep it coming. You know, tell us what we should do, what we shouldn't do, what we're doing, like what works, like, cause we want to build these great experiences. I'm, you know, my list of things that I want to launch is like nearly infinite. I bet it grew in the past few days. It grew, exactly. It was long before, but now it's like, now it's much longer, but we feel really good about it. Like just the things that we want to get out the door, just like literally the next couple of days and weeks. I'm very, very excited about. One of the things that I've noticed is that the SEO, search engine optimization community, really picked up on this as a, Hey, we were missing something. And this seems to me, the piece that was missing in terms of things are changing or Google has AI overviews now, and AI bots helping on top things are really changing and people were worried. And this seems like the next thing that makes sense. Do you feel the same there? I do. So much of the stuff is just, do you have the right information and the right data and the right ability to understand what's going on? And I feel like that's so much of what we've anchored on is like, can you understand what's being crawled, where it's being crawled, who it's being crawled by? Then can you make your business decisions and then can you enforce those? And so I just think more information is incredibly powerful and making it visible. A lot of them, we've been thinking about bots for 15 years. And so a long history of thinking about bots and what's a human, what's not a human. And this is just the next iteration of it. I do think it's accelerating and it's accelerating in terms of how we think about and how we consume content. But just step one has always been like, let me get more information and understand what's actually going on here. One of the things is actually on bot authentication, getting to understand that the bots that you're saying that they are those bots are really the correct bots. So this is also part of the announcement message. Signatures are now part of our verified bots program, simplifying bot authentication. Authentication and trust, even in the bots area is also important, right? I am so incredibly excited about this work that this team has done. I think it really addresses this just really big need that hadn't been addressed for a long time, which is like, how do you know a crawler is who they say they are? And a lot of folks do it today with IP addresses and do a reverse DNS lookup. That's amazing. That's harder if you're a smaller company, if you're a smaller crawler, if you're a startup, and that we felt like that was still missing a lot of things and that there was so much of the Internet is built on cryptography, right? And so we think about DKIM for email and TSL, SSL certs for websites and the C2PA stuff for digital content provenance. That's the same concept of signing what you're doing and doing that for bots feels like a natural extension out there. So you can be able to say like, yes, this is who I am and here's how you can actually cryptographically verify this is who this crawler is from. I think this unlocks so many interesting use cases, not only to identify the largest crawlers that are out there that are ingesting content for training, but it becomes really interesting when you think about search. And when you think about agents, and when you think about how to sort of like build an agent that's operating on behalf of someone, we're going to have like a certificate chain that starts to say like, hey, here's me. And here is my remote browser operating on my behalf, that I can then sort of like punch through and tell the end website or origin that there's a real human behind this and see that sort of full certificate chain from there. This is one of the most exciting things I think we've worked on. It's a proposed open standard that's out there. Anyone can sort of look at it, work to make it better. The team on the Cloudflare side have been working on this are just phenomenal researchers and product folks. And I think this is like the beginning of a lot of really powerful work. Absolutely. I did actually a full episode about this a while ago, a few weeks ago, with Thibault Meunier from our research team specifically, only about this. And it's quite a very specific, but quite interesting for trust, even. Yes, for sure. For those that want to stay tuned about this topic, what can we say about what's coming? Things that we're betting that people should be excited about? Yeah, a lot of, you know, continue to push forward in the web bot auth and the verification. How do you really think about that working at scale across the board for agents? And how do you as a website owner, a publisher, a content creator, have those granular controls to say, great, I want to allow these sorts of agents in, or great, I want to challenge these sorts of agents. So really thinking about that work is a lot of stuff that we're actively developing and pushing forward on both open standards. On the paper crawl side, it's new. I'm open about it being sort of an experiment. But the level of interest is wild and amazing. And I'm incredibly excited to continue to experiment there. There are a lot of ideas that we have to sort of take the program as it exists today and just scale it up. But also think about new payment mechanisms, more open source standards that we can incorporate, and then really sort of build the thing out. So lots more to come in the paper crawl side very, very soon. For those who want to interact with this area, read the blog posts for sure. There's a lot there, even call to action, like how can you fill up forms and for verified bots or even test out paper crawl specifically. But what can we say specifically if they want to reach out for any topic? Yeah, I'm on Twitter slash x at William Allen. Love to hear from you there. That's probably the best way to reach me or on LinkedIn as well. And yeah, we'd love to hear from you, what works, what doesn't work, what you would like to see. Keep the feedback coming. It's been incredibly helpful for me to get feedback from folks in so many different parts of the spectrum. People who are content creators in one aspect, both combination content creator and AI company, AI companies. There's just so many interesting areas and we'd love to hear that feedback as we continue to sort of push forward here. So Twitter slash x and LinkedIn, probably the best ways to reach me. This was great. You'll be in this show in a few months potentially because there's much to discuss about this topic for sure. So thank you, Will, for this intro to this topic. Amazing. Thank you. And that's a wrap.