Building an open agentic Internet: Discoverable, callable, and payable

Welcome, everyone. Thank you for coming to my talk today. Anyone here was patching PHP applications at 6 a.m. this morning? Just a few more. So I'm a little tired, but we'll get through it. It's not going to take the full 45 minutes. The topic for today is a big one. If you were at the keynote yesterday, there's actually a bit of a preview. This is a bit of a longer form version of that and actually some tactical examples of what we're doing to build, of course, an open agentic Internet that helps everyone, content producers, website owners, as well as, you know, the frontier model providers, which helped me a lot this morning because Claude did patch a bunch of my servers. Right. So the premise to the whole situation. You can actually check this up on radar.Cloudflare .com. If you haven't seen the website, I highly advise you to take a look. There's a whole bunch of really interesting filters that you can filter by region, activity, bot type, purpose of crawling. The headline, of course, is the majority of the traffic we're seeing across the Cloudflare proxy, give or take, we're talking about tens of millions of websites at this point, is mostly non-human or better, bot and automated. And we also know that the increase over the last, you know, 12 to 18 months has mostly been driven by all the new use cases for crawling, training being the prime examples. Everyone is building a bot. Everyone is fetching content via their agents. And this is the output of that consequence. It was mentioned at the keynote, the order of magnitude difference over the next couple of years is just going to grow. So take an example of you running a small little website. You may be receiving a couple thousand requests per minute today. You can probably expect that over the next years, that number is going to go way higher and it's all going to be automated because everyone is hungry for data. We zoom in. Again, this is screenshot from Radar.com. And just as I said earlier, the number one use case, and we do this by actually identifying where possible what bot is accessing the content behind the proxy. You may have heard of WebBot off. Another really cool thing on Radar, you can go and see our bot directory and you have a list of hundreds of bots we recognize. And a bunch of them, especially the big provider ones, submit a cryptographically verified HTTP header. So we can say, yes, this is OpenAI. This is Anthropic. This is Google. This is Muse from Facebook and similar. And based on the bot, most of the providers split their bots based on use case. We can tell that, you know, number one case is training or some sort of agentic behavior to power our conversations with our new bot overlords. Search is still there. Not negligible. About 13% of traffic. That is your traditional search that you might see in Google, Bing, Yandex, you name it. Pretty constant, but not definitely more than the agentic use cases. What does this mean? So we started a year and a half ago, we had content independence day, when we started noticing the behavior changing on Internet traffic. And the cannery of the whole situation of the business model is changing. Of course, it's websites that produce original novel content. I actually run as a side one of the little news agency for my own town, local town, my hometown in Italy. I've actually seen this myself on the server looking at the logs. Human traffic is dropping a lot. Considerably. And I run the website on behalf of a friend who's actually the owner of the website. And just like these big brands here on the slide, his revenue from ads is just going down. His revenue from subscriptions is going down. Why? Because today you open your open AI chat GPT. What's the latest news? Even this morning I was looking for the latest, when was the patch out for PHP? I could have probably checked PHP.net, given it was the front page article on the site. But I actually asked Claude, and Claude fetched the page for me and gave me the information I needed. Did I as a human go to PHP.net? No. And you can see how this is changing behavior for everyone. And those who make money from human visits are suffering the most. Pretty impressed. Digital trends. This is actually published by a third party organization that looks at this data. So not even Cloud for Data. 97% less human traffic hitting the site. This is just complete disruption of their business model. Okay. So some would argue, you know, historically you get a crawler accessing your site, getting your site indexed. That's good. A bunch of us probably have an experience with search engine optimization. Is this bringing any value? We can actually see, looking at refer headers. And again, this is public. Go to radar.cloudflow.com. How much are these providers crawling and how many humans are they sending back to their site? Sometimes you actually may see, you know, the chat agent is giving you a referral link and you may click on it and actually visit the site, consume the ads, buy a subscription or something. But it's pretty bleak, let's be honest. Look at perplexity. For every 3,400 visits their agent accesses any given site, they're sending one human visit back, right? And it doesn't get better throughout the board. And this means more resources, more load, more servers that you need to step up to keep your sites online. We track this over time. If you actually click on radar, you can actually see the difference over the course of the past couple of months. And these, of course, are just the top ones that we picked, handpicked. If you go to the bot directory, you can actually see on any given bot that you might be interested in many more providers than the ones listed on the slide. So the big question is, what do we do now as an Internet community or people that benefit and enjoy exploring and using the Internet as part of our daily lives? We need to essentially make sure that the business model stays up so that the Internet doesn't devolve into something that's worse, let's say worse over time. As I said earlier, the canaries here, of course, the news publication websites, they're suffering the most. But you can see how this trickles through, for example, companies that are publishing research, journals, news, weather companies. And eventually it will probably hit everyone, including eCommerce installations, small blog providers. I have a personal blog that I keep updated, and I'm probably going to see the consequence of that. Not that I get many visitors in the first place, but it's only going to get worse. The other thing, some of you may not know this, some of the browsers are actually embedding or provide you, some of the frontier model providers have their own AI native browser. Someone heard about Comet, for example, as one example. When you're accessing web pages using those browsers, sometimes they're actually doing crawls in the back end because they come across as, of course, legitimate humans, and it's harder for bot detection providers such as Cloudflare to detect them and stop them, and as a consequence, actually making it worse for everyone. Good news, you know, new business models, this is just the way history evolves. Companies and industries have been disrupted before. So for us, it's just a matter of figuring out what is the next. A couple of very well-known examples here, the Napsters of the world. Anyone here used to use Napster? Yeah? They've disappeared. Replaced by Spotify with a subscription model. We can debate forever if that's good for the artists or not. Some will say yes, some will say no. There's been some very interesting, you know, scams and similar going on some of the subscription-based platforms. I still remember as a kid going to Blockbuster, the Italian version of it, long gone, replaced by Netflix, and, of course, I did see someone actually with a paper-based newspaper in the tube in London the other day, but fair to say that's the minority, not the majority. Everyone's reading news online. Okay. New business model. We're starting to see it take shape. Some of the large content producers, for example, have already been able to, and this actually started early in the year and into even the late end of last year, signing direct deals with the big frontier model providers such as OpenAI and Thropic and similar. The good news is they actually want to sign these deals with the content producers because there's a benefit for both parties, right? If I sign a deal with a big news corp and I'm a frontier model provider, I can probably optimize how I receive the data from them so that I don't have to crawl 1,000 times or continuously refresh and fight against them to try and bypass their mitigations so that I can provide a better service to my end users. This is the start. I would say at this point, the majority of the large news publishers, even if they haven't maybe publicly declared or not, like People, Inc., have actually now got in place deals with the frontier model providers for them to access content, especially because freshness of content is extremely important, right? You want to know what's going on in your local town. You ask the chat agent. They should be able to give you really up-to-date information. It's also important that they get up-to-date information from the agents because we as humans are starting to rely on them for our everyday tasks, and if the agent doesn't have fresh information, we've all heard of hallucinations, right? We may actually get misguided, misaligned, get the wrong content from the content producers, so actually it's good for everyone if both parties work and find a good way of working together. This is just an example of, of course, Google AI overviews, maybe putting people at risk on health advice because sometimes they have just out-of-date or no access to the latest, best content with the correct information. So at Cloudflare, you've heard this at the keynote. We're trying our best to be the impartial party, figure out how can we help both content producers, website owners, our frontier model providers. Of course, at some point we talk about money, so monetization is arguably one of the end goals, but we cannot just do that, right? We need to make it easy for both sides to achieve their end goal. So these are a few tactical examples of things we've been working on. Number one, making it easier to crawl. Now, arguably lots of people here do not want to be crawled, but I prefer to say some of us do want to be crawled, and if you have to crawl me, I want to make sure you're as efficient as possible whilst doing that so you're not taking my website down or your crawling doesn't look like a DDoS attack against my infrastructure. I've actually had a couple of sites go down because I was getting hit quite heavily by meta ink crawlers a while back. I had to block them even if I was trying to give them product catalogs and similar. If you go to the Cloudflare dashboard and you have a website on Cloudflare, who here has a website on Cloudflare? Okay, quite a few of you. If you want to give access, there's a little crawl option. The proxy will automatically issue a new endpoint, which will only be accessible, by the way, by a signed verified bot, so we're trying to keep things as clean as possible, and we will parse the content out of your site and actually provide a much easier to crawl markdown syntax to the bot so they can access the content over and over again if they so wish to do so without hitting your backend server or increasing load to your application. Okay? So this is one of the ways we're trying to make it better for both parties. Second one, make it easier to search. You may have noticed if you do a prompt on a frontier model provider on Cloud, Opus, or whatever, very often the first thing the agent will do is kick off a bunch of subagents that will actually go on off and perform searches on the open Internet. This is one of the initiatives, but fair to say are still very much under development, with a plan of building an open index for all the sites sitting behind the Cloudflare proxy, give or take we're talking about 20% of the Internet, so that the frontier model providers or whoever, if you want to build one tomorrow, again, it's about having diversity in the ecosystem, can actually refer to the index and actually get content without having to search for it, which is a lot more resource intensive. Even this, by the way, is not opt in by default. You can go to the dashboard, you can decide to turn it on and opt in into the index or opt out. It's your content, it's your application under your control. Plan is make it easier so you don't receive the load and, of course, it's a safe path to AI adoption over the long term. Last one and not least, search is part of it, and the index definitely helps. Whilst we are building out this index, of course, the search engines and the crawlers have a discovery problem. I remember I used to work at a company before I joined Cloudflare, one of the things we were doing was an Internet service survey and discovering new domains and sites for the purpose of surveying the Internet was one of our biggest challenges, and we actually used to partner with the likes of Microsoft to get a list of domain names that were signed up or by registries. Similar sort of incentive, we actually did a research, and you can find it on our Cloudflare blog, we partnered with OpenAI in this case to give them signals of new domains that are being added behind the Cloudflare proxy with the crawl endpoint turned on, so opting into being discovered. We did it on a very small percentage of traffic, so not the entire application set behind the Cloudflare network, and you may have heard at the keynote, this resulted in OpenAI in this case discovering several million new domains and applications without having to necessarily bombard all of us with their crawls. We think this is the win, especially for the smaller players on the Internet, so let's say you're starting up a new site tomorrow, you will be no-one on the landscape, but with something like this, the providers will be able to find you more easily, and you actually show up in their search results or within the agent, so you can get access to your end users. Okay? There's a lot more we're doing. These are, I think, three of the best concrete examples we're actively working on, so homework, go in the Cloudflare dashboard, find the options, you can make the decisions to access or not. One last thing about the slash crawl endpoint, which actually powers some of the research we did with OpenAI, there's actually a standard called crawl control, which we only let bots who are at least it's a trust-based system wanting to access your content for a given purpose to actually give them that content, be it training, inference, or, for example, requiring referral back to your site or not, and all of these options are for you to configure and decide. We're actually pushing for a standard to be done by the industry. There's still some work to be done there, but over time, we're confident we'll make it happen. Okay. So frontier models, crawlers, building a new one yourselves, you're a content producer trying to connect the dots for everyone. Now we can start thinking about monetization. These are the last two things of my presentation today. Number one, pay-per-use. This little video is a background. You as a content provider or content producer want to get, you know, some part of the pie when it comes to making money at the end of the day. There's two big challenges here. Number one, you could actually get paid every time a crawler tries to access your content, and you may have heard of our monetization gateway, which is my next slide. Everything is good and great. You can set a price. You can say if you access this API endpoint, I want to be paid 10 cents, and there's a cryptographic way to make it happen. However, there's a bigger problem, which is what we call pay-per -use, which is still under development. We're figuring out how to make it scale up, which is a frontier model provider only needs to crawl you once, and they may reuse that content 100,000 times. So you need to think about how do we make sure that every time the content provider uses your content, you get a little portion of that value back in your site. So if you're wondering what's the difference between pay-per-use and monetization gateway, this is the crux of it, is reuses out of scope versus accessing your content. Pay-per-use is based on the trust system for now, so it needs to be signed agents accessing your content. You set the price in the dashboard. Open AI comes along. We take a look. We say, yes, this is a price we're willing to pay, no, we're not, and then they will add the content to their search engine, to their agent and frontier model providers, and in theory, every time they use the content, you receive a little value back. Everyone following so far? Second part is, it goes to the next slide. Seem to be stuck. Do you help me at the back advancing the slide? Monetization gateway, and the video, there you go. Second part is monetization gateway. This is in private beta. If you go to the Cloudflare dashboard, there's a little on the left of the account level, there's monetization. Click on that. One click to request access to the private beta. This is essentially the technology built on top of the proxy so that you can then define at access time how much you want to be paid. Easier from a technological standpoint because we're not relying on the trust-based system. You define the prices. A signed agent comes along. Of course, we'll receive a 402 payment required response from the server, and assuming it's a signed agent and they've implemented integration with our protocol, they will submit the payment to the relevant payment provider and then reaccess the content with proof that they have paid. This is the token. Proof happens, and then they access the content that you're providing. In this case, it can be an API. If you expect you have a service and you're not building necessarily content, but it's an API service, you may want to get paid one cent for every API call. Monetization gateway, which by the way is an open standard, is a tool for you to achieve that so you can scale out payments to agents without having to worry about collecting money from each one. Probably very useful for the smaller players that cannot go off and have big signed agreements with the frontier model providers. I advise you to sign up for the private beta. You can set up rules on all sorts of filters you wish. You can decide to make portions of your API requiring payment versus different filters per region. You control the rules. We implement them as part of the proxy. Under the hood, who is here at the X402 foundation? A couple of you. Not too many. We actually partner with Coinbase and a couple of other technology companies to come up with a standard and the protocol that makes it happen transparently. I'm not going to go into the details of this right now. Again, we are one of the providers that implement the standard. There are others out there. Of course, we're the best ones that made it happen. Up to you. You choose. You can even implement it actually on your own servers without a proxy if you wish to do so. Everything is cryptographically verified. The only challenge here for us has been over time is how do we scale up this concept of micropayments. You may also have heard that we issued on our building out a stable coin called NetDollar, which is part of one piece of the puzzle to hopefully make this happen as it scales up to everyone using the Internet at large. Yes, and hopefully, maybe by the time we meet again next year at a similar conference, we'll start seeing the new business model of Internet taking shape. With that, I promised it was going to be sweet and short. Thank you for listening to me today. Yes, any questions from the group, I'm happy to answer them. Ben? I think we have a mic so everyone hears you. On the monetization side, are you guys thinking about bid-ask-spread? I think most people have no idea how to price. Good question. Thinking about the two use cases again, subsequent usage of content that was crawled versus payment at access time. When you go into the monetization gateway, which is, again, fully built, you just need to access the private beta. Once the team approves, you get access to the dashboard. You just define the price for the endpoints, and that's the price you set. You can figure out what you want, and the bot can decide if to pay or not. For the pay-per-use, which is something we're still having fully rolled out to everyone, which we're trialing with some of the big content providers, that is more of a bidding system, because it's a bit more complicated to give value to something that gets reused over time, especially if it's not crawled time, but that does have a bidding system where the providers can say, I want to pay you X, and then you can decide the rules of I accept or this is the range I'm willing to go. We try to optimize a little bit like Google AdWords, you can imagine, to get you the best value out of your content usage. Having said that, this is well under development, so I'm sure there's going to be lots of subtleties that need to get worked out still. Down there. Since you're targeting content providers, are you looking at ways where some content provider, let's say New York Times, right? I can allow OpenAI through crawl to get the headlines, but if the user then says, okay, now I want to read the full article, then I go in, click, and then I do a pay -per-use? That's a very good question. You could easily build that yourself today by using the monetization gateway. Let's say you wanted to get paid even for the headline, because sometimes lots of readers just read the headline and they're happy and they move on. I've actually done something similar on one of my sites. You essentially would have an API endpoint that responds to the sign agents with a summary of each articles, and then you can apply monetization gateway on top of that. As part of that summary, you have the referral back to your content, and then if the user clicks more, then they would subscribe through your standard process. The challenge with that, by the way, becomes then just generic bot mitigation, because what we're seeing, okay, we're doing all the right things, and the big providers are actually willing and wanting to make this new business model work out. There's a very long tail of other providers or potential crawlers that are trying to bypass all of these mechanisms to the point we've seen sophisticated agents even sign up to subscriptions, crawl all of the content, and then redistribute it. There's actually services out there, legitimate businesses in many cases, that are doing that on behalf of content providers. The complexity becomes in locking down your content in a way that you're happy it's not leaking. One of the side effects of this whole new business model has been the web archive, for example, because they're essentially a very legitimate good reason to exist, but now they're getting abused because they're just echoing content that you may not want to get leaked. Any other questions on that? Hi, very interesting models. I'm just curious, how do you balance the monetization versus discoverability, because isn't that you're adding hurdle or gateways for free indexing content? So our current position is you as the website owner and content provider make the decision. It's near impossible for us to make a judgment on your behalf. We do not want to make the judgment. We're providing, or we think we can help everyone solve their problem by providing the tools and then it's going to be in each individual or business that makes the decision. I want to block bots and agents versus I want to allow them for free. That's something that many companies and businesses want to do versus I want to monetize. It gets complicated if you start thinking about it because I agree with you, it's a hard problem. A smaller news agent may want to give access to the agents for free because ultimately they're smaller and they're trying to grow. Maybe that's a good thing. Whilst the big company that has big bills to pay wants to start monetizing from day one and it creates very odd balances, but it's your decision at the end of the day and we'll figure out how to help you the best. Hi, how are you? So do website publishers need to structure their website content or data in any special way, whether it be meta tagging or markdowns or hierarchy to maximize their monetization opportunities? Yes, absolutely. This is actually what the slash crawl endpoint does. Of course, it works better for smaller sites. If you're talking about very large publication like BBC, we're going to be building something custom. Markdown is the language that a lot of agents are accepting and it's lower weight. Having said that, modern agents are able to parse anything out. You may have seen our announcement for our new model, Cleric, that we just launched. The whole point of that model is to convert unstructured content into structured content, which is solving that problem. Having said that, if you look at the big content producers, they all have alternatives to their standard human readable sites, which are optimized for agents. This is the new CEO of the agents and agent optimization is all part of that, making it easily readable by robots. Question at the back? Hey, I just set up AI search on one of my sites while you were chatting. Very cool. How do you think of the AI search functionality along with some of the other things like the crawl feature? The crawl feature is creating the markdown file. The AI search is a different feature. I just created an MCP and put it in my cloud so I can interact with it. How do you think about that universe working together and not creating different functions? We're trying different things at the same time. Arguably, to your point, slash crawl versus search are complementary or overlap in some ways, but the slash crawl assumes that the bot or the frontier model or the crawler builder knows that you exist as a website. So if I know your domain, then I try to access it with my signed verified agent and because I'm signed and verified, the team has made sure that you're doing the right thing, so I follow content control directives. I can access your slash crawl, get the content. You never see the request in your server, so you're happy and we echo that. If I don't know about you, though, in the first place, I don't know where to go. This is what the search does. So the whole point of opting into the AI search is that you'll be added to an index that the content crawlers can separately access to discover new sites on the web. So that's sort of their entry point. They may get content directly from there, which is even more efficient, but once they know about you, then they'll be able to hit your slash crawl. So that's the difference from a technological perspective between the two. Great questions, by the way. Any more? Yeah. I have one about, so you mentioned like there's a surgeons of like these resellers who are like just purchasing a subscription and then reselling it. So how do you kind of avoid this race to the bottom for like all these content providers? Like if anything takes off basically, right, then there's an incentive for someone to get it and then redistribute it to agents via that. So is there like a... That's why this problem is hard and that's why we exist and we're trying to make it better. You've all maybe have heard of residential proxies. Yeah. So this is where if there's companies out there, you go there, you want to crawl something, you get blocked. You go to another company, they give you a bunch of residential IP proxies. That's simple way. We detect that, for example, but that's the simple way where people are trying to bypass protections. It is a race to the bottom if you're open and you don't have visibility of what's going on in your site. That's why, again, we're for Cloudflare. We have our bot detection product. It's trying to detect every occasion that someone is trying to bypass your controls to make sure you're creating scarcity and then you can have a business conversation. But yes, that's the challenge we have because it's a race to the bottom otherwise and the whole business model of the Internet falls apart. Yeah. And residential proxies, by the way, it's relatively easy now. There's way more complicated cases of people bypassing detections. Questions at the front? Yeah. So I'm one of those people that we want to get crawled. I'm with Maricopa County elections and we want to get the information out there. Right. Right. Because we want the right information out. I'm curious as we're updating throughout the day, if we do that, is that a constant scrape in the update or is that just what's the frequency on? On the slash crawl, you mean? Yeah. On the crawler. Is it kind of pushing out, oh, hey, something just changed or is it very passive still? That is the idea of the so some of this is still in development. We're still figuring out the idea of the search index ultimately is also to be able to push to the content creator a notification of this content has changed and therefore we're optimizing the whole system from the ground up. Right now, it's still not happening at large. So even if you enable the slash crawl, you're going to get hit on the slash crawl. At least we're taking the load, not you as the content provider. You may have the challenge that you do want to get crawled and therefore you need a way to notify that your content has changed. That's something we're actively working on. We're trying to solve with the search index and also with an integration with the relevant providers to ping them back, sort of subscribe to the content. This is where you get the fresh version. It's an ecosystem that we're building out. Any more questions? One more? So there's this like view that a lot of the pre-2023 content was disproportionately more useful than content that's maybe being created now in the sense that it's already in training corpuses and either like through open deals or just through like straight up scraping. I'm curious how you think about the value for like information that is being generated, created, and put on the Internet now versus what's previously existed and sort of the economic value of forward-facing information versus like what's already been scraped and crawled. So I don't have the perfect answer to that question. What we do know is the subsequent reuse is important to, there's some monetization value there. That's why pay-per-use is technically a different concept from monetization gateway. Think about Spotify, right? Every time you play a music, a song from an artist, they will get, you know, the relevant charge for the service, right? And the same applies to Internet content or we believe should apply. That's why pay-per-use is the thing. Content definitely value over time goes down. This is why for pay-per-use, the question Ben was asking at the start, there's more of a, we're building out a sort of bid slash response over time with the content value changing and we're trying to build a system that optimizes that for the content creator. We're also in a stage where the Internet is blowing up with content and replicas of each, I don't know how that's going to play out. As we build this out, hopefully we'll start fine-tuning what works best so we maximize and everyone's happy. It will take some time to get there. One more question. Anyone? Okay. So thank you for coming. This was a great set of questions and homework. Check out the features, especially monetization gateway in the dashboard and enjoy the rest of Connect. Thank you.