Introducing the Agent Readiness score — check to see if your site is agent-ready

Hey everybody, my name is Brendan Irvine -Broque, Senior Director of Product at Cloudflare, and I'm so excited to be talking to you at Agents Week. I'm here with André. André, you want to introduce yourself real quick? Yep. Thanks, Brendan. Hi everyone. I'm André Jesus, a Systems Engineer here at Cloudflare based in Lisbon. I mainly work on a product called Cloudflare Raider and other similar products like the one we'll talk about today. Yeah. And so André and I have been working on something fun. It's called Is Your Site Agent Ready? You can go to it, isitagentready.com. And what it does is it lets you scan your website for compliance with all the emerging different AI standards out there, whether your site supports MCP, whether your site returns markdown back to agents to make them more token efficient, whether agents can discover how to authenticate against your API, all kinds of different things that we're doing at Cloudflare. And so we want to talk to you a little bit today about some of those different standards and walk you through that. André, maybe tell me a little bit about the workflow here. So why do we build this? And maybe we looked at some data on Radar around adoption of these standards more broadly on the Internet. Yep. So basically, the problem that we wanted to solve is that most websites were built for humans, and now agents are everywhere. And they are already browsing the web and most websites weren't built for them, weren't built for humans. So we need to help them find their way and navigate and consume our websites. So we just gathered a bunch of standards. Some of them are here for quite some time. Some of them are still being drafted. And we gathered them and we built this tool. And then we use the tool to scan like the top half million domains to check the current state of the web. And we also present that data in Radar in the AI Insights section. So feel free to check it out. It's updated weekly. And as we can see, some standards like Robot CXC and Sitemaps that are here for some time, they are the most adopted ones. We are starting to see adoption of some more recent standards like markdown negotiation, content signals, but there are still low adoption on the other standards. So we hope that our tool can help increase the adoption of those. Yeah, I thought the data was super interesting just to show how early it is. You know, at Cloudflare, we've obviously been working on AI products for a couple of years now. And sometimes it feels, you know, you go on the Internet, it feels like everybody's kind of already built 10 MCP servers for whatever they're doing. Everybody's already agent-friendly. But when you look at the broader Internet, there's so much work to do. And, you know, we've been doing this for a while. We work on so many different standards at Cloudflare that we're excited to kind of see all this stuff grow from here. So maybe let's go through some of the categories here. We kind of laid out four different things, discoverability, content, bot access control, and then a bucket around kind of how agents discover APIs and end up using them. Maybe let's start with discoverability. Andrey, you mentioned robots.txt and sitemaps. Those are things that most people are probably familiar with if they've been building websites. But there's another one here that I think most people haven't heard of with link headers. Do you want to tell people what that is? Yep. So first we need to make sure that agents can navigate through our websites. And that's what the standards in this discoverability section are for. So sitemaps is probably the easiest way to do it. So we just expose like a map, basically a list of all the paths in our website. But you can also use link response headers. So instead of letting the agent or the crawler parse our HTML to try to find links to useful resources like our API catalog or developer docs, we can surface the links directly in the response header. And then the agent can just read the header and it doesn't need to parse any HTML at all. Amazing. I'm really excited about some of these RFCs that people have been working on for many years now. And we've been kind of digging them up and going like, wait, somebody built this for something else that was unrelated to agents, but we think it can really help agents get the context that they need. We maybe mentioned Markdown at the start of this, the next one up was content, and we've been making a big push here on Markdown. Why does Markdown matter? I've heard this thing about LLMs at TXT. What's the difference here? Yes. So if agents can navigate through our website, we also need to make sure that they can also consume our content. And usually when an agent just fetches some resource from our website, it will get a wall of HTML, style sheets, JavaScript bundles that it doesn't care about. So it only cares about the actual content. And Markdown is the preferred content for agents and LLMs. So Markdown keeps the structure like other formats like HTML, but it uses much less tokens. So it's the preferred format. So before we used LLMs at TXT, which was kind of a new standard, like one year ago or two years ago, which was like a readme, but for your website. So it's just a file that you can expose and tell important things about your website, link to important resources, important pages. And Markdown content negotiation goes a step further. So instead of having to maintain this LLMs.txt file, Markdown content negotiation works dynamically. So whenever an agent fetches a resource from our website, it can prefer, say that it would prefer Markdown. And it does that by setting the accept header in the request. And then either the origin server or a middleware like Cloudflare can convert the content type of the resource to Markdown and the agent will receive Markdown. So this is very efficient because it uses less tokens and it's great. Here at Cloudflare, you can just enable it with a single toggle. So that's great. Yeah, it's really cool to be able to just turn this on and like everything works better for agents. So the next one up is bot access control. And there's a couple different things here. Some of them connect back to robots.txt. But there's things that you could do in robots.txt that are maybe specific to controlling what AI tools can access from your site. Tell people a little bit more about how that works. So robots.txt is a standard that's here since I think 1994, I think. So it's pretty old. And yeah, basically it allows you to expose the access controls for your website. So you can define which bots can crawl it and which you can be specific down to the path. And now there are new standards, for example, content signals that allows you to define the AI usage of your content. So if you want your content to be used for AI search, but not for AI training, you can do it with content signals. So it's just a directive that you can add to the robots.txt file. So I feel that these new types of standards is great because we are just improving a really already used standard like robots.txt. So it's a nice one. Awesome. Well, there's this next section up in what this tool will audit for you. That is near and dear to my heart because it's where it gets extra nerdy. We've got a lot of different standards in this bucket around how agents discover APIs, how they authenticate, etc. I've been hitting Andre with a couple questions, so I'll explain maybe one or two. A couple that I'm excited about here, we build our own skills for developers who are building in Cloudflare. And we publish these, we make them available, we encourage people to install them when they're getting going with our platform. But we looked at this agent skill standard and we said, wait, there isn't really a way for somebody to give a URL to an agent and have the agent on its own go off and discover that that site actually makes available and publishes different agent skills. And so a couple of people on our side, Matt Silverlock at Cloudflare proposed a kind of addition to the agent skill spec where you can check, an agent can go off and check at a well-known URL to understand what agent skills a given URL publishes or a given domain publishes. And we're really excited about this. Just think about the workflow that so many developers have, right? Like you're trying out a new tool for the first time, you plug in the docs, how does the agent go off and know? There's these awesome skills that I can lean on to do this particular task. But there's a bunch of other things here, especially around like API discovery. Andrei, you were talking earlier about the other standard around kind of link discovery at the start. Well, there's kind of something else here around, you know, how a site can expose that it even has an open API spec or an API catalog. Yeah. So agents not only consume content from our website, but they can also interact with it, either by using APIs or MCP servers. And we can also expose files like the API catalog file, which is basically a catalog or a list of all the APIs that we support. For each one, we can also link to the status of the API documentation. You can also add the link to this API catalog using the link response header that we mentioned earlier. As I mentioned, we can also expose an MCP server. And there's also a file that we can host, which is the MCP server card. It's still being proposed, but it's a good idea because with a single file, we can let agents know how to find our MCP server, the capability, the different tools to authenticate. So it's also a really important file that they can expose. Yeah. And you were talking about MCP servers. One of the things that I think a lot of people don't realize about MCP is that there's this addition to the MCP spec that adds server cards, where if you publish an MCP server, you can have a well-known URI that exposes to agents that, hey, this site makes an MCP server available and here's how to access it. This is something that David, who's one of the co-creators of MCP, has been working on for a while. I think it's part of the draft spec. If I have that wrong, somebody can correct me. But people can start publishing this right away. And one of the things that we're also working on on our side is trying to encourage different agents on the client side to adopt some of these protocols and different standards so that you're a user of an agent, you plug in a URL, and the agent side does the right thing and kind of abides by the standard. So a couple of other things that we got into as well with commerce. Andrei, you want to talk a little bit more about X .402. I know we've been making a big deal about that at Cloudflare. Yeah. So agents can also buy stuff. So you can give them your credit card and they can buy for content or buy tickets for airplane or something else. So there are a bunch of new standards to standardize this agentic commerce and X.402 is one of them. So the way that it works is that an agent tries to fetch a resource. So it does a request to our website, but it will receive a 402 status code, so payment required, because access to that resource requires payment. And in the response, it will also contain information about how the agent can pay for the content using stable coins. And then the agent does pay for it, retries the request, and then it can get access to the content. And Cloudflare launched, together with Coinbase, the X.402 foundation some weeks ago. So there's a cool blog post about it. So feel free to check it out. Yeah. We're really excited about site owners enabling this, but also just people trying to build entirely new types of businesses where maybe that business is really just an MCT server that accepts payments when it gets you know, and accepts payments via X.402. Or it's a website that, you know, exposes interesting context back to agents as a service, and agents can kind of pay on demand for that content. So there's so many different types of things that people can build with this, but we're really excited about where it's going to go from here. So that's the tool. You should go try it out. Isitagentready.com. We're going to be, you know, continually updating this over time. There's going to be new standards, probably going to be a lot more to add to this over time. We'll be tracking the adoption of these different standards on the Internet as we do just broadly at Cloudflare Radar already for all kinds of things on the Internet. So, you know, expect to hear more from us soon on the updates there. And yeah, anything else you wanted to add, Andrei? No, I think that's it. So feel free to use the tool. It takes like 20 or 30 seconds and improve your website. Oh, we didn't even mention the prompts. So you enter your website in this thing and you get a prompt back for your agent where you can fix your own website and have it go off and make these changes for you. Yeah. And how do we not mention that? It's also important to note that not all standards are applicable for every website. So if you only have a content website without an API or an MCP server, you don't have to expose API catalogs or MCP servers. If you don't have a commerce website, you don't need to implement X.402. So feel free to read the commentation, learn about the standards using the website and improve your score the best you can. Wonderful. Well, with that, thank you very much for the time and enjoy the rest of the week. Thank you.