How to Make Your Website Agent-Ready | Cloudflare Engineering Meetup Lisbon
最初直播時間:11月17日,上午9:00 - 上午9:30 [EST]
English
文字記錄 (測試版)
Okay, let's start. Hi everyone, I'm André Jesus, or Jesus for friends. I'm a systems engineer here at Cloudflare.
I mainly work on a product called Cloudflare Radar. And today I will talk about how you can make your website's agent ready.
So agents are everywhere.
So there are different types of agents like coding agents, like OpenCode, Clods, personal assistants that run 24-7 like OpenClaw and other types of agents.
And they are not just simple LLMs, they do a bunch of stuff. They call tools, they browse the web.
And they are probably already browsing the web and accessing your websites.
So I'm sure that most of you have interacted with an agent in the past week, or maybe today, or even in the last hour.
I know that I did because I have OpenCode running on the background, and hopefully it's not deploying things to production.
But the thing is that agents are everywhere. And the problem is that websites were built for humans and not for agents.
And while a human sees a beautiful UI with good navigation, animations, images, an agent just sees a wall of HTML style sheets and all the CSS a developer had to do just to center a div.
So the agent doesn't care about this.
He only wants to make a request, retrieve the content of your website.
So most of the time, like 80% of what's in the response, it's not actually useful.
So how do we fix this? Just like we learned to care about SEO or accessibility, now we also need to care about agent readiness.
So there are some standards that are here for some time, and new ones are being drafted.
And today we'll talk about some of them.
Yeah, so let's start. So first we need to help agents navigate through your website.
And probably the easiest way to do this is to use sitemaps.
So sitemaps are XML files that you can expose in your website. And as the name implies, it's a map of your website, so it lists all of your pages.
They are referenced in the robots.txt file.
You can have several of them. You can have sitemap indexes, which are sitemaps that point to other sitemaps.
And yeah, this is also a good way to increase your SEO, improve your SEO, because search engine crawlers like Googlebot, Bingbot, they also use sitemaps to crawl your website.
Another thing that you can do is to define link response headers. So instead of hiding links to important resources in your HTML, and crawlers have to parse the HTML and try to find the links, you can expose them directly in the response headers using the link header.
So when an agent makes a request and they just need to look at the headers, and you can expose a link to your MCP server or to your API catalog and so on.
So now agents can navigate through your website, but can they actually read the content of it?
So back to the problem I mentioned earlier, when an LLM does a request, it receives a bunch of stuff, files, HTML, CSS, fonts, whatever, and they only care about the content.
So one thing that you can do is to expose a llms.txt file.
It has a markdown content type, and basically it's like a readme, but for your website.
So you can mention important things about your website, important links, and so on.
So an agent just needs to fetch this file, and basically it knows the most important stuff about your website.
And there's also llms.full.txt, which is basically this file, but with all your website's content for crawlers that want to fetch everything at once.
And then there is a markdown content negotiation, which goes a step further.
So using the llms.txt files, every time you update your website, you need also to keep and maintain these files.
And with markdown content negotiation, you don't need to do it because this is dynamic.
So when agents are requesting content from your website, they can say, oh, I would prefer to receive the content in markdown.
And they do this by setting the accept header using the text markdown MIME type.
And then either the origin server or a middleware, like CloudStore, can parse this accept header.
And instead of returning HTML or other content type, they can convert the response to markdown and the agent will receive markdown.
So in Cloudflare, you can do this by just using a single toggle.
And markdown is also good because it keeps structure.
So it has headers and everything else that other formats have. But it also helps with the token reduction.
Yeah, so it's also cheaper, they use less tokens, and basically it's the preferred format for LLMs.
Yeah, now agents can navigate and can consume your website, but probably you don't want to let every bot or every agent to do it.
So not all crawlers are equal. Some of them are from companies that you know, and some of them are from startups that you never heard of.
So you can define rules about access control in your robots.txt file. So you can define access rules per bot, and you can go down to a specific path.
For example, here we are disallowing Googlebots to crawl that specific path.
Robots.txt is here for quite a long time, since 1994.
I did not exist back there.
And it's a really useful standard. Now content signals goes a step further.
So instead of just saying, so you can crawl or you can't crawl this path, you can also declare the user's preference.
So the AI user's preference of that content.
So for example, you can say that, okay, Googlebot, you can crawl this, my website, but only for AI search, you can't use this for training or AI inputs just for search.
So this is a good way to define the user's preference of your content.
Now, it's also important to note that robots.txt is not enforced.
So it's just a file that you host at your website. And we expect that well-behaved bots will follow it, but others don't.
So you need to enforce these rules at the network level.
And to do that, there's a product with Cloudflare called the AcroControl that leverage Cloudflare bot detection to block the requests from bots, specific bots.
Now, there's also, you can do this by using the user agent, but it's not a very good idea because anyone can spoof a user agent.
So I can just set the user agent either to Googlebot and bypass any user agent detection.
So there's a protocol or a standard called WebBot Auth that basically helps with this problem.
So the way that it works is that bot operators generate a key pair, so a public and a private key, they publish the public key in a well -known path, and then they use the private key to sign their bot requests.
And then you or a middleware like Cloudflare can identify the source of the request using that signature and make sure that that request came from that bot.
Yeah, this is also useful.
Now, agents can not only navigate through your website, but they can also interact with it, interact with your APIs.
But first they need to find them.
So you can host a file called the API catalog, which is basically a catalog of all of your APIs.
You can set the status of your API, the documentation of the API, and agents just need to fetch this file and read all the information to know where are your APIs.
Now, most APIs also require authentication using OAuth or other protocol, and you can also host files to help agents know how to authenticate.
For example, you can host this file with the endpoint for authorization, the different grant types, and so on.
So it's also useful to let agents know how to authenticate.
Now, and since we can talk about AI without mentioning MCP, so MCP stands for Model Context Protocol.
It's an open source standard to provide more context and access to tools and external systems to AI models.
And everyone has an MCP server. And if you have one, you can also host a JSON file, which is the MCP server card with information about your MCP server, and the URL, how to authenticate, the different tools that it supports, and so on.
So agents know how to use it. Yeah, and agents can also buy stuff. So they can buy content, access to content, buy pizza or airplane tickets.
You can give them your credit cards, which I don't recommend, but they can buy stuff.
And there are also commerce protocols to do it.
So X.402 is one of them. And the way that it works is that an agent tries to do a request, but the server responds with the 402 payment required.
So you can't access that resource because it requires payment.
And then in the server response, it also contains information about how you can pay for the content.
And then the agent can pay it using stable coins, which are cryptocurrencies that follow stable currency, like the US dollar.
And then the agent retries, and this time it gets access to the resources because it pays for them.
Yeah, so that's it. So a bunch of standards, and I didn't talk about all of them.
So some of them are proper RFCs. Some of them are just proposals. Some of them won't survive.
And you don't need to memorize all of this. Because Cloudflare launched a tool this Agents Week that basically lets you type any URL and get an instance of it across these standards.
And as you can see, my website is not very ready for agents.
But yeah, you can access the tool at isitagentready.com. And the tool not only tells you what's failing.
It also tells you how to fix it. It gives you links to documentation, the RFCs, and so on.
So it's a really cool tool. And we not only built the tool, but we also used it to check the current state of the web.
So to see if most websites were ready or not for AI agents.
So we scanned the top half a million domains.
You can get these lists through Cloudflare Radar. We have the domain ranking section, which is updated daily.
So we get these half a million domains.
We filter out categories that didn't matter for this test, like email servers, search engines, CDNs, malware, and so on.
And then we run a bulk scan. So the short answer is that the web is not very ready for it.
So this chart here shows the adoption of AI agent standards.
So this has the results of our bulk scan.
As you can see, standards that are here for quite some time, like robots.exe, sitemaps, they are the most adopted ones.
We are starting to see some adoption of recent standards, like Markdown Negotiation, Content Signals, LLMs.txt.
And then there are a bunch of standards that are not very adopted, either because they are recent.
And some of them are old and weren't adopted at all.
And another important thing to note is that not all standards are applicable for every website.
So if you have a website with just a content or a simple personal website, a blog, a portfolio, if you don't have an API, you probably don't need to define your API catalog.
Or if you don't have an MCP server, you don't need to expose MCP server cards because it doesn't make sense.
And so that's why most of those standards are not very adopted. So this will be available on the Cloudflare Radar in the AI Insights section.
It's not yet, but it will this week.
And yeah, so to conclude, we've been here before, but for SEO.
So in 2003, I was one year old and most websites weren't optimized for search engines.
So it took years for SEO to be a standard practice. And we are at the same point, but for agent readiness.
So my recommendation is go to IsItAgentReady .com, scan your website.
It only takes like 30 seconds and fix what you think makes sense to fix.
And yeah, so I think that's it. Thanks for your time. And I'm happy to take questions.
I'm Alex.
Thank you for the presentation. So robots.txt and Sitemap, I'm guessing those are also very useful for SEO and their maturity.
I will say. So question, for the website IsItAgentReady.com, do you have the MCP server so I can make my site ready following your rules?
Yep. The site has an MCP server. It also has an MCP server card.
The site is also agent ready. So you can use it and you send the MCP server or the API to your agents and help out.
And also the site has a feature, which is basically a button where you can copy all the fixes and just send them to your coding agent.
So it's also easy to do the fix. Cool. Thank you. Thanks. Done with that.
Anyone else? Hello. Thank you for the presentation. I was thinking about the SEO parallel you draw and I'm wondering if you think any of the present day technical properties that the AI is crawling, do you think it will evolve in the future if you're already seeing like alternatives of those robots and Sitemap, if there are other technical...
Because if you think about SEO, there are way more technical implementations of schema markups and stuff like this.
And do you foresee AI going in that direction too?
Yes, I think agent readiness is related to SEO because agents are bots and the SEO is also a set of standards for crawlers and bots.
So, yes. So some of those standards are kind of shared between SEO and agent readiness.
And some of them, for example, these ones for MCP are not very useful for SEO, but are useful for agents.
I'm not sure if I answered your question. Yeah. The question was more about if you foresee in your opinion, if this area of like how SEO evolved, do you think the agent readiness domain will evolve to include more tools in the future?
Yeah, I think so. Because there are... So some of these standards are like one month old.
So there are a bunch of standards and RFCs and drafts being created and agents are here to stay.
So probably there will be more standards and more stuff.
All right. Thank you. Yeah. I think there are questions. Yeah.
Thank you for the presentation. And my question is about this extension that you mentioned about robots.txt, about the intents that bot should expose, how they will use this data.
And what about the major bots that currently exist? Do they follow it?
Who follows? If I'm not mistaken, Google uses the same bot for training and for searching.
So it's just... So I'm not sure if we can call it a standard.
So it was proposed by Cloudflare like a year ago, some months ago.
And we expect good bots to start following it. But I don't have data about if it's followed or not.
Because we don't know that. So if Googlebot does a request, it retrieves the data, but we don't know what will be used for.
So ideally, they would respect that.
Yeah, but we don't know. Thank you. Hi.
Thanks for the presentation. I just wanted to make a comment, more of an experienced warning, if that makes sense.
SEO and a lot of the things that we are talking about how SEO is stabilized and their standards and all of that, it's because it works, right?
So one of the things that I would warn about is that when we are implementing a bunch of new things to make a site agent ready, sometimes we might disable some of the things that made SEO ready.
So I'm not saying that, of course, this makes sense, a lot of sense.
But we have to be aware that sometimes by implementing some of these, we'll actually implement, for example, we might use some more crawl budget from Google.
We might add new pages that don't make sense for Google to index, but are still going to be indexed.
So just as a warning, more than anything, that I've seen sites implement some of the agentic stuff, but then actually get penalized on Google.
Maybe even get some traffic and some visibility on LLMs, but then get penalized there.
So at some point, there's going to be a balance between the two things, because, of course, we need the two worlds to work together.
Yeah, that makes sense. Thanks. So this, we are in a very early stage of this.
So this is more of an experimentation phase of this, of this stuff.
Yeah. Yeah. Hi, thank you for your presentation.
My question is, is there like an order of specificity for these different standards?
So for example, with the new LLMs.tech .txt, in the case where you have rules in LLMs.txt and you have things in sitemaps, which one do the agents pick first?
What's the source of truth, the single source of truth?
So I think that depends on the agent. It depends on the agent purpose. So this is just some things that you can help them.
But I'm not sure of, yeah, it depends on the agent.
So there isn't a right answer. So the agent might fetch like the sitemaps if you ask it to crawl the website.
But then if you ask about the purpose of the website, it might just fetch the LLMs.txt.
So I think there's no clear answer.
So there's not a priority on the resource. So my question is more of semantics.
So you mentioned... Yeah, I think that's just because we have robots .txt, we have ads.txt, security.txt, and they just wanted to use .txt.
But yeah, Markdown is a better format. That's why. But LLMs.txt sounds better than LLMs.md, so.
I think that's it.
Cool. Yeah. Let's think back. Wait, we missed a guy. This one isn't...
No, it gets recorded this way. We want to hear. We're only supposed to launch this on Friday.
Use the microphone then. We're only supposed to ship this Friday.
But we discussed here, I think we're okay to do a demo of the website. You want a demo of the website?
Yeah, just run a scan for people to see. Can't go wrong.
No pressure. Well, yeah, it's not shipped yet, so it's okay to fail. Okay.
Now let's use my range and scan.
And yep, okay.
So you get the scorecard here. So this is a summary of your scan. And then you can, as I mentioned, you have the improve your score button, which is just a Markdown text that you can just copy and send to your LLM.
For the checks, you can check the audit details.
So the request it made, how the check works. And then for the failing test, you also have the goal and the issue, the skills that the site hosts, and also links to documentation about that check.
And yeah, that's it.
Yep. You are being penalized. I am. Score. But if you don't have API on your website, it doesn't make sense to be penalized.
Yes, it doesn't. We didn't find a perfect solution to do that.
So here you can also customize your scan. So for example, my website is more of a content website, so it's more blog and stuff.
And this will only run some checks.
And I'm also, I'm still failing at Markdown for Agents.
But yeah, so as I mentioned, some checks don't make sense for every website. Like commerce or API for my website don't make sense.
Yeah, we are still working on it.
Yep, that's it.
Thank you.
