From ranking to recommended: Answer engine optimization (AEO)

Hello, everyone. I'm Will, this is Ashish, and we're from the London office, and it's great to be here today to talk about Answer Engine Optimization, AEO. So today we're going to, I should say, we're both researchers. We've spent quite a lot of time thinking about this problem lately. And so today we want to present our sort of take on this space as a minute, as it's quite nascent. We want to share some interesting research we've done here and how it's helping shape our AEO products. Currently in a closed beta, but hopefully we're going to move out forward fairly soon and start getting more people using it. Cool. Right. So for years, being found online is typically a search engine problem, right? It's a ranking, and you want to be high in that ranking. And the whole concept of SEO, search engine optimization, was how to boost your presence in that list. And the tracking of this and how well you're performing is to do with things like impressions and click through. So this field is still very much active and it's not going anywhere anytime soon, but things are changing in the sense that now we have answer engines, right? So we have my digital assistants, we have things like ChatGPT, et cetera, Gemini, which you're having conversations with. And increasingly we're seeing that users are asking these assistants to compare, to explain, and to recommend. And what's really important about this in the distinction somewhat from SEO is the decision is therefore happening before any sort of ranking is taking place in search engines, right? There's no search happening prior to this decision being made. And interesting, the signals are different too. So it's not the same as these impressions and the click through, which we were previously looking at for SEO. So now we care about citations and mentions. And then not only citations and mentions, it gets more complicated because then we want to start looking at like semantic meaning, the intent behind these conversations, how you're being represented. So it's a very complex problem. And it gets even more complex when we then look more towards where the engine is heading and having agents go off and have these conversations with answer engines, right? And so at some point they'll hand off to the human, but at that point you don't really know what's occurred between those two, how your recommendation has evolved. And so I guess the take home here is SEO is fairly emergent. It's something which lots and lots of people are working towards and we're going to take you through basically the flow and how we're approaching it. But I think we need to consider three sort of cool things here, which is one earning consideration, right? We want to know that in high intent scenarios, you're being recommended. We also want to be able to see the competitive field. We want to know if you're being recommended alongside who you consider to be your competitors, whether or not they're being recommended ahead of you or why that is, or maybe you're being recommended in completely different sectors to one you consider yourself aligning to and why is that? And finally, and one of the most important aspects for us at Cloudflare is how can we connect that to outcomes, how can I share actionability here? And so that's something which Carver is very well positioned to do because we have a lot of these really interesting signals, which we're going to share with you today. So before we get into agents, I just wanted to spend a few minutes to talk about the network at large. So we analyzed Cloudflare radar between May of 2025 versus May of 2026, and we chose OpenAI for this analysis. So as you can see, there's been a 116% increase in the OpenAI search bot crawls. What this means is OpenAI has been increasing crawling to build out this search index. We also see a 32% increase in chat GPT user bots. These are essentially user initiated bots for folks like you and me to try and ask about questions of like, what's the local pizza shop? We also see only a 7% increase for the GPT bot, and these are bots trying to crawl the web to train the next flagship model. So what this tells us is that the trend is shifting. The trend is shifting from training to providing answers to agents. And this definitely puts a front and center. And I think while this is an exciting space and a time to be in, this is a really hard problem to solve. And the reason I think it's a really hard problem is because of these four gaps. And generally with AEO platforms, there are, you know, a few common patterns, a few common designs. You've got these synthetic prompts, you funnel these synthetic prompts through your agentic harness of choice, if it's OpenAI, Anthropic, Gemini, Perplexity even. And then you kind of roll up these scores into some understanding of where your brand is being visible. The thing with these AEO platforms is that it kind of exposes these four gaps. So the first one here is like for prompt bias. So prompt bias is essentially when you're changing wording, keywords, tone, and style of the prompt, and that actually greatly affects the responses coming out of these agents. When you want to control for prompt bias, you end up actually probing the same prompt n number of times, three, five, even eight times. And that actually kind of results in the second gap of probe cost. So a lot of customers are trying to figure out what's the balance between identifying a good prompt and then probing the right number of times so they can have a semblance of a grounded AEO score. Gap number three is the action gap. A lot of the times AEO scores are after the fact. Customers don't know if their AEO score is because there's a gap in the content or because there's some sort of a bot positioning issue. I think providing intelligence to understand where that gap is and providing actionable intelligence is the name of the hour. The fourth most important gap for us is the missing ground truth gap. And I think this is the one that we as Cloudflow want to index heavily on. We want to leverage the edge, the patterns we see in terms of who's crawling whom, what the referrals look like, where do active human crawls happen versus agent crawls. Understanding those patterns and weaving them into our product is something we're definitely deeply interested in. I'll hand it back to Will to talk about the proposed solution. So what does the complete answer look like? We think, based on our research we're going to show you in the coming slides, that we should treat it as a cyclical process with always three key steps, but the whole thing should be a feedback loop. And we should iterate, we should be able to actually see incremental progress over time. So to start off with, we want to measure what assistants say. So these are different assistants, not just one. We want to make sure that we're covering what assistants people tend to use. So that's like Gemini, GPC, Avrobi, et cetera, Claude. But it's important to track the mentions, it's important to track citations, work out a share of the model, tease out other relationships we can do by taking all these in parallel. But this is always going to be a sample of synthetic answers. That's not to say that you can also augment that with additional curated prompts from sets you purchase. They're very valuable and lots of work goes into that, but at the end of the day, they're telling you what an assistant might say and not what real conversations are being had by potential customers. So with that in mind, we need to actually observe what really happens. And that's where the signals in our network come in. Essentially, we want to know what's fetching your content and for what reason. And we also want to know the actual transfer between an agent and a human. So when someone, an AI referral, essentially, when an agent refers the human and the human clicks through and then engages with your domain. And so that's the first part of your ground truth. So that's number two. And number three is acting on it. We want to make sure that we can actually put things in place to improve upon these things. So once we start noticing trends, we want to then see, can we impact those? Can we actually get you recommended at an earlier stage? Can we get you so that the click through is actually happening? And that's something else we're going to talk to about in a bit. But I think essentially, if we go back to what Matthew was saying yesterday in terms of the deli shop owner, them knowing your names, or the teenager building their first site, and wanting to help them with that. Essentially, the Internet works when it knows you and helps you to get things done. And it's not just about your rank. So we need to make sure that we're treating this as a much more sort of holistic process. So we're going to build to idea. And that's what we've been researching. And yeah, we're going to share some of the findings now. We've conducted a series of experiments with the intention of identifying how to best target these problems. So over here is basically our experiment where we conducted, where we tried to elicit the number of times Cloudflare was mentioned or cited with these 50 prompt samples being sampled eight times across each of these three vendors. The intention of this is to look at, does it matter who you ask the question to? So if you look at the mentions for OpenAI versus Google versus Anthropic, that's the left textured bar, we see that for OpenAI, Cloudflare is being mentioned 74% of the time, but it's cited just nearly as half that with 47%. With Anthropic, it's 57% on the mentions, and then it's being cited 11%. So and then more importantly, with Google, it's been mentioned 90% of the time, but it's been cited 19% of the time. So the point here is, even with the small sample size of 50 prompts being sampled eight times, each of these vendors have their own logic in terms of mentioning and citing. So it's really important to understand with these AEO platforms and scores, how you're reasoning about what that overall rolled up score looks like and how to actually understand the nuances for each of these vendors. Along with understanding who you're asking these prompts to, the next part of the puzzle is what's being asked. So over here, we looked at understanding different prompt styles. So we tried to understand how much do the AEO citations and mentions change depending upon what's being asked. So bottom up, if you look at prompts that are more conversational, versus the plain question, we see the citations go from one to four. And then if it's slightly more keyword focused or comparative, or even ranked lists, the number of citations jump up. So the intention here, again, is very much emphasizing the way you phrase the prompts really matter. And it kind of makes it front and center for these synthetic panels we're creating. How do we actually put one together that is one grounded, but also more expansive so that we understand what the user has been asking? How do we make sure that it handles all of these different prompt varieties? And more importantly, handling cost, because the more varieties you have, the more you have to account for the different probe variations for each of these vendors and their costs. Thanks. So yeah, the previous two slides that she shared with us, prompt bias is a thing, right? And we wanted to see what we could do in terms of mediating this, in the sense that we want to reduce costs, but also we want to reduce noise, right? We want to boost that signal coming out of these. So what we've been playing around with is using decision models, classifiers. Essentially, we've got a huge amount of data now, we're collecting a lot of synthetic prompts, our own sort of in -house prompts as well. And so given that data source, but also the data source of our knowledge graph, which we've been building as part of the AO product, which looks at sort of, yeah, not only citation and mention patterns, but also sort of co -linkages within these networks, sync, co -citation, looking at labeling by industry, etc. We have this sort of wealth of knowledge, which you can start to tie together and actually start to use to make things run better in the AO space. And so as an example here, what we have is we fine-tuned a decision model. And then we realized that actually you don't need a decision model of that scale, right? So you get still fairly decent accuracy, but if you just distill it down into a much smaller classical machine learning model, right? And you can run that on the edge. So that means essentially you can filter your prompts for essentially zero cost as you're using them. And by filtering, I mean, we can train classifiers for different purposes. This one example here is looking at how we can boost for productive prompts. So there's prompts which tend to give you a result, which will cite you or a competitor. So, and it lies with your industry area, but we can also boost it in different ways. Like we have classifiers now, which are running to boost certain semantic intent. The point being that actually we don't need to spend a huge amount of money on probes. We don't need to have these really expansive prompt sets to get meaningful answers. So this is reducing the cost massively, but still getting the signal and it's removing a lot of the noise associated with bad prompting, which is quite a bit of a bit of a problem. And so the intention here is that we're going to use this on real AO prompts. And so that we can then make that sort of, if we go back to the earlier slides with the synthetic prompts, these other things, we can make that more robust and run at scale. So that's all covering the first two gaps, which she identified earlier around the prompt bias and prompt cost, the probing cost. The other things we need to start tackling though, is actually how we do the observability and the actionability here. And so I want to talk a bit about signals from a network, because we really want to know what people do with the information recommendations, which assistants would have told them. Because an assistant, and it's an answer style to have a model already knows inherently, right? So they've learned that from past crawls. What we tend to find from the synthetic side of things, the prompting side of things, is it's interrogating the model's core knowledge, but it's not really looking at the latter steps in the answer agent process. So what we need there is the signal which comes from the network. So I guess what I want to take home from this big here is that if we normalize the site size, which is a massive co-bounder in terms of the signal coming out for AAR, because the larger your site, the more likely it is to get referred to an answer engine. And that's, it's the driving signal. But then we have these other signals emerge when you account for that. And we actually find that the mentions and the citations actually have very little impact on whether or not you'll get an AI referral from an answer engine. And actually the things driving that are coming from the crawls, they're coming from the training bots, and they're coming from the user initiated fetches. And so what this data is telling us is those are things we need to target in terms of actionability. And actually just going to talk a bit more about. So I think the key here is that user initiated fetches have a correlation with referrals. And I think our next part of the experiment was to try and analyze what happens if with user initiated bots, if your firewall is misconfigured. We tried to analyze what are the different permutations and combinations where bot positioning affects user initiated bots. So the top row here is we sort of assembled this list of 27 test sites. And each of these test sites had custom paragraphs where the LLM had to go and fetch them so that they could answer the question because all of them were made up facts. So the intention was if they can read if there's a well configured robots.txt, if there's a well configured firewall, how do Anthropic and OpenAI crawl the site and actually get the responses back? If things are well configured, then in almost all scenarios, they're able to crawl effectively. In scenario two is when your firewall is misconfigured, and the robots.txt is configured correctly. So over here, as expected, you would get a firewall block with the 403. I think the more interesting piece here is scenario three, when we noticed that with Anthropic and OpenAI, when both of the robots.txt block the user agent bots, we noticed that Anthropic actually doesn't crawl your site because it adheres to the robots.txt. But for OpenAI, it actually crawls. And this is part of like how they're positioned. With OpenAI, their positioning is that user initiated bots are an extension of the user. So they're sort of ignoring the robots.txt. So the intention of this slide is to showcase that with just two different agenda vendors, there's so many permutations of how robots.txt, firewall, network on the edge in terms of crawls, referrals span out. And it's really important for us to actually make these insights actionable with like call to actions that are more easier to digest and once they can actually flow into the UI. So we're just going to quickly switch gears to a demo. So for the interest of time, I'm just going to sort of like provide a layout overview of what our AO page looks like. And for those folks that want to actually do a deep dive, I'm happy to answer some questions. And we're also going to be in the speaker lounge today. So the general layout for our UI comprises of like actionable intelligence in the top section. So like I mentioned, if you happen to have bots that are misconfigured, in which in this case, we have an example of the chat GPT bot, specifically, we have the search quarter being misconfigured, you have the option of reviewing that particular robots.txt, you can enable or disallow or allow them. Or if the block is intentional, you can actually keep this policy and then you can train our system to understand what your behavior looks like. Furthermore, with the 403s and the firewall blocks, we also capture what's happening on the edge. So in this particular example, we can see that 20 27% of visits are refused. And the intention here is to give signals back of what's happening on the edge so we can act on it. And then in terms of rough layout, we along with identifying configurations for industry category and defining your competitors. Once we have this information of who you are, and what does your narrative look like, we also capture performance metrics for citation and mention. We also perform share voice analysis. So we can identify where you stack up against your chosen competitors. And we're also providing signals from the edge in terms of the crawler and referral operator by operators. And the intention is we can show you either on a seven day average or a 30 day average, like what do your crawler operators look like. And we're also visualizing that in a time series format to give you guys better insights of where the trends are. Is it improving? Is it dipping? We also have an AI visibility tab. And I think this tab particularly addresses some of the issues or gaps with prompt bias where we're actually capturing all of the AO metrics by vendor. So we can perform deep dives by different prompt categories and also look at each individual prompt and their responses that come out. And we're also capturing citations as a way for us to understand what are those sites that actually have the most amount of clicks that are occurring through them. If you can hop back to the slide, please. Thank you guys for attending. We'd love to ask some questions. Go for it. Just gonna give you a mic here. Is it on now? Oh, good. Okay. I noticed at the bottom of most of the slides, it's something about using API as opposed to like the actual consumer app. Is that what is also being experienced in the AO product itself? And is there a reason? I guess it's a multiple, like why? What are the consequences of that? And what is the customer feedback? Yeah, it's a good question. And yeah, it's something. These are more of the research slides, which we were using the API for. And it's very good question. The product now is moving towards using the consumer app because it needs to mirror what the consumers are doing. Interestingly, as well, we're also finding that a lot of the API parameters aren't being respected quite often around location and things like that. And particularly when you've got a regional business, and you want to be represented within that region, things like that really matter. And so the API is a bit limiting for that. What we're actually finding, we didn't present the data on these at the minute, but the pre-pending certain sort of like context into these is much better than actually using the APIs in terms of getting domain locality and all that sort of thing. So yeah, the app is going to be released, it's going to be much more focused on the consumer driven side. Yeah, good question. I would also be curious if you have a hypothesis, untested, I suppose, about how the memory in the consumer apps would affect this. So for example, if I'm over years having conversations with Claude that imply that I like Cloudflare, it seems like it would be more likely to recommend Cloudflare at the top of those. Any thoughts on how that could be measured? That's a really interesting problem when we've discussed. We're trying to do it at the minute by setting up my personas and trying to model it as like different consumer types. It's a very hard problem though, because that's like an infinite space, right? So we need to work out a way to sort of approximate the consumer pools. And so I think this is going to be a case of having to do some quite extensive data science prior to rolling something out like that out. But at the minute, everything's going to be quite generalized. Question there. What is the timeline to release it? That looks exciting. You guys killed half of startups in San Francisco. Good question. I'm going to hand it off to David who's in our product team to answer that. We just made researchers. I went to pawn that one off, guys. I really appreciate it. Yeah, so as you can see, we're kind of in the research stage. And right now we're in closed beta. So we're working with a few partners, mostly Cloudflare existing customers, because everything builds on top of the CDN data. We don't want to jump straight into picking random prompts and seeing if you're in those answers. And so everything starts with what is the agent bot and crawler traffic, then we'll use that to formulate what synthetic prompts we pick from there and make sure they're representative. But to get to the question, it's coming. We're in closed beta at the moment, probably expanding that to a broader pool within the next, I'd say, month or so. And so probably end of this year, early next as well. Hi, you know, in one of your slides, you mentioned that we add robots.txt, then it starts showing up, right? So does that mean, like, now LLM functionality is getting overridden? Because if there's no robot.txt, then it's not coming up. So robots.txt is, if it doesn't exist, then bots are implicitly allowed. So I think the nuance there is with user bots specifically. With user bots, Anthropic is honoring robots.txt even for user -initiated bots. But OpenAI doesn't honor it because their intention is if it's user -initiated, then it's an extension of the user. So I think there's slight nuances there between robots.txt positioning generally means implicitly allow if it doesn't exist. Is it similar to like, you know, putting the Google tag and then they have start appearing it? Sorry, repeat the question. Is it similar to like, you know, putting the Google tag, then they are appearing in the, like, Google search? It's, I think the paradigm is like slightly shifted now. It's really about getting the configuration right about how you want to be visible for brands. I think it's all about understanding if you want to be implicitly allow or explicitly disallow for certain bots. For example, I know new sites tend to explicitly disallow certain sites. So I think that it's similar to the idea for tags. But I think it's a slightly more AO view. Thank you for that. Thank you. Really cool research. Thank you for sharing that. I have a question to follow up on that robots.txt. Is there a reason that you're benchmarking against robots .txt rather than LLMS.txt? Is it because the adoption level is not there yet? Or, you know, or do we expect, in other words, do we expect some kind of apples to apples way and crawlers when it comes to LLMS.txt investment? Good question. So I think from our analysis so far, robots.txt is the most widely adopted. So the intention for this analysis was to see across the most widely adopted configuration, how do different agents behave? That was the main reason for choosing robots.txt. I think LLMS.txt is also partially used. However, I think robots.txt is the primary configuration tool. Cool. Thank you. So there's been a lot said that really good SEO is also really good AEO. I'm wondering if you've done any analysis that shows that sort of just having really good SEO means you have really good AEO. That's a good question. I think we actually started off as an SEO shop back in the way back in the research days. So I think there's definitely parallels between SEO and AEO. So a good example was the original paper that inspired SEO and AEO specifically was the parallels between page rank in the SEO industry and the idea for like prominence in the AEO space. Where you're trying to understand, for example, within a given LLM response, how many of the tokens in the response can be attributed to a certain brand. So there's definitely parallels that you can draw between SEO and AEO. I think the intention there is if you want to try and move towards the agendic flow is where the gaps between SEO and AEO becomes a lot more prevalent. That's a good question. And there's one behind you as well. Did you analyze the frequency of visibility based upon schema markup, whether there's a markdown text in the page, other kinds of parameters like that? That's on the robot. So basically we have a product you can use now, Agent Readiness, and a lot of those things are actionable items such as markdown, etc. What we're aiming to tease out now is given those as a set of variables. So if we go back to one of those charts I showed before, when we had the sort of the crawl statistics and the prompts and mentions, that was just a subset of all the different variables we're looking to interrogate. They all appear and they all have a signal. The ones we pursued in the start to present here were the crawls because it was the strongest link. But yes, we want to go and we want to look at those other things. And then it becomes a much more sort of complex analysis in terms of the covariance between each of those. So yeah, it's on the roadmap and definitely anything which is actionable, particularly in Agent Readiness, which you can then go and suggest, you go and you change it in terms of the Cloudflare dash. That's exactly what we want because that goes into the feedback loop, which then allows you to track. And then you'll see on the charts we showed when those points in time when you initiate in those changes and hopefully you'll see your trends increase. And that's what we're finding so far in the initial studies. Yeah. We're interested in one more question up here. Thank you. In your research, are you able to or have any insights around which search indexes are being queried? It used to be that Google was the only game in town and Bing. We love you, Bing. But now it seems like everybody's trying to build their own index, including Cloudflare. So it's a great question. I think we could say yes, and then we might we might publish a blog post around that shortly. But we can definitely tease that information out. Yeah. And you're right that in terms of index use, it's a really interesting area. And working out what all these agents have built upon is really interesting and something we're working on. So yes, great question. And yes, is the answer. It's a good watch this space, snap cancer. Yeah. Great. And we are on the speaker thing. Yeah, we got one here as well. If you want to sit in, sit in one more. Okay. Hey, I'm just curious in terms of the prompt bias, because the prompt can come from the user and the user prompt can go to different agents. Sometimes agent will rewrite those prompts. And in order to actually intervene this part, I don't think either Cloudflare or as the source provider have any capability to intervene those two steps. So how are you going to reduce the variance in terms of the prompt bias? So are you asking in terms of how real people are using answer agents? Is that what you're asking in terms of, can we? No. So in the four problems you laid out, so the problem number one is people might ask various different, asking various different ways. Yeah. So that prompt can come from user, can also come from agent rewritten prompts. So how does Cloudflare actually be able to tackle that part or maybe generalize that part? Yeah, generalization is tricky because at one hand you want to generalize it and it works at scale because this will be immensely costly if we wanted to basically tailor it to every individual persona and agent, etc. And it depends a lot on their context. So to begin with, we're looking at keeping it generalizable, but we need to come up with a way that we can, like I said before, these personas where we can actually interrogate it based on the user group who are particularly interested in a certain thing. So it's not just about your domain representation in answer to a question, but it's also linking that with the certain users and how you appear to those users. So, yeah, it's a good question. We're primarily trying to do this based on small synthetic curated prompt sets to make this work at scale. But we realized that we're going to have to then probably augment that with some sort of additional prompting. And so what we're looking at at the minute is how we can sort of build that iteratively and maybe start with like a core generalizable prompting set. And then maybe we can tailor that and see how that looks with these different use groups. But also what we need is the user research, like what kind of users do we want to model? What domains are we focusing on here? I think we need to narrow down the problem space a little bit, which is what we'll probably end up doing as we start iterating on this first. Like we can't just unleash an AIO product which works for every single domain and every single sort of user group you're targeting. Yeah, I think so. Ashish, do you have a different thought on that one? I think the intention for us is to try and look at our traffic on the edge. And I think we do have products within Cloudflare that do have access to human prompts. I'm just going to let David speak about the roadmap there. But I think there's potential for us to understand real human prompts, synthetic prompts, and then build this cross-sectional funnel so we can actually have a good quality prompt set that can help customers. Yeah, but we can talk about this more in the speaker session. Yeah, thank you.