New use cases for Cloudflare’s AI Agent

Hi everybody, thank you for being here. My name is AJ Gerstenhaber. I'm joined by Harsh Saxena and Ayush Kumar. We are all product managers who are working on various AI security opportunities within Cloudflare. I'm going to be moderating a brief conversation today between Harsh and Ayush to talk about features that they're releasing as part of AI Week. If you're not familiar with AI Week, you can check out our blog, but it is a brand new innovation week where Cloudflare is going to be announcing tons of holistic improvements across our platform, both for developers and builders and security practitioners, all about interacting with, using and securing AI. So Harsh, I think we'll start with you. You have some exciting advancements related to Cloudy, our AI agent. Do you want to get started? Thanks AJ. Hey everyone, I'm Harsh. I'm a product manager within Cloudflare's application security team and overseeing advancements of Cloudy and ensuring how we can get the most value out of it. With that in mind, so for security teams, the most important thing is reducing time to mitigation. Once you identify that something is wrong or you want to get to basically stop an attack as quickly as possible. And with that in mind, we have released a bunch of Cloudy features during a security week earlier in the year, including helping you create better rules, using a natural language interface, and also helping you understand your already existing rules to quickly identify if something's going wrong. Now we're taking that one step further and helping you essentially investigate your logs and your traffic in more detail, in a more conversational manner. So traditionally, when you see an attack, what you would do is go into security analytics. You would filter your traffic and try and understand what's going wrong. But that assumes that you know what to look for, that you know what kind of attack it is, you know what's happening in your own traffic. And for lean security teams, that is often a disaster. I can spell disaster because you often have a very limited window within which you need to take action. So enter Cloudy. What we did was we created a smart AI agent and made it conversational so you can talk to it like you would talk to any of the other chat-based agents that are out there. And what that does is it will start with some context. It will already have context of the attack that is taking place. And even before you ask it anything, it will tell you what has happened. And it will give you actionable recommendations or steps that you can follow to investigate. So ideally, you could go from, hey, I see an attack, to, hey, this is a rule I should create without even typing anything, just by following recommended actions. Now, we have already rolled this out to a bunch of our customers, and the engagement has been great. People are already talking to Cloudy, using it to create rules, investigating traffic. But then we realized that this is not enough. We need to go one step further. It is great to analyze your own blogs and your own traffic. But Cloudflare, because it is Cloudflare, also has access to a wide area of attack that happens all over this network. And this is where we get into Cloudflow once and our Threat Events Platform. So our Threat Events Platform basically has data points and indicators of compromise across spanning our entire network. So you can see your compromised devices, volumetric redos attack activities, and cybercrime groups. But again, you run into the same problem, because just as you have tremendous data on your own network, Cloudflare network, because of the scale, also has a lot of data, and we have a lot of information that you can sift through. So we thought, what if we apply the same logic and make it accessible via Cloudy, and you can ask and create it in a more conversational manner? So that is something that we have built as well. So now you could basically go to Cloudy and ask questions like, who is targeting my industry vertical? Or who is targeting my most credit country? And you can ask questions like, how has a specific adversary progressed across cyber kill chain over time? And all of this is available within a very user-friendly chat -based interface. And so just to make sure I'm understanding correctly, users can query Cloudflare's threat intelligence directly, basically? Yes, exactly. You could query what events that we have tagged. You can query our entire threat event platform. So basically, you could go in and query a specific IP, and it can tell you information on what ASIN it belongs to. If it's tagged as malicious, it is part of a VPN list. And if it is, then you can now, you're not just restricted to querying certain data at one place. In the same chat interface, you can say, hey, do I have any rules that leverage this? And Cloudy will tell you you do not. And then you can create, in the same chat conversation, you can go on and create a rule for it. So this just makes your life on the dash simpler. And that's the goal with Cloudy, to reduce the friction on dashboard as much as possible and connect the disparate access points you have to different kinds of data sources. It sounds like that would actually take customers from attack detection or threat detection to remediation a lot faster. Have you seen anything like that? Yes. So we have been observing user journeys in terms of how quickly an average user would, or a SOC analyst, would go from threat detection to mitigation. So on average, a detection is a specific kind of error. Normally, if you take an analyst around five minutes to get there, we have seen people get to identification to mitigation in a matter of two minutes. And for lean security teams, that is a massive, massive jump. That's huge. At the top of the call, you were just talking about the window of opportunity to create a mitigation once you've detected an attack. It sounds like that's really impactful. That's really cool. Very cool. Is there anything else that you want to share with us before we move on to hear about email security from Ayush? No, I think I would recommend that this feature is now available to all our plans and all our Cloud Force One customers. I would recommend everyone to go and try it out. And as always, your feedback is paramount in helping us a lot this further. So Cloudy, our agent, is available for all plans, all platforms, and then the engagement with Cloud Force One is available where customers have their threat intelligence subscription. Is that right? Exactly. So the way we have built Cloudy is it only has access to things the user engaging with it has access to. So you don't have to worry about, oh, is it going to accidentally query something users should not do? So if you have access to certain platforms, you can query from anywhere. And for analytics, you don't have to worry about, oh, do I have access to bot management or do I have access to attack scores? If you do, you can leverage it. And if you do now, Cloudy will tell you that it cannot do certain things for you, but it will redirect you to the documentation or other helpful links so that you can learn further. That's super, super cool. Thank you, Harsh. Congrats on that. Very exciting. Ayush, good to see you. Thank you for being here. I believe you have some exciting advancements related to email security detections. You want to tell us a little bit about that? Yeah, absolutely. So to kind of piggyback on the work that Harsh and their team have done with Cloudy, what we wanted to do was expanded or, I guess, extended to email security. So one of the biggest gaps that we saw in when talking to customers was that, look, we have a plethora of ML models, right? They look at sentiment. They may look at ASN reputation. But the way that that gets distilled down in our UI is kind of this one line kind of detection name, right? So it can be quite difficult to understand, especially if you don't know that nomenclature. And we noticed that a lot of teams, especially as you're having kind of this reduction or kind of this focus on more junior SOC analysts, like it really had a steep learning curve to understand, like, what is church mouse, right? That's our sentiment model or a group of sentiment models. And what does it mean for a BC detection? Well, you know, if you string these words along for us, it was pretty clear what that detection was doing or looking at. But for some of our customers, that definitely there was a big gap between that. So we wanted to do was see if we could figure out a way to have Cloudy in a more, you know, human-readable way explain what is happening within the message. And so, you know, that was our goal. And what we are launching in a closed beta is our kind of first iteration of a system that is able to go through and kind of take the findings that we're finding within emails and output them and not just this like singular, you know, one long line of strings of every single model that kicked off, but rather a more readable way of saying, okay, you know, in this message, we found that the sentiment is financial. It's also a first-time sender. And these are all the models that have kicked off. And this is kind of why we found it to be malicious or spam or bulk or whatever our disposition was. Very cool. That's awesome. So it sounds like it sounds like both between you and Harsh, there's a lot of sentiment about the steep learning curves with kind of complex technologies and even somewhere like Cloudflare where we pride ourselves in being very usable. Some of these things are just difficult to learn. Would you say that like in teams with junior stock analysts engaging with email security, that this is having any impact on their ability to ramp up and be effective and kind of understand the lay of the land? I think the one of the things that we do really well at Cloudflare is we treat ourselves as our own customer and we use our own products. And so I think the first step of this was to actually what we call customer zero, which is us internally is actually launching this internally. So this is, you know, on our dashboard, we ran this past our SOC teams who were like, hey, this is, this is better, right? This is, does this make more sense as you're going through investigation flow? And again, because we're launching in a closed beta, I don't have like the full stats, but what I can say anecdotally from our internal team and testing is that it did improve their ability to like contextualize what they're seeing and then obviously be able to take quick action quicker or be able to come to a conclusion a lot quicker on what they need to do with the message itself. Very, very cool. So again, faster time to mitigation just across the platform. That's great. So I know this has been, this has been a challenging development cycle because a lot of what we do involves using our own products, right? And working on workers AI and, and other AI gateway and auto rag. Has your team, what does your team run into when working through some of those things? Yeah, that's a great question. So I think the reason we're, we're calling this a closed beta is because in our first iteration of kind of using kind of the base level of cloudy was that we found that it, it hallucinated quite a bit. And that was to be expected because you're taking these large language models that, you know, are trying to correlate what the next word is going to be based on the words that you're feeding in. And oftentimes what we have like internal like nomenclature around models, it would not understand that that is a model, right? So for example, we have something called Thor, right? That's one of our models. It looks at various other parts of the message. And, but if you're an LLM, right? Like Thor is, it's a, it's a God right in, in literature. So the hallucination rates were quite high and what we wanted to make sure is, is twofold, right? One, it has to make sense. Like we can't just say, you know, you're looking at this message and our model Thor kicked off and then you suddenly get a whole blurb about, you know, the, this guy. Norse mythology. Right, exactly. And the second part is we also wanted to be very tight that the hallucination rates should be very minimal because we don't want the, the, or for cloudy to essentially come to a conclusion like, Hey, actually this isn't malicious, right? And if you're maybe a senior SOC analyst, you may be able to catch those clues and say, okay, wait, no, something was really wrong about that output. But again, if you're, if you're one of the newer SOC analysts that's been hired and are going through it, you may blindly take that as at face value. So what we wanted to do is, and what we are doing is kind of really adding, you know, structure around it where it can only access and, you know, query our detections. It knows the guide rails of what is this model doing? So then it knows how to contextualize itself as it's doing those summaries. And so really this closed beta for us is to one, make sure as we are slowly rolling this out, that we have a good understanding of the hallucinations and, you know, certain detections may have certain guide rails that we still need to put in so that it gets better. Because the last thing that we want to do is cause an incident in a company because someone was like, yeah, I trusted what cloudy told me. And it did hallucinate, right? So that's kind of our internal metric of success. And that's kind of why aren't rolling this out more ubiquitously across our customer base. No, that makes sense. That sounds like a high important bar to have. It sounds like in most situations, and we know this about all of our usage of AI and the way that we encourage our customers, but you should trust but verify, right? Generative AI models are great, but it's always important for a human to double check the work before you put it out into production. Either way, it sounds like that's really, really exciting. This has been great. I'm super excited about the things that y'all are building. Do y'all want to tell me a little bit about what may be coming down the pike for you next or what's upcoming? I can go first. Just building on top of what Ayush just said. The next step for us is basically building more context and making the agent recommendations from context aware. And we are basically treating the entire detection to mitigation pipeline as one single user flow and moving earlier up the chain. So what we are just launching fixes the investigative piece of it. And we are thinking now next, we are going to solve the detection piece of it. So Cloudflare has the ANS system, the alerting system where you could configure specific kinds of things you want to be alerted on. What we are thinking of doing next or planning to do next is make that alert more contextual. So imagine you get a notification that, hey, you are getting a specific graph rule is being triggered X number of times. And by the way, we looked at your traffic historically for the past seven days, this is what happened, and this was your data saves. And we think this particular rule is not working correctly, go and fix it. So now what we are trying to do is that step of going to an alert and investigation is cut short. But as Ayush mentioned, we are being very, very careful with it, because if we want to recommend something, we want to be more deterministic about it. But yes, we are very hopeful about it. And hopefully you'll be able to see that very soon. That's awesome. I'm really, really excited about the model of like closed loop AI security functions, right? This idea that you could trust an agent to detect a problem, recommend a fix for the problem, implement a fix for the problem, and then monitor it. And it sounds like y'all are getting really close to that. So that's really exciting. Awesome. Ayush, what about you? Yeah, on my side, for our customers, for email security, you're going to be seeing these in your UI probably by the end of this quarter. That's our goal. I think we have some great kind of preliminary data coming through. But again, that's subject to change. But I'm really excited to see this in the UI for all of our customers to use. Very cool. So to recap really quickly, how customers can get started right now. Harsh, it sounds like Cloudy is available for all customers on all plans. Ayush, your email security advancements with Cloudy, you're in a closed beta. How can customers be notified about that, get signed up with that? What's the plan? As of right now, there are no signup ways. But as soon as we start launching it out, it'll be GA or generally available for everyone when we flip that switch. Awesome. That sounds good. Okay. Thank you both so much. Really appreciate it. Have a great day and exciting stuff for AI Week.