Closing the loop: Agentic threat intelligence and threat response

We're going to talk to you about how we think about agentic threat intel and threat response at Cloudflare here. So I think one of the one of the top areas of what Javier and I wake up every day thinking to ourselves, what are we doing with humans that we shouldn't be doing with humans? And how can we make cyber response operations more effective with automation and AI? And so throughout this presentation, we'll cover kind of a couple different areas that Javier will get into. But at the end of the day, we're always thinking to ourselves, like, how do we get faster? How do we make better decisions? How do we become more deterministic in the way we think and create a more regimented playbook in order to have the right response and not make errors? And that's a lot of the focus of this presentation. Yeah, and so the goal is to have a tier one workforce that we can offload so that our analysts can become tier two and tier three. So today we're going to talk about three agents that we kind of have, and they kind of play a different roles in our ecosystem. The first one is a threat intel agent that's responsible for handling requests and leads and then outputs reports. The second one is a managed defense agent that handles our SOC alerts and generates WAF rules and triages things. And the last one's a vulnerability discovery agent that goes through code and outputs WAF rules. So the first agent is our threat intelligence agent, and this one's, there's nothing magical about it really at the base. It's a standard flu agent deployed in a worker, and it uses skills and tools to kind of triage these requests. The magic is that we kind of have put all of our intelligence into these skill files. And as we get requests, we try to, we use these skills and tools to give us a response and we keep updating these skills and tools as we go through these responses. If something wasn't done correctly, we then update the skills and tools. And overall, it's kind of like putting all our collective knowledge into these things so that we can answer these responses. Now this is interesting because this is a little bit of an open ended investigator. It's not just something that we're very restrictive on. It can kind of scour the web for all these leads and pull in information to kind of output some context or some information about the lead that we have. So I would say like, this is one of like the areas that's super interesting, which is this of all the agents we'll discuss today is the most abstract in that we have data we look at with the agent, but we don't necessarily always know what we're going to find. And so as something I always worry about, and I think Javier worries about a lot is how do we ensure determinism and how do we ensure we don't just get garbage input to garbage output and model hallucination, things just making up results that fit a narrative because that's always our concern. And all of our output for these things, we try to cite it in concrete evidence, whether it's a URL or raw query that we kind of got, we try to tie it to the output so that we can gain confidence that there was no hallucination in the process. But again, this is a very basic one. Our next one for managed defense is one as a SOC one that handles alerts. And we have to be a little bit more prescribed on how we deal with this agent because it's going to generate WAF rules. This is going to actually cause actioning in your environment. And because of that, it's not open-ended like the first one. It's a little bit more restrictive. So all the recon we do is via code. And we do recon to pull like customer telemetry, global telemetry, threat intel cases. And we do this because we want to be able to recreate it, recreate any of our analysis in the beginning when we're doing collection. After we do collection, we want to be able to either accept the alert or reject it. And to do this, we go through a series of clef logic for each parts of our recon saying, does this recon support the alert or does it reject the alert? And at this part, we can actually dismiss the alert or let it go on to the next phase. And we choose clef, a decision model, or clef is Cloudflare's version of Jeff. We choose clef because it's a little bit more, or it's less hallucinogenic. It's more deterministic so that we can create it. We're doing an automatic action. And we need to be able to reproduce this when something goes wrong. After we accept this alert, this will go on to a more traditional agent that then interprets our facts. It interprets the recon, the clef output, and it will suggest a WAF rule that is then validated against the network, the customer network. Is this going to disrupt their domain? Is this going to take care of the attack? And that is given to, that's put in logging mode in the customer's account. And the SOC analyst can either accept or deny it, like a pull request, to take an action in the account. So again, we're being a little bit more prescriptive in how we're doing this because the risk is a little bit higher. We're taking action. We're not just doing an open investigation. And importantly, that agent, like we built that in just, what was it, like two, three, four weeks of work. That entire workflow we just showed you was basically about four weeks of work. We couldn't do that without the power of the Cloudflare Developer Stack. So we're able to do rapid iteration with the Cloudflare Developer Stack, with the tools that you all have access to today through the developer platform. And those tools have really changed our ability to build things in rapid succession. Well, not just that. The first iteration of it was almost the same kind of agent that we had in the first one. But as Clef and Jeff came out, all of a sudden we have to pivot to something that's a little bit more deterministic that helps us. And we're still making these pivots within the ecosystem to kind of improve and be more effective. So again, in the beginning, we're very focused on getting all of our facts concrete, doing our recon via code, moving on to the next step, and using Clef and Jeff to be a little bit more reproducible in our automatic action. And then we pass all that to the agent for interpretation. The next one is our vulnerability scanner. So our vulnerability scanner, a lot of people have been making these agentic vulnerability scanners. They look at the code, they try to find vulnerabilities, and they output 4,000 vulnerabilities. And now it's on to the engineering team to figure out which of these 4,000 we need to deal with. And the agent does its best to evaluate it, but it's just going off static code. And I don't know if anyone's ever done malware analysis before and analyzed a Windows executable. When you look at it statically, like the disassembly or the decompiled, you kind of understand what's going on, but it's not until you introduce dynamic content you kind of see the requests that it's handling, you see the requests that are going out, that you really understand what the code's doing. And so for this approach, we do the same thing. Cloudflare proxies a lot of our customers' domains. And then when we have the code, we're able to not just do the vulnerability scan, but we're able to take context. We're able to take their web traffic and see where the hot paths are. Where are the users going? How are the users interacting with the service? How are other services interacting with the service? And so instead of just pulling like dependency modules, like for like PIP installed things or Node installed things, we're actually pulling the way it's being used so that we can have a better evaluation. Not only that, we're using WAF rules. Why are we using the WAF rules? Because this lets us know where the attackers are targeting us. Like this lets you know where the attackers are like near misses to some of the vulnerabilities we might find. And these might be more critical than others if we were just looking at it statically. Now this harness, I won't dive into too much because Dan Jones is going to talk about it tomorrow. But each one of these steps is a sub-agent that can be configured with different types of models based on how computationally expensive or how often we call these things. But outcomes, vulnerabilities with a priority, again, not just based on code, but based on your network traffic. And also what that comes with there are suggested WAF rules so that we can apply virtual patches in your tenant instead of just waiting for engineers to patch these 4,000 vulnerabilities. Security can be empowered to put these blocks in place to kind of protect yourself now rather than waiting for these cycles later. And again, this is just an overview again of we use the paths to find the hot paths. We identify the routes. We use the network context, WAF context to prioritize our vulnerabilities. And then again, for the source code validation, it's kind of like the vulnerability scanners that you've seen, the agentic backed ones. It's bounded by code investigation. We get the vulnerabilities, we validate against the code, and then we give remediations. So everyone's making agents, right? Everyone's making agents, so we're all doing it in different ways. And we've kind of had some like lessons learned. I feel like that's the biggest thing that's kind of happened. And so the agents we've picked are very scoped. We don't have one agent solve everything. And that's done purposely because we don't want an agent just to... Different agents have different risk in different scenarios. Again, the threat intel one, low risk. You're just pulling in investigation. You're just pulling in information. The vulnerability discovery one and the SOC one, more risk. You gotta be a little bit more stringent in how it's going. Also, sometimes the input might be inherently malicious. So in that case, you can be a little bit more aggressive with your agents. We're very big on evidence over instructions. Again, we don't wanna tell the agent how to do all this investigation. Sometimes we know the facts we need to collect and we should do that via code if we can. Everything we output, we should validate. All of the stuff that the models output should be brought back in and have pointers to where it got that information for tracking. And we should always save input and output, all the recon and thinking, because this is gonna help us later on when we're trying to change models in and out. And it's gonna help us if a WAF rule is suggested that maybe didn't align with how we felt. And we need to be able to look back and see how it's different. And on the point of model changing, it is empirically critical as cybersecurity professionals that we have access to every model there is. We need to be able to see how a Chinese model might think about a problem, how a model from Europe might think about a problem, how a model from somewhere else might think about a problem. Because you have to understand, we have threat actors that have access to those models. They may use that model against you if your company is afraid to use this Chinese model to test how your business is gonna perform. You may get hacked by that Chinese model. And so we're seeing this throughout like the Fortune 500 where there's a lot of companies that are like, well, we're scared of the Chinese models. It's like, well, that's not the right answer. We need to be able to use all the models and we need to be able to change which model we're calling over time for cost reasons possibly, for safety reasons, for just understanding attack surface of those different models. Because each one does think differently and does have kind of greater strengths and weaknesses. And then there's also cost efficiencies. Do you wanna talk briefly about like how we think about cost efficiency? That's actually like a really big thing because like every time, again, it's always fun in games when you're making models until all of a sudden you're constrained to a budget or you're constrained to resources. And we've bumped up against that a lot. We process like 60 ,000 alerts a day and it kind of turns into this traditional problem. We used to be developers. We used to be developers. Some of us are still developers. Some of us are prompters, but used to write code in Python, right? It processes the 10,000 requests. Everything's fine and dandy. And then you get to like a hundred million requests. You're like, oh, Python doesn't work. We have to move to C or Rust. And the same thing applies to like these agents and the models we use. So in order to kind of deal with that, we again have to store inputs and outputs in our recon so that we can evaluate and change things as these models change. We have to be able to do benchmarks against them. And so, you know, like I mentioned earlier that we went from like a frontier model or like Kimmy or GLM, and then Cleft came out. We immediately wanted to start switching over to the cheaper model, the quicker model, but we had to do it with confidence. We had to make sure that we could run a score. Again, we can evaluate and see how many changed and evaluate the output. And not only that, when you're hitting these resources, sometimes you have to have a fallback. You need to fall back to a different model because the other ones might be out of resources. And you have to be able to do that with confidence. And you have to be able to do things like asynchronous counts you know, like maybe something, this alert doesn't need to be triaged real time. Maybe we can offload it to maybe when it's a little bit lower resources or lower expense. Maybe it's at night, you're able to offload all the requests. Maybe talk just briefly about like, when we talk about model fallback, right? Like this model becomes available. How do we switch to the other, another model while ensuring that our efficacy does not change? Yeah, so I mean, that goes back to the evaluation of inputs and outputs. We have, we save all of them. We can run them against the various models that we have and see which ones are comparable for the step that we're dealing with. So each one of those steps in the vulnerability scanner, we've compared the models ahead of time to say like, okay, we can fall back from like a Daybreak Blue model to a Kimmy model on this specific step and we don't lose much efficacy. And then with like AI Gateway, we're able to put the model and just simply have a fallback model. If it fails the exponential back off, it just falls back to that model. And we're a little bit more confident that like this one's not gonna perform worse and put us in a bad situation. And then last thing is like with that AI Gateway, we're already, we're getting caching. So if you're doing the same request, you're getting caching and you're, we're able to kind of like, again, reassess our models to kind of like do what's optimal. And sometimes it's not even going from LLM to Clef or whatever new versions out there in the future. Sometimes it's just going back to code and making sure our code can be equivalent to what's going on with some of these LLMs. So we're in a good spot right now. Again, we built this tier one workforce. Our threat Intel agent is capable of responding to 80% of our requests and leads without analyst input. Our analysts review it, they check the sighting, they check the data behind it. It looks good, we move on. If it doesn't, they add the information into the skill file so that we know how to, where to pull the information or how to evaluate it given certain contexts. Our alerts that we're dismissing in SOC, we can do under 10 seconds. And if it doesn't make it past that phase and into like our, in the SOC hands, our psych analysts hands, they can do it within three minutes. Again, 90% of these actions are done via these agents and from end to end. So I guess the big takeaways for all of this are, when you need to be deterministic and you need to define facts because it's important for auditing, it's important for changing models, it's always good to have that as like your base. Like everyone should do that. It's always fun making the model and the agent like day one, day two, but there's gonna be month three, month six, and you're gonna have to maintain these things. The second takeaway is again, make it audible. If something happens and the output's done, you need to be able to cite the sources. And the last one is, again, we're trying to build this tier one source. We're trying to figure out where humans are doing things that they probably shouldn't be doing things because we can automate that. And that's not just to replace the humans. That's to turn the tier one work into automatable work, just like we used to do with code. And this is an important point. We actually had people in our org that were like, oh, you guys are trying to replace us. And it's like, we actually have more work than we do people. So we actually don't need to replace you. We actually just want you to do other work that's more meaningful than like false positive, false positive, false positive, false positive. Like this is not an effective use of your time. I actually need you to use your brain cells to analyze more complex attacks than marking things as false positives. So like we're trying to move people to do the right work to have a greater impact as opposed to doing just trivial work that can be done automatically. Yeah, and that's a good point because there's a lot of ABTs out there. There's a lot of intersection with Cloudflare's telemetry and these attacks, and it overwhelms us. You know, we have 60 ,000 alerts. We're not gonna be able to go through all of those. And so falling back to this stuff really helps us to kind of like level up everyone and so that they can do like the important work. Never once in my career have I thought to myself, wow, if only we had less people, there's no work to be done. Like that just doesn't happen in cybersecurity and never will happen. As good as like AI is, there'll always be cybersecurity issues that come up. And, you know, having the humans being capable of responding to those issues, it is the best, right? And we're just trying to optimize and become more efficient. Yeah, so moving forward, we're gonna continue like our mission to add AI observability into all of our agents, not just these three, but the others so that we can continue to adapt as new models are coming out. And we're gonna keep exploring different ways to scale. Is it using SLMs? Is it using fine-tuned models? Do you wanna talk about what SLMs are in case people don't know? Small language models. So just not the large ones, just smaller. It may be talked about also how we think about SLMs and where they might be more applicable. So we're looking at using SLMs probably more in the, again, in that vulnerability discovery region where something's being called like thousands or tens of thousands of times. You don't wanna waste like paid resources on that. You wanna be, or you wanna use a general model. You wanna use something more specific and tailored towards that. So just we can kind of scale and be more effective with that. And I think the SLMs are interesting because it also like Javier's earlier point in one of the earlier slides was about how we wanna have determinism, right? And so like an SLM might be more deterministic in how it inputs or outputs data because it's trained on less volume of data. So we might, it's just like an additional determinism that we can add to a workflow that gives us more confidence and how to be confident to block something or take out some sort of other evasive action like wiping out someone's hard drive. Obviously we're gonna do that. We wanna be very certain before we just start erasing someone's device. Correct. And so that's how we're using agents at Cloudflare. That's how we're using them across to kind of like level up our workforce. And yeah, always interested in what everyone else is doing with agents in the same scenario. Thank you.