The rogue agent problem: When AI adoption outpaces your security

Thanks for joining me here, everybody. My name is Milo. I do incident response here at Cloudflare, and today we're going to be talking about what happens when AI adoption outpaces your security. So I've been doing this line of work for about six years now. I've been at Cloudflare for about one and a half of those. Right now I'm doing internal investigations, incident response. Previously, I was more on the customer-facing side, customer incidents, investigations. In addition to all of that, I build security automations here. I do some of our insider threat investigations and buildouts for capabilities there. So I'm pretty tapped in to the problems that we are facing with traditional versus AI-driven development from the security perspective. What we're facing here is the difference between what we used to have before AI, which was clearly defined infrastructure, some baseline levels of shadow IT, nothing too crazy. You would have the occasional unapproved application that a developer would bring along for their use, and you also had very well-established best practices. We had this thing called the secure software development lifecycle, which feels a little outdated nowadays, because where we used to have all this very clearly defined stuff, now that AI is in full force, it kind of feels like we're going through a change in the way that we're doing our work every single month. It feels like there's a new thing going around, a new harness, a new skill, a new MCP setup, a new architecture entirely, month to month. We're also seeing much more extensive unauthorized application usage. If you don't approve a harness that somebody likes, most of the time you end up seeing that people install that harness anyway. If you don't give people a model they want, you see that they're using that model anyway through public endpoints, or even just bringing their own subscriptions to those models that they buy with their own money to use for work. And we don't really know what we're doing. We still have our practices very much in development while we try to figure out the best way forward as we go about developing our AI infrastructure. And we have such a wild west right now that realistically, I'm going to show you a list of our tools, of things that we see in every environment out there right now. And I'm very curious, I want you all to think to yourselves, how many of these do you recognize? I bet most of these are things that people are at least vaguely familiar with. Maybe some are new enough that people haven't really encountered them before. Another question, a slightly scarier question, how many of them are approved for use in your environment? Maybe it's all of them. Maybe you just say, okay, whatever your developers are using, that ends up being the best for their personal use case. Fine, you can take it with you. Or maybe there's a certain risk tolerance in your organization. You end up with a certain subset of these that are approved for use. You have a certain token centralized spend area where everyone is sending their prompts. Okay. Now onto the scariest question. How many of these tools are actually being used in your environment right now? Do you know? Because the answer is not always clear cut. With the way that our infrastructure, with the way that the actual AI landscape keeps changing month to month, week to week, it's very hard to stay apace of these things. You end up in situations where you end up with brand new harnesses every single month. You end up with new kinds of MCP servers. MCPs weren't even really a thing until more recently. Some of these harnesses, they support plugins, which are completely different from MCP servers. You end up having additional capabilities, additional callouts to external services that you're not necessarily anticipating. You end up with things like skills. People are getting skills from public repositories on the Internet. Are they reading those skills? Do they know what contents they're in? Maybe those skills have things like instructions about send a little bit of telemetry back to the home server. Maybe send a little bit more than just telemetry. There's also some things that people don't necessarily think about on first pass. Things like AI integrations in your existing SaaS products. Salesforce now has AgentForce. You have just about every single SaaS provider shoehorning an LLM into the product that they offer, even if it's not necessarily one that you asked for. And are people using that? Do you have a ZDR agreement, zero data retention agreement, with your SaaS provider, with the LLM that they're using to power that integration? And then there's local models. That's obviously a bit less of a risk. Those are running purely locally. But maybe you don't necessarily want all of your end users to experience the full cutting edge that some of these local models can do. They're getting very good very quickly. They're just a pace of the open weight models that are running on more traditional infrastructure here. And maybe you want to reduce, maybe you want to regulate or, say, just make sure that they can't go about doing insider threat things. Make sure that they can't go about accessing resources that they're not meant to. Do you necessarily have that level of insight and control for that? And, of course, I made these slides a month ago. Who knows how much has changed since then, right? The infrastructure landscape is very much in progress. It is shifting rapidly day by day. And that's not necessarily just a, you know, theoretical risk, right? It's obviously inherently bad if you have data leaving your environment in ways that you didn't anticipate. That is inherently something that we all understand is not good. But what is the actual consequence of something like that? Well, we can really get into it. So that slide, that little square on the right, is, I believe, IBM's state of the breach report. And they say it adds about $700 ,000 to your breach cost when you have shadow AI usage. And this is not really theoretical, right? You have surveys out there that are saying 88% of people are sharing work-related information with public AI tools. Companies are, you know, providing AI, but not necessarily the AI workers want. So the workers are bringing the AI with them. You have these surveys suggesting that 65% of respondents, or sorry, 58% of users, report receiving no particular training on security or privacy risks associated with these technologies. More often than not, I think we assume that this is being covered by our existing training, our existing legal awareness sort of annual or semi-annual required course. But this is fundamentally different. People are using these technologies in fundamentally different ways from most software. So that needs to be addressed in specific ways. That it meets people where they are and how they're actually using these tools, right? And then there's other aspects to it. There's the aspects of, okay, this is how the end users are doing this stuff, but then what are they actually capable of using this for? You have unauthorized MCP servers that you can run in approved AI harnesses that are nonetheless sending internal data to places that you don't necessarily have a lot of trust over. You don't necessarily know where this data is going. You don't have a relationship with the vendor for this MCP server. You don't really understand what that service is that's being provided, but it's happening in a harness that you may have already approved. Maybe it's happening in OpenCode you approved in. Hermes, right? But you didn't necessarily approve the MCP that they were using. You have unsandboxed agents that may be reading untrusted data, helping you debug issues that are happening in your production environments. When you have that happen, there's this untrusted data source going directly into an LLM. If these agents are not sandboxed properly, they can be prompt injected. They can be asked to do things that would not necessarily be of the quality you want. They can be exploited to return far more data than they are intended to return. They can operate in your environment without really a lot of oversight or control. So the way that we're managing this AI infrastructure and the AI revolution needs to account for the unique risks it poses. Fundamentally, that's really the long and the short of it. You need to make sure that when you are operating these agents, these AI systems at scale, that you're anticipating the risks that they pose at the same level, at the same scale. So, how do you do that? This is the part where I give you your single biggest takeaway that I want you to take from this entire talk. You should really centralize your AI infrastructure. You should route all of your usage through methods and infrastructure that you own. You should use endpoint and network telemetry to identify AI usage baselines in your environment. Maybe you don't necessarily know how many people are using the services that I mentioned earlier, where I gave you a bunch of icons, some of which I don't even know what they are. But maybe you don't know how many people are using these services versus your actual approved AI use. Make sure that you can leverage your existing endpoint and network telemetry to find out what the problem actually looks like for your organization. And then figure out what your risk profile is. It's going to be different for everyone in this room. Every single person in this room has a different risk profile and AI infrastructure strategy that is intended to address their unique considerations. That's a good thing. You all are going to have very different perspectives on how AI is going to be working for you, how you can leverage it in your environment and your workforce. But it's a conversation that I don't think enough people are having because if you're not having this conversation, then people are going to be doing things that you don't necessarily know about and that you haven't risk accepted. That causes a lot of problems. This is the part where I upsell you on AI Gateway. That's the best part of a company conference, right? But really, AI Gateway has been actually very instrumental for our internal usage. We've been using it with pre-approved and vetted harnesses and very much encouraging people to move onto those harnesses, giving the people what they want essentially by routing their existing favorite tools through AI Gateway. The nice thing about AI Gateway in particular is that you can actually take multiple harnesses and have them all hit the same AI Gateway endpoint to access whatever models that are proven in your environment, right? Workers' AI models, frontier models, whatever people need to use, they all live in one place. You have visibility into spending, activity levels, specific usage data. But really, you don't need to use AI Gateway specifically. Anything that you can find on the market that provides this level of control and visibility and oversight for you is going to be something that you really should consider getting. Because quite frankly, if you don't, people are going to be using their own tooling for you. Response actions, what they look like in practice, it's a lot of just being creative, right? People in your environment are using tools at a different level than we've seen in a lot of other cases. They are taking up whatever harness of the month is being deployed. The shadow IT is evolving in the way that people are actually using these things. So what I encourage you to do is to develop a baseline using the available telemetry you have, endpoint logs, network logs, figure out what public endpoints are people hitting for public AI. Like, for example, if you have a centralized AI endpoint and you see a lot of traffic going to chatgpt .com, safe to assume they are probably using that public endpoint for chatgpt. So hit them with a domain ban. Make sure that they can't access specific applications and AI tools that you haven't approved. This is going to be a lot of whack-a-mole because that's how the infrastructure landscape is right now. There's a new AI company coming out every month with a different model. So make sure that you are on top of these things. Implement application block listing. Harnesses are very powerful things, and they also happen to run directly on the host, which means you get to do things like block specific applications or only allow specific applications that are approved for use in your environment within your risk tolerance posture. You can also then build endpoint detections for potential unapproved AI activity. If you're seeing a lot of the same signals in your environment for this stuff, you're going to end up being able to construct particular patterns, particular regexes, particular queries, and you can run on regular basis to find out how bad the problem is in a particular area of your organization. Your incidents are going to be guiding you a lot in this, right? They're going to be showing you what you missed. When you have a security incident for an unapproved AI application, there's going to be a lot of lessons you get to learn from that, in particular regarding how your end users are treating AI in your organization. Like I say here, endpoint and network telemetry will show you the new tools people are using, and then you're going to be able to, over time, get better at responding to these new tools. You're going to be able to close these gaps in your response capabilities. This is how we are doing it here. You get to analyze usage of authorized tooling, right, to detect misuse and abuse. This is a particular capability that comes with centralizing your AI usage. You get to actually analyze the stuff that people are sending to your AI tools to see if they're using it for approved purposes. If you don't have central places to actually investigate the prompts people are sending through, the stuff people are building, the spend people have, you don't get to detect anomalies. If somebody in an organization that doesn't traditionally have a lot of AI spend within your company is suddenly spending tens of thousands of dollars, maybe you have a compromised identity, maybe you have an insider. The only way to know for sure is to make sure that you are centralizing that control, making sure you have complete visibility here, and then blocking everywhere that is not part of that walled garden, that single point ecosystem where you actually get to tell people this is what is allowed and other tools are not permissible. All right, well, thank you so much, everybody. Really appreciate it.