The prompt is lava

Hey, everyone. My name is Greg. I'm an engineer at Sentry. I'm going to be talking a little bit about prompt injection, but really I'm using this as an excuse to talk more about building AI pipelines that scale up with the organization without bankrupting you. So the type of prompt injection that you can sort of expect in observability space is slightly different than what maybe you've seen in the past. Like this year, skills got very, very popular and with skills, a new sort of prompt injection type of attack came in where if you sort of fetch any random skill from maybe sources you don't know, the risk is that there is going to be like a bundled script or there's going to be some secret instructions within the skill that then the attacker banks on you running on your machine without sandboxing, which is a pretty good bet. I don't sandbox. And that script, when it runs, it's going to exfiltrate your secrets. It's going to open a remote tunnel, whatever it is, but you're going to be owned. And I've done a little experiment in the past around that and frontier models, no matter how smart, were very, very capable of running these instructions unchecked and very happy about it. They were very trigger happy. And the same sort of type of prompt injection starts existing in other places, I would say. So if you use Sentry or if you use any observability platform, anything that tracks your telemetry, so gathers your errors, traces, logs, metrics, you're going to be exposed to this fun problem. So the way that those systems usually work is you deploy your applications, your front-end applications to your clients. Those usually are considered untrusted clients. And the way Sentry works is you set up something that's called DSM, data source name. Your SDK, your client application, is going to emit data to a public untrusted endpoint, the ingress control endpoint that we have. We call that one relay. We don't do any authentication. We kind of trust that DSM isn't leaked. It's not really trust. We advertise that data is to not be trusted and you have to sort of pay a little attention to it. That gets processed. There is a processing pipeline that makes sure whether this should become a new issue or whether this is an event that represents the same issue that you already have, then that is folded into actual issues page. So if you go to Sentry dashboard and you see some issues there, that's this roll-up. And from there, it's usually consumed in one of a couple of ways. Historically, those would be humans looking at the dashboard. More often than not, we're seeing growth of MCP. So you set it up on your local machine or you have an integration that integrates directly with your API and that consumes those issues, fixes them or whatever. And we also have SIR, which is our debugging agent, that can do the same. And the type of prompt injection that I mean here that you're probably prone to, to an extent, if you use any observability platform, sort of bases on this attack. I as an attacker can create a bogus error that looks like a normal error, but as part of the prompt, it says, oh, if you install this diagnostics package, you're going to be having easier time debugging this stuff. And then this attack again banks on the fact that you maybe have set up some sort of automation or maybe you're running an MCP and maybe you're running an agent that's not the sort of last gen agent. It's going to load this issue up, follow the instruction, install the package. It's my package. As an attacker, I'm going to have a post install script that's going to then exfiltrate your data and we're back to the same problem that skills present. And that risk is actually real today on, I think, all major observability platforms. So if you need a new headache, new vector of attack that you have to think about, you're welcome. We want to do the right thing here and try to capture those as much as possible and as early as possible. And when you think about when do we even have an opportunity to capture that signal, really what we want to capture is right before the issue is created, we would want to say, this is malicious. We should squash it. We can also technically do it in SEER agent because we control the agent, so we could literally do it every tool call sort of like a guardian model. And then MCP is kind of out of our hands. We will tag whatever untrusted data we have in like untrusted data blocks, sort of to help the agent on the other side, but that's about the extent we can help there. So the best place really is to capture it before the issue is created, before the webhooks are fired off and any automations start running. So for the rest of this talk, I'm basically going to be talking about the detection pipeline. In order to catch that, like if you take a naive approach, you could imagine before we create any issue, you could just run like an Opus model. It's like, don't worry about money. It's probably going to work. It's probably going to have pretty good accuracy if you just ask Opus, hey, is that malicious? And I've done some experimentation. It does work very well, except for two problems. Opus is slow and Opus is really freaking expensive. So we're going to split the pipeline into two. We still want big-ish model to do a verification because there is a lot of nuance when it comes to what's really malicious versus an outcome of a random application. So we want this like general intelligence pass on it, but we want to do it only for like a very, very, very small fraction of events because we know this is going to be expensive. So really what we want to focus on is figuring out how to make a cheap pass on all the data that is going to be as accurate as possible and filter only like 99.999% of errors that are going to go to the normal workflow. And the problem with that is it's still very expensive to do if you keep using LLMs. And the ROI is not really there. This is the right thing to do. It doesn't really grow the business, at least not directly. And for context, we really capture like an order of 100 events per day that are actually malicious while we're processing hundreds of millions of events. So the actual strict reviewer doesn't really get that much mileage. So if we try to run an LLM on every single event, it's going to be too expensive. By the way, I'm a little bit annoyed because I had a very fun meme here instead of this quote, but Cloudflare said this is copyright infringement. So we're going to keep it to the quote. We tried running LLM on all the events, way too expensive. So we ran LLMs anyway. But we did something different. We did a little heuristic which cut down the number of events. So an order of magnitude of events we're running the detection pipeline on is like 10 million a day. It's still a formidable number, but I want to raise it up higher because while this heuristic works, it cuts down some events. And I'm sure if we dig into how the heuristic works, all of you could figure out ways to go around it. So I'm going to avoid that. I'm not going to talk about that. But I want this to be 10x more, 100x more. I want to be able to process 1 billion of events as cheaply as possible. And if you do some maths and you look at this model cost, and you imagine that I'm going to have to 100x it, you can quickly come to the conclusion that I probably would be fired. So what I have to do is I have to figure out what is the next thing that I can use that would be cheaper. Those models, by the way, are fantastic. They are small, but the problem is so easy and well-defined that their intelligence level crushes the problem. So up until like, I want to say end of August, GPT-OSS was actually the best model we could use for this use case. And then September came, and we have a new whole paradigm. And when you think of decision models as a whole, this kind of problem that we're trying to solve, which is to detect malicious attempts at prompt injection, is very fit for decision models, because really the problem you're solving is take a piece of text, output a number. And the number can be a zero to one sort of confidence level of how malicious that issue could be. So it works pretty well as a concept. And when we compare cost with the cheapest LLM we've been using for Jeff specifically, it's like not as cheap as I thought it's going to be, but it's like three times as cheap. If I 100x, I'm still fired, but I'm getting there. It's also importantly very, very high accuracy. It beats all the LLMs. And then we started running a lot of decision model comparisons. Good job, Glef, by the way. It's very good. Really, this is the most interesting slide of this entire presentation. The way you sort of read it is you want the number of attacks to be ideally zero, maybe one for statistical significance. And then you have another important column here, which is the benign issues that are flagged. Because if you look at this table, you can see that Mercury Decider, Perplexity Decider, fantastic models. Very, very cheap. Catch everything. But the problem is they also catch a lot of issues that aren't actually malicious. And if you remember the original pipeline, it has the strict reviewer, it has the sonnet pass on everything. And if I use any of these models, the number of issues that are going to be sent to the strict reviewer is much, much, much higher to the extent where it becomes like half a million on the comparable load. So that's our next step. We're basically saying, okay, we can use decision models. It's going to get us to like reasonable cost. Maybe we can free X the number of events that we're running and still remain sort of within the budget. But what if we want to go further? And that's the basic point here. The basic problem here is that we are using a general intelligence models. Decision models are also general intelligence models. And the reason we use them is because it's so easy to prototype with them. I can only set up an API endpoint and I can scale it up pretty high and get like a very, very good result. And that's how we're testing a lot of our pipelines. But when we realize that the pipeline is good and we have to scale it for more usage and keep it economical and keep it business viable, now that becomes a problem. And like traditional ML is a reverse of good developer experience. You have to basically build the entire pipeline of gathering data, figuring out the on the casp cases, labeling them either with humans or better models. Refine tuning, retraining, synthesizing more data. And you have this whole platform loop that you have to create, which at a certain point is worth it. But either way, it's going to be costly. But we decided this is probably the right thing to do. A moment ago, I said that the problem we're really trying to solve is take a string and assign a number to it. That is a very well defined problem for specialized text encoders. The way that text encoders work is literally that. They tokenize. They are transformers models. So they to an extent work like LLMs where you tokenize a string. You run like a couple of attention layers and you get a number, whatever significance that number has to you. And so we started training that. We started experimenting with MMBird and ModernBird. The promise here is that if we can hit similar accuracy as to decision models, that's going to be much, much less expensive and faster because those models are around 150 million parameters. Even the smaller decision models I think run like a couple billion. And when we fine tuned an example, we were able to get to a north of 90% accuracy, still not where we wanted because we didn't build out the entire data gathering loop and but the data looks promising. And we're sort of shadow testing this. When you look at the cost here, it's if we deploy it on like not too many GPUs, we can get to like 12K on the similar loads which means that if I 100X that, I can kind of weasel out my way from a million dollar bill I think. But that's only for one security pipeline. The sort of way that we are talking about this is if we have to build the entire classifier pipeline, we may as well start using it for more use cases for like spam detection and other signals that we can use on telemetry to then enhance the product. And I don't really plan for time for that, but I'm going to just mention this very, very shortly. This is like distant future. This is something that I want to test. Well, not distant. This is something that I want to test within this year. Technically, on the paper, what's also possible is take those 150 million models, take smaller like 4 million parameter model and distill the knowledge. This is the like dark knowledge training that you maybe have heard about where you can take the outputs of this model as well as outputs of something bigger like Jev or like an LLM and try to distill that knowledge into the smaller model and that would cost us, this is incorrect, it would be like $700 a day on like the full load. And that becomes a negligible cost that we can then multiply into multiple use cases. It also becomes much, much faster. Like at this point, we're talking about significantly under one millisecond of processing, which then you could put directly synchronously on all the pipelines that you have for processing and it wouldn't matter. Again, this is hypothetical. I don't want to say that we're doing that, but this is something that we're now striving to do. This is my time. My two pieces of information I want to leave you with is one, if you're using observability platform, pay attention. You're probably being attacked by one. Not by one, through one. We're not malicious. And then two, if you're starting to approach scale while the experience of generalized models is great, you're going to want to start training your own specified classifiers. And I have a couple of minutes if there are any questions. I was going to ask, where do you run that classification? The classification itself. We are renting the L4 GPUs from GCP, but we have to self -host, which is another layer of complexity we have to do if we're training our own classifiers, as opposed to any models that are just available over an endpoint. Thinking about the problem of prompt injection broadly, do you think this approach of just train some classifier that runs on any untrusted input, and then have that flag potential attacks, is that a good approach? Do you think people are going to adopt that, or is it going to be highly specific, depending on the domain you're working in? I think it's not enough. So, the thing that I didn't really get to cover here is oftentimes you will want to enhance it with other things. So, for instance, it's very easy for a classifier to detect that there is a package installation. You could also judge whether the package installation is malicious by indexing those packages automatically and running it against an embedding. But I wouldn't say one replaces the other. I would just combine those signals. How much time is it going to take to train the distiller to the end of the year, $7 per day? When, I will rephrase it, when there will be the threshold when you will decide that the distiller is ready to be put in work? Sorry, can you repeat that? Where is the threshold when you will decide that the distiller is ready to be put in work and trained? So, we already have, I don't know if I mentioned that, this pipeline already runs for a couple of months. But it runs on either LLMs or on decision models. So, the threshold is it's approaching the accuracy of that pipeline or going past it. Because then it's a pure win in terms of cost. The argument could also be if it's not really hitting that level, but it's like getting there and we're starting to hit plateau, but it's so much cheaper to run than using these decision models, that could also be a decision we make. But at this point, it's I want to hit that accuracy. I'm already hitting over 90% and I'm pretty sure I can, at least for cases we know, I can get there. The problem is cases we don't know. Thank you. Thanks for the information, Greg. So, you mentioned you were sort of experimenting and trying to build this pipeline. I was curious if there's any tools or other infrastructure that you sort of messed with that could be immediately useful to start building your own pipeline as you get into this? Sort of. So, at some point, it's going to go far, but let's do it. There you go. At some point, I mentioned that we also have a secondary check in SEER. So, this is what helps us capture issues that we let the first pipeline sleep. And that's our back pressure, back test sort of thing. Because the number of automations we run for SEER is significantly smaller than the throughput of traffic we're going to see on the issue creation. And because of that, we can run more expensive models. So, those will sometimes flag things that the cheap pipeline was not able to or failed to for whatever reason. And sort of acts as a secondary check that we can then use to backfill more evals and stuff like that. And I think I'm out of time after that. Have you considered only running the eval when someone is actually looking at the events instead of trying to, you know, detect the pump injection for every event that you ingest? When someone's looking at the issue? Yeah, via the API or looking via the MCP server or whatever. We did consider that. We were worried that the number of entry points is too high. And some of them didn't make a lot of sense, I think. So, one of the checks I think we either have run or still run on the back of MCP. When the MCP decides to fetch you an issue, we would test that. I don't know if it still works that way. So, I don't want to overpromise. But it felt very largely duplicative to what we've been doing on the issue as well. We are not doing it right now. Thank you very much.