Auto model routing saves money, makes users happy, and is inherently flawed

So I'm Andreas, I'm going to talk about automatic model routing, so how it can save you money, how it can make the users happy, and how it is inherently flawed. So first a little bit about myself. I'm Andreas, I was the co-founder and CTO at Replicate that was acquired by Cloudflare almost exactly a year ago now. Before that I was a machine learning engineer at Spotify, I was actually training models back then for music understanding. And if you don't know what Replicate is, it is an inference platform for open source and proprietary language models, image models, video models. So a lot of the kind of thinking behind our automatic router came from the work I did there, but obviously the world has changed and now everyone has a model router and we need a model router as well. So I'm going to talk about how we built it, what the problem is trying to solve, and whether it's actually the right solution or whether there is something better we can do. So, what is automatic model routing? So say you have an AI gateway, like Cloudflare's AI gateway or any of the other AI gateways that are out there. You have a bunch of models and users are using aliens to talk to those models. Sometimes they ask simple things like publish these slides to Cloudflare pages, sometimes they ask difficult things like solve the Navier-Stokes problem. I wrote these slides a few weeks ago when Astra was still the best model. For one of them you want to use an expensive, capable model and for some of them you want to use a simple model. So the task is really like what is the optimal way to let people solve their problems with as cheap models as possible but not too cheap. So how does this work? So what most people do is they train a model, a small kind of surrogate model that can run in the sort of the hot path of the agent to take a prompt or a trajectory or a subset of the trajectory and then output some probability distribution over the models. And that works but it's really hard to update with new models. If you want to add a new model to that pool you have to retrain the router and that can take a lot of time, it can cause drift and all of these things. So what we did instead was we made the router the trained model instead of predicting the model from the trajectory we predict what task is this prompt trying to do from we have a set of I think it's 14 tasks that we're predicting over and a few dimensions like what's the complexity, what's the ambiguity, is this a high stakes task or is it a just like a sort of prototype task and how dependent is this query on previous context because different models handle that differently. So we train that model independently of the actual model selection and then from there we have this heuristic algorithm, I don't even call it the model because it's just a kind of a matrix of weights from the task probabilities and dimensions to the model probabilities and that decoupling means that we can add a new model to the model pool without having to retrain the difficult to train router model. At the moment we only expose the actual router endpoint but you can imagine actually exposing the trained task prediction model as well so people can make their own routers from this. So that's kind of another benefit of this decoupled architecture. It also makes it easier to reason about a model choice so if the router says that this prompt should run on Opus but we look at the prompt and it's not that complex like what happened there we can debug it by looking at is this the task predictor that failed or the model selector that failed and most of the time it's the second which means that we can tweak those weights without having to retrain the route. That's our architecture. And how does it work so how well does it work when we released this a few weeks ago in the blog post we saw 90% Opus 5.5 accuracy at 64% of the cost. Now we have actually improved it further and also with the new models that have come out Haiku 5 .5 and Sonnet 5.5 they form a kind of a smooth Pareto frontier so you know it's like Anthropic owns the whole Pareto frontier at the moment but that has also meant that we have even reduced the costs even further. So this is something that we are using internally at Cloudflare and it's working and some people have switched away from it we don't sort of force people to use this but most people who switch to it sort of stay on it and are okay with it and it's you know it takes away some of the feeling that like the cognitive debt you have when you know that you're wasting a bunch of or spending a bunch of money it takes that away from me as the user and puts that on the platform. So that's kind of the one of the goals of AI Gateway is that users shouldn't have to worry about their agents running off and spending a bunch of money that should be a platform concern. So you can use this today this is basically like a save money button so you press the button and cost goes down and you can just change your model slug on AI Gateway to Cloudflare slash auto and it just works. So that's great right and this problem solved not quite. So there are some inherent ceilings in this type of model routing and the model routers they need to be really small. So our model is a modern BERT fine tune with some extra heads on it it's 150 million parameters it's really small it runs in like 100 milliseconds and our edge GPUs is Cloudflare which is great but it's not very smart like it's never going to be as smart as a as a real large language model. So the routing decisions or the task predictions it's making can like there's a very sort of hard ceiling on how good that can get. Another problem is that when you change them all when you change to Haiku you might get to the right response the right answer but it might take longer time and that's really hard thing to reason about like it's kind of like a genetic halting problem. How do you how do you predict how many steps it's going to take to solve the problem. There are some benchmarks that you know measure this at the average but that's not that doesn't really generalize to all queries. And the third issue with this is that I I use Opus differently to when I use Haiku like when I use Opus I can say there's a bug on the about page and Opus will figure out what I mean and you can be very ambiguous and vague with Opus and just sort of understand my intent but with Haiku you have to be much more prescriptive and there's a bug on the back button that's 15 pixels too high and you need to fix that that that all sort of goes out the window when you don't know which model is going to pick up the next prompt. So that's another problem with with this type of automatic routing. So what's the solution. And I'm going to be talking about this is not like a Cloudflare product. This is just kind of how what I've been doing since we shipped auto routing and I have in my local harness I have built a sub -agent system that can offload tasks to cheaper sub-agents. So I think this is part of the solution. So you know a sub -agent tool normally it has a prompt input and it has a description of what a sub -agent is. If you just add a model parameter to the sub-agent tool and you describe the models it has access to how much they cost what what they're good for and in the description to the sub-agent tool you say something like use sub-agents to save money. Expensive or token expensive tasks that are not that hard should use a cheaper model. And then if the orchestrator is something like Opus it reads those descriptions and it figures out that if I'm doing deep research on the Internet if I'm reading log files stuff like that that goes to maybe Haiku or maybe an even cheaper model in a sub -agent. And the thing that comes back to the context then is just the response. So these kind of sub-agents effectively become like a compaction of the individual sub-tasks and that's been working really well. You can keep the orchestrator context really small it doesn't grow very fast. All of those things happen in sub -agents that can use cheaper models. And that's obviously going to be a lot smarter than a tiny surrogate model. There's a bunch of other sort of harness level tricks here that people are exploring. Spotify released a blog post where they talked about their shunt system. I have no idea how to pronounce that it's like it's funny that the Swedish company makes a word that's unpronounceable for a Swedish person. So there the idea is that any large file reads at the file read tool level you check is the output here larger than some threshold. And if it is spawn a sub-agent to summarize the thing that the orchestrator wanted from that file read and put that into the context. There's a simpler version of that as well where large tool outputs again in the read file tool or in the shell exec tool you cap the output at say one kilobyte. And if it's more than that you put it on a temporary file or on blob storage or something and then the agent uses grep and those kind of tools to read the file. And compaction as well. I don't know if any of you were at Ming's talk about AI Gateway downstairs in the big orange globe earlier. But she talked about a lot of the things that we do at Cloudflare on the AI Gateway level like setting compaction rules so that we compact at 250,000 tokens instead of letting it go to 900,000 before we compact. Because obviously as the context grows it's kind of exponential how much it costs to make each call. So there are a bunch of those kind of harness level tricks that people are doing. There are a lot more of them as well. I think if you're interested in this just take some of these kind of keywords feed them, give them to your agent and let it sort of find the blogs and the posts where people talk about this. People come up with these new tricks all the time. So to summarize this I think we should press the stop bleeding money button and use Cloudflare Auto. It does work. But I think we should also start experimenting with harness level routing to save money. Over the past two years I've built probably half a dozen different harnesses for different things. It's really fun to build harnesses especially when you have durable objects and you have frameworks like Flu and Agents SDK and sandboxes on Cloudflare really tightly integrated. It makes it super easy to build these harnesses and there's a lot of alpha in there. I think people think that building something like Cloud Code is really hard. The Cloud Code code was leaked and most of it was just like an agent loop and a bunch of tools and some clever prompts. There's not that much to it and you can find a lot of things for your particular company that you can encode into tools and state machines and stuff like that inside the harness. So I encourage people to try out building their own harness and experimenting with these cost-saving techniques in there as well. Yeah that's all I had. It's not much to it. It's just change the model slug and you're done. Yeah.