Code Mode: Fit 2,500 API endpoints into 1,000 tokens

Welcome here for this session. My name is Léo. I'm a solution engineer at Cloudflare. I come from France and I work in Paris with a lot of startups and scale -ups, so I'm helping them to implement our products. And today I'm going to do a presentation about CodeMod, which is a way to implement MCP server, a more efficient way for token usage and also in the way the agent is going to behave. So I'm going to go to the technical and the theory, but I'm going to also show you a real-life example of one of my customers in France who implemented it for production services, and I hope you will be able also to use it for your application. Maybe before going through the presentation, who is using MCP within their organization, Okay, so almost everybody, so I guess you are facing some issues. Who knows CodeMod already? Okay, a few people. So that's great because I'm going to go back to the basics. Before explaining the theory behind CodeMod, I'm just going to explain a bit how we were using agents at Cloudflare, how we were using MCP. One way to use MCP is, of course, through an agent coding platform. It can be OpenCode, it can be Cloud Code, CodeX, all of that. And the way you do this, every time you want to plug a new MCP for your agent to be able to interact with external system, you need to plug a server. And then the right server is going to have different tools, so you can plug as many as you want. It can be our internal portal, it can be the Cloudflare API, it can be the docs. But in your use case, it can be whatever system you want your coding agents to interact with. It can be GitHub, it can be anything. For non-technical people, you can also build some kind of chat, agentic, or application. And this is something we released sometime at Cloudflare where we allowed non-technical people to interact with AI. And there was a way to connect MCP if the, I don't know, salespeople or marketing people wanted to interact with the wiki, Jira, or whatever system they needed to. And we allowed them to connect and plug MCP and different tools. But we did something. We set up a limit in the number of MCP they were able to connect. And some of you might wonder why. Some of you might know why we are putting a limit. It's because when you plug too many MCP in an LLM, each of them have a lot of tools. And if you multiply the number of tools, you are a bit flouting the context. Because every time you start a new session for this agent, you put all the tools, all the MCP, and his context is kind of exploding. So the agent is less relevant, less efficient. It's consuming also way more token for nothing. If I just want to check, like, my last email or something in the wiki to do some research, I don't have to plug the entire MCP catalog for my enterprise. So this is why we put the limit. If not, you get, like, with context problem. This is even more relevant if you want to plug. So I showed an example if you want to plug a lot of system. But even if you want to plug only one system, which is really complex, like the Cloudflare API, you can have an issue. Because the Cloudflare API has more than 2,500 endpoints, which is a lot. And if you want to create, like, a tool for each of these endpoints or each of these use cases, you end up with a lot of tools. So one thing we did at the beginning, we thought it was a good idea to divide our MCP to different categories. So we didn't have a super MCP that your agent could interact with to do everything. So we had different MCP for different use case. That was great because it was working well. Each of them have, like, a few tools. But the issue is if you wanted your agent to be able to do everything on Cloudflare, you are limited because you need to plug another one. You need to unplug one to select the tool he can access. And it was too much. So we needed a way to find a strategy for the agent to be able to perform everything without providing too many tools and flouting the context. And so this is not something that was made by Cloudflare. Actually, if you read some papers from Apple, they already did some research about it before us. But we realized, some of the engineers in our team realized that LLM are way better at writing code than writing tool code because, of course, they were trained on a lot of data, a lot of code base. So it's way more natural for an LLM model to write a bit of code, API calls, than writing tool code, which is something less natural for it. So going in that direction, we think, okay, maybe instead of giving a lot of tools to my model, to my LLM, I can just give him, like, an open API spec and I can ask him, please write, like, API calls to target this endpoint. So this is a bit the strategy we wanted to go in. And so what would happen if I ask, like, my super, in theory, agent to protect me against DDoS attack? Instead of going and calling tools, the agent is going to write code, fetch some API endpoints. So this is just for illustration, but this is a bit of the context. So he's going to say, okay, which API endpoint would I need to create rules to protect me against DDoS? He's going to get the endpoints, and once he gets the endpoints, he's going to be able to perform the request because he knows the correct body to send to perform the action. So that's way different than the way we do today if you want to call tools. And that's what we wanted to implement. So how is it working in practice? So you have still the LLM, which is still interacting with an MCP. Why? Because an MCP is a really cool way to interact. It's universal. You can plug it really easily, and every LLM is compatible. So you still have an MCP, but with only two tools, one which is search, the other one is execute. The search tool allows the model to get the endpoint, get the information he wants to perform the call. Then he writes the code and sends the code to the execute tool. So when the execute tool is called, what's happening is the MCP needs to execute that code. Right? The execute tool. And to execute that code, of course it's untrusted because it was generated by the LLM. So we don't want our server to execute an arbitrary code that we don't perform any control on it. So we need a little sandbox. And we have something really ideal for that, which is called a dynamic isolate, which can allow you to trigger little workers. It's a little isolate that can allow you to execute JavaScript, and that's the way you can do and execute the code model. So I fetch the endpoint API, my LLM writes the code, he says to my server, this is the code I want to execute. The server gets the code, executes it in the little sandbox, and then this is how we perform the action. So that's the theory behind code mode. And if you implement this for the entire API, you can see that it's a way more effective way than just doing a row of an API spec, and we typed it to be more token efficient, and that's the kind of number we can manage to get. And I'm just going to show you a little example. So this is my backup demo. I will try to do a live one, but with LLM it can fail at any moment. So I created a JIRA -like system, so it's to manage tickets for engineering, so it's really simple. You can perform only a few tasks. There is two modes. One is classic MCP. Maybe it's too small. I don't know if I can... Yeah. So one is classic MCP, and the other one is implemented with code mode. So they have the same feature, but the two different ways to implement it. So if I ask them something really simple, which is listing the five most recent tickets, we might see that code mode is a bit overkill because this is something really simple. Just one tool call, listing tickets. I want to see the five last one results, and then my LLM can interpret the response. In code mode, I have to write code to execute the code, so that's maybe a bit overkill for only one single action, and you can see actually that in token consumption, it's consuming a bit more. So I'm not saying that you should use code mode in every situation. If it's just simple tasks, you might not need it, but if we try with a more complex prompt, which is... I mean, you're going to see it here, but listing tickets, closing some tickets, stuff like this, we're going to see that... In MCP, I'm doing a lot of tool calls one by one, waiting for the response, then doing it. After that, it's done. It's a bit more long in code mode because I don't remember which model I'm using, but it's taking a bit of time, but we should see in a few seconds that I generated a bit of JavaScript to perform the API call, as I said. The value is returned, everything is done, and we should see that the token consumption is way more efficient. So for this complex task, complex, having a lot of action to perform, a lot of tool calls, we see that we're 50% more efficient in token usage. And this is for one single agent, one single prompt, but you can imagine that within an organization, if you have a lot of agents in parallel, a lot of complex tasks, it can scale and can save you a lot of token usage. So that's one of the main benefits of code mode, token efficiency for, I mean, of course, the cost of the token, but also for the efficiency of the agent, how well he's going to perform to do the task. So it worked well. I can just skip this. And just one example, so I have a customer in France, which is a health platform. Everybody in France who wants to book an appointment with a doctor, they might use it. It's a complex workflow, like the one I showed you before, because you need to get the referral record of the patient, you need to find the directory of all the practitioners within the region, then you have to find the slot to book the slot. So there is a complex workflow with different steps, different tool calling. So if you want an agent to be able to perform all of that, you would need to implement different MCPA within different systems. So that's where code mode can be useful, because instead of building different MCPA, different tools, and then chaining everything together, and maybe having a more dumb agent, which is a bit lost sometimes with all the tools he has, he can just get the API and call them sequentially. So finding the referral, checking every calendar, and then booking for the agent. So that's a real life use case that was implemented. And maybe I'm just going to show you a quick example of code to what it looks like if you want to implement it yourself. So basically, we have an SDK that can allow you to deploy MCPA on Cloudflare. So you can deploy an MCPA the classic way with different tools and stuff like this, but you can also deploy MCPA with code mode in Cloudflare. So the way we did it for our API, if you have a really complex API and a really complex system, you can also do it really easily on Cloudflare. And the way it looks is here you can see I'm loading an open API spec. So something really basic that you might all know already. And the way we use it, so everything is bundled in an SDK. So that's really easy for you to use. I have a spec here that I can reuse in my open API MCPA server bundler and then everything is set up. So that's really easy to set up. Nothing crazy. You don't have to manage the infra because it's running on workers. So no Kubernetes, no containers. Everything is stateless also with the new MCPA spec. So that's really easy to deploy. So that's a really simple one. Because I only have two endpoints. But even for more complex use cases, that's way easier than going from scratch, implementing tools like this. And if you already have an MCPA running with different tools, you can also implement code mode with a wrapper we have in the SDK. So you don't have to go from scratch. If you already started building something and you face a limit in the number of tools and so on, you can work from that and move to the code mode API. I'm going to also show you. So this is if you want to build your own MCPA for an external audience, I would say. Something also useful that we have at Cloudflare is if you want to expose MCPA to your internal user. So I don't know the role of all the people in the room. I guess some of you are building stuff, building APIs, building MCPA. But some of you might also be responsible for the internal infra and the MCPA you are exposing for your user. So there is something really cool at Cloudflare which is called MCPA portal. So I don't know if you know it. So it's part of the Cloudflare offering. And it allows you to expose in a really safe way external MCPA. So not the one you are building to your developers and your users. So basically everybody within your organization who needs to access MCPA, you can expose them via the portal and you can have like logs of all the record, the access control. You can put policy plugged in your Octa or Entry ID and access whoever is able to access whatever MCPA and block some policies and allow some of these people. And what's cool. So if I have only two MCPA in my portal, it's easy. But if I give like 20 MCPA, 30 MCPA, 40 MCPA, I face the same issue I was mentioning before, which is people are getting too many MCPA, too many tools. So there is a way in the portal to enable code mode by default. So everything which is going to be accessed by this user using this MCPA portal will be optimized with the code mode. So it's going to be really more efficient for them. So there is two benefits of the portal. Of course, code mode for everything I mentioned before, but also a really secure and safe way to expose MCPA and external system to your developers and your users. And this kind of thing, we're using it extensively at Cloudflare. So everybody in the company that want to connect to an MCPA, they have to go through our internal portal to authenticate and everything is optimized that way. That's it for the overview. If you have some questions, I might be available to answer them at the end. But that's it. Thank you.