Scaling MCP: A simpler, safer enterprise architecture

Hello everybody and welcome. I am so excited to be here today with one of the authors of a blog post that came out during Agents Week. I hope everybody has been devouring the blog. If you didn't read this one yet, it's time to go read it. I am so lucky to have Sharon Goldberg here to talk a little bit about what is up in this blog, how to deploy MCP in the enterprise. And Sharon, actually, before we get started, can I have you introduce yourself? Yes. Hi, I'm Sharon Goldberg. I'm a product director here at Cloudflare. I work on a bunch of different, really exciting things. One of them is our AI security suite. I also work on post -quantum cryptography. I also work on data sovereignty. It's been a really fun couple of years here working on these really advanced technical projects. So yeah. And then this blog was really fun because we pulled together a whole bunch of different things that different groups in the company have been building and also deploying internally and using it actually on our own workforce. So we thought that it would be fun to kind of pull it all together in one place. And as we were working on it, it kind of grew into this giant reference architecture that we're like really happy to release today. It is so cool. And I think that like we might have a little bit of prerequisite that we need to do. We're going to go a little deep here and I want to first, I want to talk about what MCP even is. Is that okay? And Sharon, will you tell me how I do with it? Okay. I'll do my best, Craig. You're the star at this, so I'll sit back. So no, no. At the fighting MCP. So MCP, if you haven't used it yet, model context protocol, right? So the LLMs are needing to do things. They're needing to have more context and they're being able to be given tools. A lot has changed in this space and a lot of people have different feelings about this. And there has been some security issues that have been raised and some people have said MCP is dead. I am very excited to see what we have built here. And I think one of the things that first, very first started happening, which is interesting is people were running things locally, right? So you took this MCP server and I installed it on my local machine, my way of communicating, I guess my MCP client or my way of communicating was local on my machine and that could go off and call things. And the LLM would come in and do things locally on my machine and then go out. And I think that there was some problems with that, if I'm not mistaken. I mean, I just think in general with AI right now, we're still a little bit in the like first few innings. There's a lot of things that happen with AI that feel to me like we're still in the 90s, right? And like one of those things is that people run things locally with AI. They're doing like these agents will be running locally and like updating your local file system and stuff like that. It feels like very sort of like the way we used to do things. And so I would say with MCP, that's a similar situation where this all started was that you would, let's say I want to interact with GitHub or GitLab repository, right? You would put an MCP server in front of that to make it easier for an LLM to speak to that GitHub repository. But where is that MCP server hosted? Are you hosting it on your machine? Like this machine that I'm speaking in front of right now? That's what you were doing at the beginning. And that's what a lot of organizations are still doing. Now, the problem with me running an MCP server locally is who wrote that MCP server? Who wrote that MCP server? Is it patched? What version is it using? Have we done security scans against it? Does my IT team really think I should be deploying this MCP server against our code repo and running commands through it? Like maybe there's a bunch of tool injection attacks in that server. Who knows? If I run it locally, it's harder for my IT team to actually administer and control that. And even though that's still done in a lot of organizations, we actually don't do that here at CloudClare anymore. And so that's the sort of first part of the reference architecture. Well, yes. Speaking of the reference architecture, I saw a diagram. You had a diagram a while back that I saw in a slide. I grabbed it. Can I share it? Go for it. All right. Right. Okay. Awesome. Okay. So here's the diagram. I yoinked this right out of a presentation that I saw you giving. So up here on the left, we have these LLMs, right? Yeah. And we run through here, through this AI gateway. Let's talk a little bit about AI gateway. Yeah. So actually, maybe let's start from the user, right? So I've got my user at the bottom, and my user wants to do something with MCP. So for example, I may want to make a command or something that says, hey, can you go look for all the GitHub repositories through all my GitHub or my GitLab repositories to find all certificate authority implementations that we have at Cloudflare? And so it'll just go do that for me, which is really useful. So I don't have to read all this code, and it knows what a certificate authority is. It just finds it. How does it do that, right? So that's the MCP client that's sitting there. The MCP client calls out to an LLM in order to make these kinds of queries. And so that's that leg that you see there. You can have any LLM that you want. At Cloudflare, we don't connect the LLMs directly to the MCP clients. In our reference architecture, we actually go through an AI gateway. An AI gateway you can think of as a proxy to LLMs. And so what it can do is it can, the most important thing that it does, the most basic thing, is it allows you to switch LLMs. So maybe a task that you're doing needs a really cheap LLM, or maybe needs a local LLM, or maybe you want a really expensive, high-powered LLM, and you should be able to easily switch models from different providers. And that's something you can do with AI gateway. But the other thing you can do with AI gateway is cost controls. So in AI gateway, you can have something that says, you know, engineers can use this many tokens a day, and salespeople can use this many tokens a day. And so all of that is controlled through AI gateway, and it can give you like those kinds of controls about how much you want, how many queries can come from each individual person in the organization and track what they're doing. So that's how we sort of set up the MCP clients. And earlier I said, you know, people used to, like in some places, people do run MCP servers locally, right? And so that MCP client would talk to an MCP server that's sitting on their actual laptop, and then goes and calls out to whatever. In this picture, you can see GitLab in the bottom corner, for example, or GitHub. You can see that in the bottom, right? So that is a way to run MCP. But again, as I said before, running the MCP server locally is not the best because it really takes control out of your IT team and security team, which should be making sure that you have at least trusted, up-to -date implementations of these servers and not random stuff you downloaded off the Internet. Yeah, that's a lot of responsibility for somebody to say like, oh, this is a thing that I'm going to run here locally on my computer with my company data too. Like that's scary. That's a responsibility. Like the more you talk about it, that responsibility is scary. Yeah. And then, so now that we like sort of take this idea, we're not going to run the MCP server locally because how are we going to run it? So the way that we do it and the way we talk about it in the reference architecture, we have in our monorepo at Cloudflare, a whole workflow that can allow developers to build MCP servers and go through security checks and other checks before those MCP servers then get deployed to the whole company. And how do they get deployed to the whole company? We use Cloudflare's developer platform to host those MCP servers so that they're globally distributed across our network. And so any employee, wherever they are in the world will, when they're actually accessing the remote MCP server, they'll go to their closest colo, which will be running that remote MCP server and they can access it through that. So that's like the basics of how a remote MCP server would work. And really the reason to do it, if I'm focused on security, is that it gives a security team a way to audit and control and patch and upgrade these servers, rather than just kind of like not knowing who's doing what locally on their machines. So that's that part. Now, when we talk about MCP, if you look down, you see ZTNA access on that diagram. So that is around authentication. So you can have an MCP server that may sit in front of something like Cloudflare radar, which is our data repository that scans the Internet. There's no authentication to get into Cloudflare radar because it's all public information. But obviously, if you're going to be accessing the company's production code repositories, you need to authenticate and make sure you're a Cloudflare employee. So how do you do that? You can do that with a tool for authentication, Zero Trust Network Access tool, that's ZTNA, Cloudflare access. And so what we do with our internal MCP servers is that we build them, we host them on Cloudflare Workers, and then we put Cloudflare access in front of them, which means that that does single sign-on, it does MFA, it can do device context, it can say only allow people in these certain countries, all of these different features that you would expect from really deep, fine-grained authentication. And you can do through there and put that in front of your MCP server. And so that protects it from unauthorized access, which is obviously really important if you're letting it use your internal, you're putting in front of your internal ClickHouse clusters or something like that. And then the last piece there, you can see right in the middle, is the MCP server portal. So MCP server portal, it serves a bunch of purposes, and the very simplest one is the following. If I'm an employee of a company and I'm just starting to use MCP, I don't know what MCP servers are out there. It's really hard and kind of scary to figure out where they are, how I use them, and how do I connect to all of these things, and how do I set them up? So actually, and I mean, that is actually really, maybe the biggest barrier to adoption is like, how do I even connect to these things? So with portals, what you do is you just connect your MCP client to the portal, and that portal will expose to you, here are all the servers that you have access to as an employee of this company. And those servers could be some that we built internally and we hosted internally, and those could be actually third-party servers, because Slack and PayPal have their own MCP servers that they stand up for their customers, and you can connect to those through the portal. But I don't have to like, you know, find out how to do that and connect to them and do all that complicated stuff. I just go through the portal. So as an employee, it makes my life easier, but also as an IT administrator, this is another point of control where I can write policies about who can access what MCP server, I can log what they do, I can run DLP, data loss prevention, policies through the MCP server. So all of that is happening in the MCP server portal, along with something else really cool that we launched now, which is, yeah. Because there, I mean, I would imagine, I know that we have, I think there's 11 or 12 MCP servers that might actually have additional MCP servers. So that's a lot of information, right? Right, yes. A lot of tools to connect to. And if I'm right, I might be jumping the gun here, but are you about ready to talk about code mode? Yeah, I am. Let's talk about code mode. So this thing here, I believe, really only has two, it itself is an MCP server, right? These clients, when we look at it, it's connecting to this MCP server. And instead of showing all of these tools, we use a thing called code mode where it shows two, right? Yeah, so it's a way of reducing the amount of information that has to be processed when you interact with MCP servers. And it's a really cool story because we had a team internally that found a way to reduce the number of tools that are exposed from a single MCP server to just two tools, which are search and execute. Search looks at what tools are available and execute will then write a little piece of code that will call those existing tools and do something with them, right? And so by doing it that way, you reduce the context so much that you can have huge cost savings. That's just a very simple explanation. I didn't build this. I'm speaking for my colleagues who built this super cool thing, but then they thought, okay, so we did this for an MCP server that we built, but how do we make this useful for the world? That was the clever idea. They actually added it into the MCP server portal so that anything that the MCP server portal sits in front of gets this code mode optimization. And now you can interact with all these servers, which don't have code mode. You get it if you put them behind the portal and now you get code mode. And so that was something we released now, which is super cool. And that's like straight up full new feature of MCP server portals that went live this week. It's so cool. And you save so much money and so much like you can imagine in that bit where we're making those remote MCP servers. If we have a lot of remote MCP servers and we're hearing that people are using these internally, right? So it's kind of a way to do internal enablement too. And there's a bunch of them and how do you find them? And that search is really, really important. So putting this thing in front of it is really nice and you can choose. There's a, you know, I love running through this because you can kind of choose which ones you want to use in there too. Which one as a user you want to use. These are all offered to you. Which ones do you want to actually enable right now? It's really nice. Really, really nice. Right. And like, if you don't know, you're like, look, there's like an MCP server for Jira. I didn't know that. Maybe I should try using it to do Jira things, which like so, so good for discovery because, you know, a workforce doesn't necessarily know what servers are available and they're being added all the time. And so it's just a great way to kind of roll things out really quickly across the org. And I think that's one of the reasons why we've had such a broad adoption here, which is interesting because I don't know how true this is across the industry, but we have, we do not have it like MCP usage is not restricted to our R&D team. It is used by our go-to-market teams, used by marketing team, by our finance teams because now they have access to all these internal resources that they can interact with using an agent. And I think that's just really powerful and cool to not have it be so restricted just to developers. It is it, and it is such an unlock. And again, like if we are the way that you would do this in the past, right. It's like, Hey, here's this, we were talking about it. Run this locally on your machine. Like imagine handing that out to everybody. And then that ends up in this, like, that ends up in a dangerous place. Right. I feel like you could throw them back to the nineties. Like you were talking about, about like shadow IT. We're having a shadow. Is it a shadow MCP moment? Is that what we're having? We might be having a shadow MCP moment. So two parts to shadow MCP. One is locally run MCPs that people are downloading and running locally, which actually we don't have a solution for in the product yet. That's more like an EDR feature that I could see coming out. But if people are using remote MCPs in an unauthorized way, then you can actually see that if you use Cloudflare one as your enterprise networking stack, that's our SASE platform. So in our SASE platform, we have a secure web gateway that can inspect HTTPS traffic, TLS traffic, and guess what MCP traffic is? It's HTTPS traffic. And it has certain markers, like there are certain headers, there are certain formats, it's JSON RPC. And so in this blog, we also recently worked with some customers to figure out how they could use Cloudflare gateway to discover shadow MCP, like unauthorized remote MCP servers that are being used by their employees. And we can pick that up with a bunch of like DLP regular expressions on the HTTP bodies. Anyway, I'm getting way too in the weeds, but the point is there are some tricks you can use and you can pick out this traffic. And so the other thing that we wrote about was how to actually use a secure web gateway like Cloudflare gateway to pick out MCP traffic from the remote network. And that's something that we've been asked about by customers constantly. And I'm really happy that we were able to put those rules out there. And I really hope, you know, if you're operating a secure web gateway, please take a look and use the rules. I think they're really useful. And we're talking, this is like AI security for apps, or is this something else? Because there's another part of the blog post. No, there's a whole other part. Okay, let's talk about that. Now, the third, okay, the last thing I want to talk about is the following. So, okay, so the last thing I want to talk about is how we talked a lot about how, you know, Stripe and PayPal and GitHub have put out MCPs that their customers can use. And we're kind of at the point where we feel like every organization should be doing that. You know, we're moving into a world where these things are going to be administered to using like operated, administered using MCP. And so if you are providing a service to customers, that those customers are going to want to interact with, with MCP, it's much better. If you wrote the MCP server for your product, then you just let them like find random MCPs off the Internet. That is potentially a supply chain risk, a software supply chain risk. It's much better to have an official MCP server than to wait for someone to write one for you. And who knows who that person is and posted it on GitHub as open source. So for that use case, which is that I have a product that I'm building and I want to offer an MCP because I believe it's important from the security and the integrity of the product that I'm building. There's two things that we pulled out and we wanted to highlight. One is you can, of course, build an MCP server on Cloudflare and host it on Cloudflare workers. And then it's globally distributed across our entire network always. So it's not like you have to deploy to every region and manage in every region. It's all globally deployed. So that's really good for latency and performance first. And second, we have a feature called AI security for apps, which is in our web application firewall in our WAF. So you can put our WAF in front of those MCP servers with this feature turned on. And what will it do? It will look for prompt injection and other AI attacks that are trying to connect with your MCP server. And so you can have the hosting on Cloudflare and then the security on Cloudflare as well. And we just really wanted to highlight that a lot of organizations need to be thinking about how they're going to provide MCP to their customers and to the people who are administering their products. So that was the last part of this giant reference architecture. That is so awesome and so thoughtful, right? I think WIRSA is moving so fast. People are able to build MCP servers so fast. They probably will build yours. So I love that. I love that we're thinking, here's how to do it. Here's how to do it safely and securely, giving you a nice way to go. And then also how to set this up internally. Thank you so much for this blog post. I think there's a lot to digest here. And I hope that everybody goes and reads it and thinks through the kind of problems that you have. And let us know, literally let us know, what more you'd like to see. Because we're all driving this, right, Sharon? This is all coming from customers about how they're seeing MCPs being used. It's coming from customers. It's coming across the business. Yeah. And we're talking to customers about it all the time. And I guess the last thing I would say was that I kind of started this whole effort. And part of the reason is I did something like, I don't know how many customer presentations on this material. And I just decided like this has to be written down so that we can scale it out across all of our teams and all of our customers. So I hope it's helpful. So thank you. Well, thank you, Sharon. And thank you. Thank you for being here. And thank you, everybody watching this. Please make sure that you go to the blog. There's so much stuff happening on Agents Week. We talked about a lot of products in here. AI Gateway is getting a bunch of new stuff this week. You should check it out. Lots and lots of fun stuff. Lots of creative, innovative things that you're going to want to get your hands on. So please, we'll hang out in the blogs. There's a big hub page of everything that's launched this week. Sharon, thank you so much for being here. And I can't wait to be on another one of these with you. Thank you. Thanks, everyone.