People, devices, and agents: How Rippling’s edge architecture evolved

I'm Ram. I'm a software engineer at Rippling, and we'll be talking about how we leverage Cloudflare Workers and the rest of the workers platform. But first, a little bit about me and Rippling. I focus on IT products at Rippling. If you are not familiar, we actually build at the very core, we have employment data, but then we attach a whole bunch of things to that employment information. Everything from single sign-on and surveys and payroll, et cetera, et cetera. It's a unique model, but I think it'll start to make sense as we go further. And a fun fact about me, I was actually an early employee at Cloudflare way back in 2011. So this is almost 15 years ago. So it's a bit of a full circle moment for me coming back here. And also it was a very different Cloudflare than now, but it's also a little bit similar in the sense that at the very core, there's the servers that run every single thing, as you heard during the keynote this morning, and then you attach more things. I think Rippling has some similarities there that you'll find, but let's jump into the talk. So I'm going to cover a few themes here. One is how do we go, how we went over the years from serving human scale traffic to then serving machine scale traffic as part of our IT suite. And then finally, as we are living in the agentic era, how we're serving agent scale traffic. First, we'll start with the people. So the core thesis here with the people problem is we have companies that use us for their services. And based on data residency requirements, they can choose which region they want to host it in. As an example, a company in EU might choose, we want to host our data in the EU. What that means is you need to route traffic for employees of that company to EU regardless of where they are in the globe. So it can't be to the closest region, which is typically comes for free with a network like Cloudflare, but you need to do a special routing here based on which company they're accessing the data for. And to illustrate this, I made a slide, but I want to apologize in advance for this, but I wanted to convey the concept of an employee in EU trying to visit a client in the US. This was AI's impression of it, of what that could look like. So when someone's travels, you still need to send their traffic back to their company's region that where they have established data residency. So what that looks like, if we take a two region example, we end up leveraging Cloudflare workers along with a simple KV lookup. I think it's probably the simplest of the use cases you can think of for a setup like this. The data doesn't change often. And once a company chooses the region, it remains the same. So that's kind of where we were a few years ago. Now let's move on to the next use case, which is our device management product, where we, as part of our IT suite, we have companies that can set policies on computers. So an example of that could be, I want to lock someone's screen when it's been inactive for 10 minutes, as an example. And also people could move in and out of groups based on where they are in the organization. And that could trigger more commands as well. Or a company might write new policies. All of this needs to then be sent back to the device so that the command can be executed there. And when we initially built this, we built it around polling. So each device started polling for new commands at a set interval. However, as the number of devices grew, it caused a bunch of challenges for us, especially around scalability and costs. Because as you can imagine, changing someone's policy doesn't happen often. However, the device is asking extremely frequently, in order to stay up to date, what is my next command? And the other problem we were even more concerned about is certain sensitive commands need to go out to the device more frequently or very quickly, instead of waiting for that one to two minute late polling interval we had set in order to meet the scaling demands. So this is a another illustration, powered by AI, of an IT admin trying to initiate a remote desktop session. I don't know why the IT admin is in a lawyer's office, but career goals, I suppose. So they're essentially trying to, as an example scenario, they're trying to help an employee with a computer issue. It would be super awkward for them to get on that call and say, oh, let's stare at the computer and watch for two minutes and see if it initiates. You really want this to happen in a matter of less than a second or a very small number of seconds. So what you truly want the architecture to look like is you want to push commands from your control plane over to the devices. But the moment you say push, what you're really saying is I want persistent connections. You can't really push unless the device is always connected. And when you really say persistent connections, you're really saying I want to do one of these things. It could be long polling, it could be WebSockets, or you could even go a little niche and do something like MQTT, which is more common in the IoT world. But the issue with doing one of those is you end up with a bunch of persistent connections back to your control plane. This, I think I'm only showing 12 or 15, but imagine hundreds or thousands, hundreds of thousands going into the millions of connections, just staying open all the time. And the reality here is it's not even just your control plane. If you look at a single machine's request path, it looks a little bit like this. You have your edge slash CDN or whichever service we use Cloudflare, then you have your load balancer. For us, that's AWS. So each machine is now opening a persistent connection to each of these layers. And that leads to a bunch of problems. One of them is cost. Each of those services will charge you for a persistent connection. If you ever look up ALB's pricing, they charge you for how many hours the connection was open for, which is not common with HTTP requests when someone's browsing on their browser, because you just send the data and it's done. So that's one of the challenges. And then your app servers help to keep it open. Then you have to run enough app servers to do that. And then there's a bunch of technical complexity that comes with that setup, which is more often than not, which is this was the case for us. We have a pretty massive 20 million line monolith and trying to keep that running to just to hold connections is extremely wasteful, because what you truly just want is keep the connection open, right? But let's say you solve that problem. You still have the issue of which of these servers should I send my command to? Which specific device is connected with server? You need to know the mapping. And if you don't want to keep track of the mapping, you then have to do a broadcast to every server. Or there's other challenges, like if you do a deployment to the service that's holding connections open, you have So where we were going towards is essentially, what if we wrote a small WebSocket server in front, and we had extremely enthusiastic people that were like, Oh, go as async IO friendly, and it's lightweight. Or some people were like, Rust doesn't have a garbage collector, it's even faster. But there's a few challenges with that. But first, I want to say both are great languages, we actually use them in our stack, we use Rust for our device client, these go in parts of our control plane. But the issue is not the language, it's actually the part where you're running the WebSocket server, right? And this is maybe a bit of a spoiler alert, hope you can cross over the hump and not be stuck like we were in the middle where you consider running a WebSocket server, which is a very tempting technical challenge, but maybe not the right business problem to focus on. And I'll show you where we landed, which is our goal was, let's push commands to the very edge of our network, and have a network that has millions of or I don't know how many servers Cloudflare has, but I know it's a lot more than how many we do. So maybe someone that has 100 to 1000x more servers should hold the connection open. Whereas we just sit there push commands, and the architecture we landed on is use a combination of DurableObjects plus Worker. So the Worker helps you act as a management interface, where you send commands from your control plane. And DurableObjects serve as essentially a addressable state for each of your devices. So the way we did it is each device has an ID, you have a DurableObject for each of those IDs. And the moment you send a command, the DurableObject ships it over the WebSocket connection. And back to the cost part I was talking about earlier. DurableObjects have this nice feature called hibernation. So if you do WebSockets to DurableObjects, but if you're not really sending anything into the DO itself, keeping the WebSocket connection open is actually almost free. It doesn't keep the DO on. If you haven't come across this, I think we didn't know about this feature either until we started doing some research. Definitely check it out. Hibernation is a nice add-on for DurableObjects. So that's the story of how we solved our devices problem. Now let's get into agents and how we are tackling agent-scale traffic. So first, just to give you an intro, we have a chat interface just like everyone else. I feel like this is the new way of software. Point and click is the old school way. Now you can talk to software. And more specifically, I think there's a lot of chatter around LLMs and GPUs and innovations. But I think the real innovation here is the human-computer interface that has gotten so much more advanced. Here I'm asking it three different things, which is, I want to take some time off and I want to see how much time off I've taken. And I also want to see how many meetings or interviews I would miss. This is not an interface that existed before for any of us. You had to kind of go piece this together, right? But how do you actually... Actually, before we get into how do you make it happen, and spoiler alert, we'll talk about the Google Calendar connection. But briefly just to show you, it showed me what is the time off taken, and then it showed me a list of meetings, like standard stuff. You've all seen this. But also I got a little more curious and asked it, is this time off I'm taking, is that too much? Like, what's my manager doing or what are my peers doing? And it instantly blocked me from showing that information. Hey, I figured out who your manager and peers are by traveling the org chart, but I don't think you should see that information. And so that's our view of how inference is pluggable. But once you connect your tool calls, and the permissions, and your system of record data is the powerful thing. That's our view on how we're approaching AI. But let's get into how we're handling agent -scale traffic. So the way you connect your agent to third party data is through MCP. Whether you like it or not, that's the protocol that's the most common. And there's a little bit of irony there. Like, we spent a decade or so talking about REST APIs with weird formats like posts to slash V1 slash calendar. And all of a sudden when it's machines, we're saying, oh, we'll give you a nice tool called create calendar event and we'll write a description and so on and so forth. But that's aside from me. But let's move on. So there's MCP, but then MCP at a company, which I think are very different levels of how you deploy this. More often than not, at a company, you want to know which MCP should be used by people, or maybe even which tool calls. Maybe a company is okay with, I want people to run Slack reads, but not writes. I don't want them sending messages. Or maybe you want to do different settings by department. And at the very least, you want to have an audit log of what's happening. And the way we built this is because of our org data, we're able to give our customers the interface. However, we still need to run this high traffic agent to third party traffic. And the way we architected that is you can go one of two ways. You can say, check with my control plane each time, which things are allowed. Or you can say, I want to snapshot those policies, push them to my data plane, which is running along the top. And you essentially end up turning this problem into an HTTP proxy problem. Essentially, that's what it is. There's HTTP traffic going left to right at the top. And then you have your customers giving you policies and so on and so forth. The moment you do that, you start to see a familiar shape. You can put that traffic on workers and durable objects one more time. And I think this architecture has served us really well, because oftentimes this traffic is very spiky. And just two weeks ago, we've seen someone write a crazy automation that was doing a ton of MCP calls to Google Workspace, which I think if it hit our control plane, we probably would have had a downtime. Versus having workers and durable objects handle that has been a thing we haven't had to think about. And once you have this setup, you can open that to not just our agent. You can do it for any agent, right? It could be Cloud, Codex, and so on and so forth. And this is where it starts to become a very general MCP gateway. And how do you then bring the audit logins back from the edge? Now that your traffic is actually living at the edge, we end up using pipelines and then R2, which is the equivalent of S3, and cheaply store it there. And then bring it back into our control plane for querying, so that we can apply the right permissions, and so on. And then, as we started working on this, and had customers looking at this, one of the questions we got was, how do I host my own MCPs? Because my engineers internally have built their own custom code to expose tool calls. And oftentimes, this is untrusted code. We could have them host it, but then you need to tell them where do you host it? How do you do auth? Or you could have them give you the code, but where do you run the untrusted code? And also remember, we're running all of this on the edge. So timing was amazing when Cloudflare came out with the sandbox feature, which is you just give it an image, and the sandbox runs closest to wherever your request is happening, and you get to serve customer traffic. That took away another problem for us. And then lastly, while this was all happening, we had another team that was building Rippling's own MCP server, which from once we heard of it from the get-go, we wanted to put it behind the gateway. Reason is the same way a company might want to control which tools are allowed for who, they might want to do the same thing for what's allowed on the Rippling MCP server as well. And we had a massive amount of tool call sprawl, which was like, we have 30 to 50 SKUs where we sell many, many, many services. So instead of sending all of those, we ended up leveraging code mode, which is a way for agents to discover and write code. But again, you then end up with dynamic code that's getting generated on the fly, and dynamic workers nicely help encapsulate that as a runtime. So that ended up solving that as well. So in essence, we ended up with this architecture where we had our own MCP server, we had third-party MCPs and customer-hosted MCPs, all behind the MCP gateway, using pretty much Cloudflare's infrastructure, while we focus all of our efforts on our control plane, which is what we think we bring to the table. So that's on MCPs. And then you'll see a similar shape here, similar to how your agents are making tool calls, they're also making inference calls, right? And those again are HTTP requests going back and forth. And you have many providers on the right-hand side, and very similar to how a company would want to figure out what rules they want to apply to tool calls, there's a very high chance they want to figure out which models are allowed. As an example, maybe Fable and Astra are not allowed for most people at my company, or how much can they spend on LLM budgets, and so on. And again, once you push policies up to your data plane, you get a similar shape one more time, with workers and durable objects. So I think we're really glad we invested in that years ago on routing our people traffic. However, with AI Gateway, there's a lot more to it. You actually need a pretty large model catalog, especially with open-weight models being able to be served for extremely cheap. And also you want some guardrails, and you probably want some smart model routing. So these are not really pure HTTP traffic things. These are like deep technical trenches. And we're working with the team at Cloudflare to figure out how to bring this all together. More to come on that in the next few weeks. But that's about agent -scale traffic. So thank you. I think we got to cover human-scale traffic, then machine-scale, and finally agent-scale. So thank you all for listening.