Your AI agent is a distributed system

It's nice to see you folks. My name is Sunil Pai. I used to work on the AI Agents team until about a month ago. And then I started working with Ali. But the last two years of my work are now in the slide deck, I guess. It's kind of weird. So the way you make slide decks in 2026 is you tell, open up cloud code, and you say, just find everything I've done in the last two years and make it palatable to everyone. And make it look like I know what I'm talking about, which is wonderful. Before I get into it, I need a favor from you folks. Kyle, actually, I need you to help me out here. I need you to make it look like I'm having an argument with these people and take a photo because that will make this the best talk that exists. Can you take it? So you are really mad at me. Okay. Can you just quickly take a photo? Yeah. Okay, cool. Okay, I'm going to send that in work chat. Okay, awesome. Excellent. So my name is Sunil Pai. I've been with Cloudflare for about five years, including the startup and stuff. And I am a React developer. I was a front end guy. And I was like, I'll hang out with the back end kids. And I thought building AI agents on durable objects would be like a two week mission. Just get in, write some code. And I ended up taking so much of my life. And this is a little journey through it, what I learned, and how I could tie it back to lessons that I'd already known before. So first, a quick story. Also, Opus does these animations now for free. Like, this is just a React app, like in my browser. Just love it. The new world means that people are doing deploys multiple times a day. There's no idea of a release cycle and everybody saying, okay, nobody do any pushes for the next hour while I do a push. We have customers who are, I personally know of at least one customer that's doing about 200 deploys a day. And we heard that there is one customer that's doing four digits, which is what, like every few minutes and stuff. And the thing about agents, unlike just request response cycles, your standard e-commerce websites, is that they're actually long running processes in the background. They're doing things even without a browser connected. That was the big late 2025, early 2026 thing. So this is an example of a customer who was using the Cloudflare Agents SDK, and it was going on for minutes. Yeah, roughly, it was getting close to about two hours. And they'd already done, like, three deploys. And then the Cloudflare Agents SDK decides to break it, saying, oh, this process has been going on for too long. It's been going on for too long, so it's probably hanging, so we're just going to kill it. Which is dumb as hell. That's my quote, just so we're clear. I'm responsible for that happening. I had it in the thing where I'm like, oh, the moment it hits, so much time it dies. And while revisiting, it's weird because, like, you do want to have some sense of boundaries, and it turned out time was the wrong one. So we'll get back to this use case, but I want you to know, like, this was a big goof up that led me down a bunch of stuff. This is a very 2025-ass quote. A coding agent that dies when your laptop sleeps, that's a tool. I want to understand, has San Francisco discovered amphetamine on their laptops, or do they still keep the screens open on the buses? I loved the three weeks when Twitter was just hundreds of those photos. But the idea is that you want to take your agent from something that you're running, like an app, and you want to treat it like infrastructure that's running in the cloud. Not on your machine, basically. And I built, well, we built something called Project Think in, I would say, in April. We shipped it in April, where we took what we called an opinionated take on what an agent look should be running on durable objects. Some folks here might know that I'm the biggest durable object fan in the world, which also means I hate it the most. Like, I have deep familiarity with everything that makes it suck. And that's what true romance is like, right? Like, after you've spent 20 years with your wife, I'm coming up on my 16th wedding anniversary myself, and we tell each other that if any of us angers each other, we can destroy them overnight, just because we know where all the skeletons are hidden. Okay, so this is what a standard agent loop looks like. It doesn't matter. It's just pseudocode at this point. I didn't even write it. I'm like, okay, it looks fine. And it seems it's nice to model an agent loop as a while loop. And you say, for every step, you pass it whatever your messages array is so far. And if it's done, you return the text. Otherwise, if there are any tool calls that were returned in the response, run that tool, add it all to the messages array, and repeat. Wonderful mental model. It's kind of weird, though, because we spend all this time working on optimizing request-response cycle, and now we're just like, oh, here's 600 messages. We're going to make it 600 messages, pass it back. We're going to get another 602 messages back. It's vulgar is what it is, but the entire world, like commerce, capitalism runs on this request-response cycle right now, which I thought is quite funny. But there is a rough edge with almost every line in this. This call is the most expensive call, because inference cost stuff, because everyone is using the top frontier model at this point. And it's not usually awaited. It is streamed in. You're built per token. It can stall. It can decide it's doing something else in the background. Might be an async call in the background. The tool call is another one, which is some of these things are doing payments. They're making destructive calls on databases. And basically a complete mess where if you don't actually run it just once, then all hell will break loose. Some of them might never run. Just a number of things that can go wrong here. And even worse is we spent 30 years building these stateless infrastructures and libraries and an ecosystem around it. But it turns out if you actually hold it in memory and your isolate or your server dies, you're screwed. Because if you want to start it again, it is a very, very expensive process. And even the user experience sucks. I don't know if you remember the first versions of ChatGPT. If you close the tab or you closed the app, it would die midway. Or sometimes it would wipe out everything that had happened and it would come back in. Just completely awful and very, very expensive. Over the last year, some other things have happened. So meter, is that what you folks call it here? M-E-T-R, the research group? They've discovered this Moore's law for agent execution where every few months the time basically doubles. And I think that actually it's not a linear line. I think it's quadratic is what it is. It's been doubling, but I think it's now tripling, quadrupling. And that's the hope inference companies have for you, right? They'd like you to set up a machine that just runs 24 hours a day and the only thing you're doing is providing context, taking stuff out. Oh, that's how they're stealing software engineer jobs. Okay, that's how they're trying to get the jobs away from all of us. They're like, yeah, it's just running on these servers all the time. Another nice thing that's happened is I remember, especially in 2025, there were a number of agent frameworks that just, I don't hear about them anymore, but they were responsible for bringing about the common stack for what an AI agent looks like. So you have a harness, you have sandboxes that you can spin up and spin down. You have a durable log where you keep all your messages and not just the inference messages. You might have some UI specific metadata, et cetera in it. Then you have APIs for running them as managed groups, like Cloud Managed Agents I think is the most, I don't know whether it's the most popular one, but it at least framed the phrase. And the other, there are durable execution services and libraries that people use for building in guarantees, consistency guarantees, et cetera, for these things. I'm going really slow on time. And then earlier this year, even the MCP spec went stateless. It's why for the longest time, the best MCP servers ran only on durable objects, which was fun. It pissed off a lot of our competitors, but now it's stateless. You can run it on Vercel, it's fine. My point being, we discovered all these problems that were running in a single machine, and it turns out that all of these things are solved problems in the world of distributed systems. The idea of a process being killed mid-turn and what do you do when it comes back? Well, that's just crash recovery. That's why you need a writer head log. That's why you need some form of replication over there. A tool called being retried. I love $12 word, idempotency. I used to say $5 word, but inflation is inflation. The idea of idempotency, it's just a wonderful word. It's what software engineers with about three, four years of experience tell each other to show that they know what they're talking about. Beautiful word. The idea of having two sub-agents working on a single source of truth. Two or more, really. This is the split brain problem, which is why you need a single writer and you need queues and you need some way of synchronizing changes. Waiting days for a human. That's just it. When you do approval, you can't just do a wait in wherever your code is waiting for a human. They might come back days later and click the approve button. Now you need some, again, you need a durable execution engine. There are durable timers. What happens when that approval is not even valid anymore? You work on how do you deal with rate limits, runway spend? How do you attach budgets for all these sorts of things? The one that I really don't think we've actually solved, which is how do you deal with context windows? Even a million tokens is not enough anymore. I've been working on, please go downstairs to expo floor. There's this weird thing called city that remembers, which is a big isometric thing and I've added dragons and a big mecha kitty that you can now use and hopefully by the end of today, you'll be able to kick the buildings over. Whoever is videoing this right now, it's an actual exhibit downstairs. Point being, I've been doing that in a single open code thread for about six weeks now. I have not looked at what context compaction it's at. At some point it starts doing stuff and I'm like, okay, I'm going grabbing coffee, come back and it just works. Anyway, so new names, old problems, and it turns out the answers for them are the same too. The thing I like about agents are that they are one-to-one, right? Opus came up with this extremely basic-ass analogy about it being a restaurant and diners, etc. But an agent is a personal chef. Every user gets their own task, their own context. At some point it'll do generative UI so the website looks different for every user. Sure, whatever. From an infrastructure angle though, it's interesting because if you try to do it the traditional way, with a system that needs to keep the machine all the time and maintains everything in memory, then you end up with this very inefficient architecture where you're just burning electricity on machines. I say durable objects, but something that gives you an actor model-style entity system. It's nice that these things only wake up when you need them to. Durable objects are actors. Quick show of hands, I'm saying things assuming people know what durable objects are. Enough hands went up, yes. Okay, so durable objects are actors. Fun fact, they're actually called actors in the Cloudflare code base. Famously, durable objects are the most piece-of-shit name for technology ever. But they're actually called actors, and I should you not, when they were going live in 2020, that was when the beta blog post went out, Kenton did try to call them agents, and he was shot down. And we have a thing inside Cloudflare where we're like, yeah, Kenton was right six months ago. We should have just called it agents. I suspect it would have affected the stock price. Anyway, so because they're actors, they have a bunch of things that are associated with it. There's an identity, an ID. You have one thread that works on it. It has storage, and you can hibernate. Hibernation actually came a couple of years later, but the fact that it can sleep when it's not doing any work, and it wakes up on messages. Okay, I'm going to start scooting through these things because I'm actually going quite slowly on time. Your mental model should be that there are some things inside a durable object or an agent that stick around even if it shuts down and comes back to life. So on the agent class in the agents SDK, it's this.state. You have a SQL database. All our schedules and alarms are durable and work even if the isolate goes down. We have workspace files. That thing has been renamed to computer, and I think we actually shipped it just today. And then on every WebSocket, you can actually set a little bit of state that persists across hibernation, which is nice. The things that don't are regular JavaScript variables, class fields, set timeout, in -flight fetches, et cetera. The annoying thing about durable objects, and that's just a distributed systems problem, is that there's no on-evict callback. You can't really know when a system is shutting down, much like a signal, which there's good, bad things about it, but it does mean that you need to be prepared for all of these things. By the way, just to make it clear, the big punchline here is that the agents SDK takes care of all of this shit for you. It just makes durable objects actually usable. Anyway, so there are five ways that these things break. I will just go through all of them. I mean, I have a slide for each of them. One is, how do you keep a durable object alive when you're doing a bunch of IO? There's a .keepalive function that you can wrap your functions in, and it uses a durable object alarm, and it just keeps the actual instance alive, even if it feels like dead air. Durable objects itself are shipping things now that make that the default behavior, but until it did, I had .keepalive around. I have this data structure called a fiber. Again, so it's what someone with 12 years of experience tries to say. Well, I've introduced a new kind of data structure, which I completely stole from React, by the way. And there are tasks where it looks very much like any of the other durable execution engines, and you can use these things to wrap functions that you think should live through eviction, et cetera. We did have a recover function, and it was about a third of the entire code base. It was mental, and then we just deleted it. This is Opus now quoting an Opus quote from back then. It's self-referential, Opus. But we removed the recovery mode itself, and instead we just simplified the data structure down, and we said, no, as work happens, just keep recording it, and when it recovers, have a strategy to move ahead. The second one is what we call durable streams. We hate, I hate, I absolutely hate having to do, I forget what it's called, recoverable inference streams. Anthropic used to have a first -class API for it. They removed it. OpenAI still has it. I kind of wouldn't trust it to stick around. The idea being that if you actually do have, if your thing does blow up in the middle of LLM streams, how do you recover from that moment? And the way that you do it is a shit ton of code. I'm not even going into detail. The point being that there are ways to do it. But I did write a blog post, and we are shipping in Cloudflare AI Gateway a way to enable this for every provider and every kind of inference. The idea is that you will back every single inference stream with a durable object that your agent talks to. So if your agent actually shuts down, you haven't lost the stream, you reconnect, and you get the rest of it back from it. I'm not the kind of guy who says you should read my blog, but you should read my blog. Again, back to idempotency, especially when you're doing stuff with money or just database destructive actions, you want the idea of adding an idempotency, you want to pass an idempotency key when you actually do stuff like refunds, etc. I don't need to spend this very standard durable execution stuff. And when I say very standard, I'm saying it means your coding agent knows it, so he'll take care of it. He'll see the API and be like, okay, I'll figure out what the thing for it is. Something, something. This is one of my favorite implementations. Every durable object has only one alarm, but really you want to manage a number of things with that single alarm. So we implemented the schedule API, we have a .q API, tasks, keep alive, destroy that, maintain a SQLite table. And every time you try to say, hey, do this thing after two minutes, do this thing after three hours, do this thing at like 9 a.m., it saves it in that table and the single alarm just goes through this and runs it. I love it. This solves so many problems. I don't work on the durable objects team, but they keep promising me that this will be a first class API. Until then, you can use my implementation. That's fine. Back to the example that we had. It turns out that the mistake that we were doing was that we were trying to think in terms of time, but that means nothing in the world where something is taking 30 seconds, 30 minutes, 30 hours, 30 days, 30 years, who knows, right? You want to be able to tell the agent that some work is happening, and every time some work is happening, you want to reset what your counter is, and that is what your budget should be. This is what the API for it looks like, and you can actually pass. There's a hook saying, hey, should I just keep recovering based on the context, which happens every time the thing wakes up. In general, you can just say return true. It's fine. I say this, but at least there are two people inside this room who are pissed off with how much durable objects cost if you don't keep an eye on them, and one of them is sitting right there. Oh, there's another one right here. There are more than two people in this room. Regarding concurrency, this is my absolute favorite feature of durable objects, which is as a dumb JavaScript programmer, I do not have to worry about multi-threaded implementations anymore. Everything happens on a single thread, put it in a queue to make writes. Make writes, yeah. These are actually two notes from Cognition and from Cursor. Did it happen because I spoke to them? I don't know if I'm allowed to share what I shared over there. Like I said, the great thing about durable object is your mental model is that it's happening in a single JavaScript thread, and you do stuff. A thing that we did argue about a lot internally, especially after we shipped durable object facets, is should we have multiple threads in a single durable object and have every thread be a facet? Turned out to be annoying. Should we have all chat threads in a single DO? Also turned out to be annoying. So instead, what we do now is multi-layer, where we have a user hub, one durable object that maintains all the threads, and then having a durable object for each one. It doesn't matter. Spin up a million, billion, trillion of these. If you end up reaching a limit on this, your next step should be to tweet at Cloudflare saying, oh, you guys did a big, you talk a big game about durable objects being, you should be able to spin up a trillion and it didn't work, and Dane will get pissed off and then message the rest of us and we have to fix it. So that should be, you can do infinite ones of these. The other thing, and I'll quickly go through these because I assume there are other talks, with strategies like code mode where you can actually write code that calls tools, you can take 1.17 million is how much context tokens the entire Cloudflare API takes, which means you cannot fit the entire Cloudflare API in any inference provider right now, but you can bring it down to 1,000 tokens by using things like code mode. And the beautiful thing about dynamic workers, of course, is you will never leak your API key and suffer, you will never leak your API key because it's just not available for it to think. Anyway, so five rules for agent architecture that the agent's SDK and your agent will take care of you. One agent, one actor, one durable object. Make sure that your, the idempotency thing is something that you have to take care of. Base it on a user ID, a time, some kind of transaction value. Make sure that any work you're doing is being written down in a database because that's where it sticks. Make sure you're budgeting for work, time doesn't mean anything in 2026. And make sure you use things like dynamic workers for actually running your, for what you give access to your agent. I did want to show a little demo here, but I'm out of time, which is what I feel like the future of AI application should look like, and they're not chat agents. I built a thing where you can embed an agent into existing apps like Ableton, et cetera. In fact, that's my demo. It's a little digital audio workstation where you work alongside AI instead of it taking over for you. It's on my blog. I don't even care about this. The point being that all the new problems that we discovered in the last 24 months or so are all old problems in very old textbooks, if you bother reading books at all nowadays. And you will find the answers for scaling these things to millions, billions reliably in those books. That's my talk. Thank you very much.