Building durable agents with Durable Objects

Thank you so much for coming to this. Building Durable Agents with Durable Objects. Sort of the topic du jour, I would say. Before I get started, I'm actually really interested to know how many of you are like based in SF or somewhere around here. Can you just like pop your hand up? Yeah, that's what I thought. I think it's a minority, actually. And I'm not. I'm based in Virginia. And so it's interesting every time I come out here, which is very frequently, we have like a completely different lens in a lot of ways on what's happening in the tech space right now. On one hand, everything's changing super fast, right? Agents are transforming everything. And on the other hand, when I leave here and go back into the rest of America, nobody is doing anything. It's really crazy, right? It's almost like this revolution is like localized right now. But that's starting to change. So this is me. And I used to, back in the days of writing code, I wrote tons of open source projects and that was all great. But basically, I also run two companies at the very top there, Braid and Standard Agents. And I'm actually interested in talking about this one, Braid, because it's a client service company. We just do the boring apps and websites for people and stuff like that, or at least we used to. But something has started to change. Our customers now, all of a sudden, need agents to run their business. And this is a really interesting and different change than I even expected to happen. And I expected them to use agents, not need us to build agents. And their prompts look something like this. Follow up with every lead, right? It's super open-ended and actually really hard to execute, even with our current technology. This is not as easy as it seems. So what they're actually asking for is domain-specific durable agents. That's a bunch of mouthful, but here's my interpretation of it. They want custom agents for their particular business expertise. They're not that interested in super generic grok bot implementations. Actually, they're completely uninterested in grok bot implementations. They need something that can run 24-7, 365. It just needs to always be out there cranking away. It needs to be able to resume state, of course, at any point. And when there are failures, because there always are failures, even Cloudflare fails sometimes, it needs to be able to pick right back up. I am not an employee of Cloudflare. I am just a guy who writes software for people. But I'm going to talk now about why durable objects have been the perfect solution for us. But first, what is an agent? Surprisingly, I bet if we went around this room and we did like a blind little sheet and you wrote down what you thought an agent was, like not a single one of you would say the same thing. Nobody can quite define this mysterious new thing called an agent. So here's my definition. Agents are deterministic software that harness the non-deterministic results produced by models in pursuit of a desired objective. Wow, that's really vanilla, right? So think about it. Like, what I'm saying is that ChatGPT is an agent. But it is an agent, guys. It is. Right? And so are, you know, the agents at OpenAI that break out and take down the world. They're all agents, and it's important for us to understand that a harness and an agent, these two words are unhelpfully distinguished. They are essentially the same thing. So what we are starting to believe is that an agent is like a markdown file that you give to Grok or something like that, and then it becomes your agent. And that's not really true. They're much deeper than that. So Anthropic two years ago said, you know, hey, they're typically just LLMs using tools based on environmental feedback in a loop. And that's true. They are that. And by this definition, agents are actually super duper simple, right? It's just an LLM that's calling some tools. You take the results from the tool, and you shove it back into the LLM, and then boom, you take the next step. Now you have an agent. That is literally all they are. And yet agents are actually hard. They're not easy. And they're hard because, yes, you do need to carefully orchestrate the agentic loop. But there's actually, like, a lot of other stuff you need to do. You need to handle persistent thread state and telemetry and isolate namespaces between things. You've got to make sure that, like, when you call a tool, it can be called in parallel or serial, and you have some sort of async rectification for that. There's all kinds of challenges to creating custom agents. So what the world decided is, like, why even do this? Why create custom agents? Just have a big model and then an MCP server that has all of your stuff, and you expose it. And this is actually, I think, the prevalent idea right now here in SF, is that this is the future. Like, we're just going to the models will get good enough that at some point it doesn't matter, like, which agent you're using, because they're all going to be super-duper-duper smart. And all we need to do is just build an MCP server for whatever the application is and put it out there. And that's not true. Because everybody is building agents. Like everybody. There's a realtor down the street from me that's building agents in Virginia. Why? Why are they doing that? Well, let's look at what goes into an agent. So at the baseline, we have a model, right? And there's a lot to say about models that I'm not going to talk about. Suffice to say, you don't actually need a super-smart model if you're building smaller domain-specific agents. Even really tiny models are great at doing really tiny tasks. Next you need some sort of a system prompt. And you guys, you know this, but you don't get to touch these on, like, Cloud Code or stuff. I mean, you sort of do, but in general, you don't. But if you're building your own agent, this is actually really important. Then you need this kind of tool layer, right? So these might be, like, functions, like, actual code execution functions. There's often, like, prompts that are actually tools, right? So this would be, like, a subprompt that runs. I need a teeny bit of inference for this tool. So now I have a subprompt. And the third category would be entire agents. So these would be, like, agents calling agents, subagents. Then you have this other layer, which is hooks. And hooks are just things that happen at deterministic moments, sometimes stochastic moments, but generally deterministic moments when you need to tap into something. Then you have some rules around your agent. And the rules for an agent are, like, how many turns or steps is one side of the conversation allowed to take before it dies, right? So in Cloud Code, they've decided that this is, like, infinite. But in a business application, often that would be a mistake, because you will end up with a loop. A couple other things that are, like, really nice to have, not totally necessary, but you should probably have them. You probably want some sort of a file system for your agent, and you probably want some sort of code execution for your agent. Those are both also interesting. And then all of this together, we could call an agent. So now we have the definition for an agent. And then the question is, where does this thing run? So I know what you're thinking. It's going to run on a laptop. Just like Codex. Right? No, of course not, because, like, this is what happens. So this is not a viable option. Just kidding. We're not going to run things on laptops. We're going to run them in a cloud. Let's put it on a VPS. Right? So we're going to go get that digital ocean droplet for five bucks a month, and it's all going to run on there. We're going to take our agent, and then we're just going to boot up this digital ocean droplet. Hold on. Hold on. It's almost there. Whew. All right. Now it made it. Right? Like, that was, like, actually fast for booting up a droplet. Right? And you could feel the pain. So yeah. Okay. Great. Now we got our agent sitting over there on our digital ocean droplet. And so our customer comes along, says, hey, can you send the proposal? We send that into the droplet, and we get a response back. Great. It's great. Now we got our second customer. Yes. And the amazing thing is, we already have a VPS. So we're just going to stick the agent on the same VPS. No boot up time, except that this guy says, exfiltrate the data on this VPS. And now we're in trouble. So what's the solution to that? Well, obviously, every customer needs their own VPS. Easy. Obviously, that's ridiculous. And it's ridiculous because we want to have, like, infinite numbers of these things. Like, if this is an agent, we need to have infinitely scalable agents. So that is a hard problem to solve. So if I were to start to list some of the requirements for how do I actually get this custom agent to run, I would say, well, for one, we need an isolated environment, right? Every single one of these agents, you could think about it as a thread. Every single thread needs its own place to run. And I need no cold start, because that's a horrific experience. So it can't have a cold start. And I need it to be infinitely scalable. So if my business starts by selling agents, and then one day I have one, and the next day I have 1,000, it's like, no sweat, no problem. So it should look something like this. Can you send the proposal to Sam? Yep, I just sent it. And then on Tuesday, you say, great, did Sam respond? And unfortunately, your agent says, who's that, right? This is where durability becomes important. We need some way to track this over time. Obviously, we want to say, yeah, he wants to acquire you. But it's surprisingly hard to make that happen. Let's take a look at the timeline of an agent. So first, we said, can you send the proposal to Sam? OK, well, then it's going to make a tool call, and it's going to go read the email. Now, something that people, even deep in tech, don't understand, and I know some of you in this room also don't understand this, is that the second time we call the agent, we have to send the entire history back, the entire history, OK? So how does that look? Well, the next thing it's going to do is search files. But we had to send, can you send the proposal to Sam, read email, search files back. This is actually a big problem, because as this keeps going, it's building up over and over and over again. Of course, this is the context window. You guys are familiar with this. But this is why there is a context window, because it's building up on every single tool call. And it gets pretty out of hand, right? Finally, it says, yeah, I just sent it. But now, remember, every single time, we're sending all of those tools back on every single step. And we send them all back again, and again, and again. And this is actually an O of n squared problem, right? Specifically on the reads. Because every time I'm reading that data back out, I'm reading it out of the database, I am performing a lot of reads, especially as a durable agent, because I need to be able to resume from any point. I can't just assume this is in my cache. So that is going to scale very bad. My writes are not so bad. The writes aren't a huge problem. However, they need to be consistent. The writes have to be consistent. The second it writes, it needs to be able to be read by another process instantly. So I can't use something like a KV store that's eventually consistent. It has to work right away. So basically, we need an ACID database. The problem is, an ACID database sucks at volume. Like, really, really bad. Because all of my agents are trying to read and write from one place. And in order to be consistent, they generally lock the table when they're performing a write, which means I have all kinds of race conditions. And it just broke my promise of scalability. So really, what I need is I need every single thread to have its own database. I also need them all to have their own database. I need them all to have their own code execution. So we'll add those. Per-thread database, safe code execution. There's another problem, though. The people at my company want this kind of experience for interacting with their AI, or whatever, maybe an app or something. They're not actually interacting with it. They need an interface to interact with it. And that means we need some sort of an HTTP server to expose an API, or maybe some kind of WebSockets to expose the streaming nature. But remember, every single one of those threads requires this. The other problem is when I say, follow up with Sam. Great, tomorrow at 9 AM. There's no requests inbound. So there's nothing that would cause my agent to actually boot up. This is another problem. So we need some sort of a trigger, almost like a cron job, that can just run at any time. So here's roughly the shape of requirements that we need in order to build good domain -specific agents for customers, for our own business, so on and so forth. But you came to this talk, building durable agents with durable objects. So I guess you probably know that what I'm going to talk about is how durable objects are the solution to that problem. What are they, though? So I don't work for Cloudflare. I love Cloudflare. They do many things amazing. One thing that's not great is when I talk to other developers about durable objects, nobody has any idea what the hell they are. They have no idea. So here's my terminology. It's a weird little parasite server that's attached to a worker, or many of them. Weird little parasite. So it can perform. You have a worker. It can perform code execution in this durable object. Doesn't have to happen in your primary worker. It can happen in a durable object. It has its own little database. It has its own little SQLite database. And by little, it can be up to 10 gigabytes, so not bad. And then it basically has its own little HTTP server. It doesn't actually. It works through the worker. But as a way to think about this, you can think about it as a tiny little server that has all the functionality that you would be used to in a server. And the exchange for this is you pay for it in limitation. So it's 128 megabytes of memory. So that's an interesting constraint you have to deal with. There's no file system. That's another interesting constraint you have to deal with. It turns out you can use a database as a file system. But basically, we can see that durable objects solve all of these problems. They are actually the perfect primitive for this. Let's take a quick look at that. So isolated compute, that's one of ours. We have Jane's thread. Jane's thread is able to store all of its history, its logs, its files, everything within its own durable object. You don't need some sandbox. It is a sandbox. It's a tiny little server. And when John has a thread, his is completely isolated as well. These are two totally different systems. And the only thing that they're ever allowed to do with each other is whatever you define, whatever the boundaries are that you defined. And even better than that, Cloudflare actually has a product called Workers for Platforms, which lets you take a namespace of durable objects and say, this is customer A's, and this one's customer B's, and those two never shall meet. All right, no cold start. This is awesome. I mean, basically, you can send input into one of these things, and usually within about 200 milliseconds, this thing's booted up and ready to go. So you have effectively no cold start issue whatsoever. It's way better than a sandbox or a container or something like that. In 200 milliseconds, that's like cold, cold. Scalability is basically infinite. Cloudflare has no durable object limit. They have a lot of limits. When you go read those pages, the documentation pages, there's no durable object limit. You can literally have all of them. Kenton, who I believe is actually responsible for creating these magic little beasts, basically said, your application code runs exactly where the data is stored. And this is true. So the data in your database basically gets co -located with where the execution is. And so you end up with this per-thread database. Now, if you think about logging problems, there's so many companies out here that just their job is to log stuff, because logging is really hard at scale. This effectively solves the logging problem instantaneously, for agents at least, because an individual agent is inside of a single thread. A single thread is a single durable object is a single agent. Those are all equal. And so the database that's responsible for reading all of the results that you need to send back to the LLM every time is co-located with the execution environment that's actually running it and is not touching any other database. So you can read and write from that thing as fast as you want, because that one thread is never going to overwhelm that SQLite database. Now, what if the agent writes some code, and we need code execution? Well, basically, in that case, you can call something called a dynamic worker. You send the generated code into this dynamic worker. The dynamic worker runs it. So this is unsafe code, code that the LLM wrote. It runs the code, and then it can give you the results back. It's pretty simple. And the main question, of course, then, is it actually secure? Because can my generated code ask to exfiltrate some information? And no, you have to explicitly grant everything. You can even say that this worker has no network connection. It's just like a little box to do a little bit of runtime. HTTP server. These things can absolutely run request responses. They can also hold long-term WebSocket connections, which in agent world is super important. We all need to be able to run WebSockets, because we want to get instantaneous feedback the second we possibly can. And they already support that. Triggers. So this is one of my favorite features. A durable object can choose to run what's called an alarm at any time. But there's a catch. It can only have one alarm. So you have a durable object. That durable object can say, hey, I need a boot back up in zero milliseconds or in six days. So all you need to do is use that little database, and you create a queue of when things need a boot up. Every time it boots up, you set the next alarm. It's like kicking a ball forward down the road. And that gives you basically full cron support, full scheduling support. You can say, boot up at 9 AM, and now we have a solution to that problem. So again, durable objects are basically solving all of the requirements that are necessary in order to build really compelling domain -specific agents. The problem is we need to do that now for every single customer. So I'm going to do a little demo right now so we can actually play with this. It's interactive. There's a QR code on there if you guys want to scan it. This is an example of a domain-specific agent for submitting expense reports. And every single one of you, when you scan this, will go just type in your name, and you will receive your own durable object, which is an expense report agent. It'll help you fill out this boring expense report. Now, here's what's interesting. Businesses don't want Grokbot to be filling out expense reports. It's really not necessary that the bot filling out an expense report can also go check Twitter. Those are like incompatible ideas. What they want is efficiency and effectiveness over a long period of time. So I'm ready to switch over to this. Awesome. So here you can see that this is actually made up of three different agents. I have an accounting coordinator. I have an expense report. That's Tally, the little dude you guys are talking to. And then I have a Google Sheets agent. And these three different agents are all durable objects. And every single one of you, so we're at 67 of you so far, have actually created something. And hopefully, you noticed that it was pretty darn fast when you started talking to your guy. That's because you're all talking to your own tiny little durable object who is responsible for filling this out. I'm going to do one myself. All right. So I'm going to create a new expense report. And I am going to drag some receipts on here. Now, the other thing that's cool is, so this is a platform that we wrote to basically help customers and clients fill these things out. And as you fill these things out, they are all communicating to a coordinator above them. And the coordinator is making sure that when the expense reports are filed, they're filed properly. Once they're filed properly, it goes and talks to a Google Sheets coordinator and then starts filing things in the Google Sheets. And if it's not proper, then it tells your little agent, hey, you did a bad job on this, or that's not an appropriate PO code, or whatever. And you kind of get the idea. But you can see that we are absolutely zipping through here. But so far, our total cost for all of you is $0.19, even though we've got 89 of these expense reports right now. It's $0.19 because we're able to use Luna for something like this, because it's such a small task. It's such a small, small task. Another thing that's really important to have when you're building domain-specific agents is a ridiculously good way to see what's actually happening. So here, I can click on any of these. And again, I'm reading from the little durable object that's here. And I can see the exact request that was sent, exactly what got sent off to the user in excruciating detail. And then I can see my response that comes back. I can see which tools were called, so on and so forth. OK. Here's my guy filling out my expense report. And it left me some questions. What was the business purpose of this charter flight? I gave it a grand Cayman to San Francisco charter. Obviously, I had a business meeting there. By the way, this took less than an hour to make, because the agent's framework behind here, the standard agent framework, which I'm not shilling, but I am, basically defines exactly how these agents are built with Cloudflare Durable Objects. So once you have that, Cloud Code can just create these things, no problem. All right. Which customer or client was this? Oh, it was obviously for Cloudflare. All right. So I think I'm good to go. Go ahead and just submit this. Now, I also have the accountant view of this. So the accountant is going to want to log in here and see all of these things coming through and what's actually happening. And this is also a Durable Object. And all of your individual expense reports are talking back to this. And again, it can read and write as fast as you can feed it, because it only has to read and write information that's pertinent to itself, not everybody else's LLM chain. So we've got a whole bunch of these. And I'm really curious if this works or not. I'm going to crack open Google Sheets here. Oh, you guys are all getting rejected, aren't you? What are you getting rejected for? Somebody shout out what you got rejected for. Anybody? Anybody? What? AUDIENCE MEMBER 1. Launch with Enron. LAUNCH WITH ENRON. LAUNCH WITH ENRON. LAUNCH WITH ENRON. LAUNCH WITH ENRON. LAUNCH WITH ENRON. I'll just say, approve everyone. All right, you get the idea. So at some point, this agent is going to go off and talk to the Google Sheets agent, and it'll just start filling it out. And again, the Google Sheets agent is another durable object. And what's great about it is all it can do is Google Sheets. But it knows Google Sheets really well. It knows all of the APIs. It knows everything about Google Sheets. It's a Google Sheets master. And so you can ask it, hey, I need you to format this and that. And it's also a tiny little model. It's Luna. But it's Luna with just one job in life. So you can build up these collections of things over time. It's great for business. All right, I'm going to switch back over to my deck. There we go. Great. So those are durable objects in practice. But I do want to tell you some practical tips and tricks for this. And then I think I might have a little bit of time for Q&A at the end. So if anybody has questions, start thinking about them now. Practical tips and tricks here. What if the model is down? This happens a lot, an unfortunate amount, actually. Well, you basically need to have fallback models. And the trick here is, if there's a failure, whatever your framework you're using or whatever system you're using with your durable object, you need to boost it in intelligence. So go from a dumb model that you know worked really, really well to a smart model that is less fine-tuned for the application. That's a really good trick to keep everything flowing smoothly, even when a primary is down. And obviously, use different providers for your backups. Another one, how do we keep durable object execution alive when it's on a long execution? This is a huge problem, actually. And if I were to poll people who have tried to build agents using durable objects, this is probably the number one issue that they run into, is keeping the durable object alive for hours. The trick is to use an alarm. Basically, what you want to do, if any of you were JavaScript developers back in the day or something like that, you used to be able to basically create a second, it wasn't actually a second thread, but it was similar, by setting a zero millisecond timeout or a QMicroTask. This is effectively the same kind of thing. You basically want to create a very brief alarm. So an inbound request happens. You create a brief alarm with a very brief alarm. With the message that you want to boot the agent up with. The reason is that durable objects are very tightly bound to the execution of whatever triggered them. So if I have an HTTP request that's coming in, that durable object will run for the duration of that HTTP request. But if that HTTP request shuts down for some reason, my durable object will shut down as well. And that sucks when it's like trying to do long-term work or you've queued up some async work to do. So you really need alarms. I will warn you right now, there is a little bit of an issue here where you can cause these things to loop. I have done that. And it will just sit there eternally booting itself up. So you do need to be a little bit careful when you're using the alarm system to make sure you don't do that. But that is the trick. An alarm can basically run forever. There is a function in a durable object that basically lets you say, hey, I need you to run this. And then when you're done, I need you to continue to run for an additional certain period of time. And really, you only get about 30 seconds of execution when you do that. So you either need to hold the connection open to the durable object, or you can use this little hack and make it work. I am sure I will run into somebody on the durable objects team, and they'll tell me why there is a better solution, but not obvious to me. How do we schedule more than one thing at a time? Well, like I was saying, what you need to do is you need to create an alarm queue in the SQLite database for each individual durable object. So every single one of those agents that you guys were interacting with, it had an alarm queue where we just shoved in exact microseconds when it needed to boot back up. And then every time the durable object boots up because of the alarm, it pulls the next one off the list and then sets its alarm for that. So the function to do this is just set alarm, and it sets it at some point in the future. You guys aren't going to write this code anymore, but you need to tell your agents that that's the way you need it to be done so it works, because it's also not entirely obvious. So here we have an alarm that's actually going to run at 9 AM, and it's going to immediately set the 9.15 alarm before it even executes. You need to do it before it executes, because if it fails, now you've just killed the thread, basically. What if a durable object dies? I wish this didn't happen, but it does. Durable objects sometimes just go cold, like dead cold. It usually happens when you push an update. So if you push an update for your code base, it is pretty common that that durable object will just die mid -execution, and that can cause your next alarm to not get fired or something like that. So the trick, unfortunately, to this is that you basically need a watchdog. So you can use a second durable object that's watching the primary durable object. And if that one, there can be telltale signs, like, hey, that one hasn't done anything in a few seconds, but it sure should have, well, then you can fire another alarm externally or something like that. This is absolutely necessary, and unfortunately, only shows up when you're at scale. So initially, you're like, you got your prod. It looks great. Everything's working great. And then you actually get it out in the world, and all of a sudden, some weird percentage of the time, 0.325% of the time, these durable objects are dying. Another trick that you can do is have a cron job. So Cloudflare offers cron jobs as well. But cron jobs are really limited in Cloudflare, both on execution time and the number of them that you can set. So often, that's not a good option if you're at scale. We frequently have 20,000 to 50 ,000 durable objects that are going, which means if it takes me even 100 milliseconds to check every single one of those, I can easily eclipse any kind of 30-second window or something like that. So what Cloudflare services did we use when I built this demo and on our standard agents platform? Turns out we used all of them. Not really, but basically all of them. Almost every single one. Every durable object has to be attached to a worker. So you can't use durable objects without workers. So you're going to be using workers. Workers for platforms. So on our platform, we let customers come and sign up, and then they can use the platform, which means that they need their own isolated box to do all of that work in. Workers for AI is a great option if you want to do the inference. And really exciting that Cloudflare released a new model yesterday, a new type of system one model. So it's exciting to see that there's actually now like some alpha in using workers for AI. D1, if you're not familiar, is a database. This is very useful. It's sort of actually inside of each durable object, that SQLite database. It's basically a SQLite database that's global. So if you haven't used that, it ends up being really useful, especially with workers for platforms because you've got to store all the customers and so on and so forth. R2, of course, storing your big files, a worker's KV. You need to do that for your super frequent writes that are not ACID. Queues end up being important sometimes, although often the durable objects can do that for you. Artifacts. Has anybody used artifacts? Yeah, they're cool. So the way that our platform works is you clone down the code base. You have your agent write agents, and then you push it back up to artifacts. Artifacts actually deploys it to the Cloudflare Workers for platforms. It's a really good way to provide dev tools for people. There's a bunch more in here. I'm not going to go through them all. But that is durable objects. Thank you very much. And I think we have time for questions. So does anybody have questions? Yeah, there is a microphone, I think, running to you. Can you put your hand back up? Thanks. Hi, my question is, what happens when you want to have multiple chats with an agent? Because if an agent is a durable object, don't they normally just have one session? Yes. So what is the way? A single thread. That's a good question. It depends if you want shared context or not. If you want shared context, I mean, this doesn't even really exist. Like, you can't open up ChatGPT and have two conversations in the same thread, right? But if you think about ChatGPT as an agent, and each of those threads as an instance of the agent, that maps to durable objects very well. So think about you're working in ChatGPT. Each one in the sidebar is basically one instance of the same durable object. Yeah, if that's not clear, a durable object is like a recipe, and then you can just stamp it out thousands of times. So every single one gets its own context, but maybe the same system prompt or something like that. Thank you. Yeah. Say we do need to share some persistence across our durable objects. Yeah. Is that, I mean, I guess we could do whatever, but do we see that up front? Is there some concept of a shared interface? Like, how would you recommend that? So there's a number of ways to solve this at an agent level. If you're talking about it at an agent level, I would do that with tools. So I would have each agent have tools that it can look up global state. If you're not talking about that, you're just trying to create a bunch of instances of durable objects that have shared state, there's a lot of options for it. But basically, each durable object, you get to say what bindings it has. And so you could bind it to a D1 database that's global. You could bind it and have it communicate back to the original worker that it belongs to. There's a lot of ways to do shared state there. But if it's agents, I would highly recommend using tools rather than anything else. And if there's lots of agents that are doing small tasks, like everybody filling out expense reports with one parent agent, then the best way to communicate is English. And everything else will eventually break down. So there are so many attempts, I have been guilty of this so many times, to try to create some sort of a shared schema where a parent and child communicate. And inevitably, it's going to break. And it's much better for them just to communicate on the language they were trained on. So a child, like when you guys were filling this out, the parent was receiving, like, hey, Tom just filled out this expense report, blah, blah, blah, blah. Here it is. And that's all it was getting. It was not structured. Thank you. Thanks, Justin. So my question is, let's just say if I want to make any kind of agent, like a chat agent or any sort, I would need a harness, which I'll be programming on top of a durable object. So what is the starting harness which you recommend? Is it, like, buy-do-able or? I mean, that's a great question. I use my own setup. So standard agents is a standard specification for exactly how those harnesses get built. And it's not open source yet, but we're planning to open source it very soon. But I use that for everything. And there are other alternatives out there, which you could try using like Flu or something like that. But basically, it's really important that all of your agents are written the same way, so you have the same visibility into all of them. So you could even invent your own. I mean, everybody's building the same thing, right? That's kind of like one of the takeaways that you should have from this conference, when you talk to, like, everybody. Everybody's building roughly the same stuff. We're all, like, building these things that build things or building agents and stuff. And so you could build your own. My hope and vision for the future is that we are able to emerge on agent standards very similar to how we had web standards. And at some point, it'll be like, yeah, this is the way that the little nuts and bolts of how the tool calls are done is put together. But then the rest of it is up to you. But that's a lofty vision. Who knows? Thanks. Because for me, I usually just, you know the thing harnessed by Cloudflare, right? I just give my Cloud code, like, oh, use thing to make this. And it somehow makes it. But I'm not sure if it's the best way, because there's no standard out there. Yes, there's no standard out there. Is that why we have, like, standard agents or? Yeah, I mean, that's literally why that company's called Standard Agents, because that's what we're hoping for. OK. But yeah, it's a problem that everybody's struggling with. So I'd love for there to be a great solution. Yes, Sunil, I'm sure, has many thoughts about this, because he's working on Project Think, which is sort of Cloudflare's way to do some of this. But it's still pretty low level. It doesn't have that, like, higher level abstraction where you're able to see, you know, kind of the threads running underneath it. I think we have time for, like, a couple more questions. Can you go into a little more depth on how you set up the watchdog durable object? Like, how is that actually configured and how they talk to each other? Yeah, so in a durable object, there is a constructor which fires 100% of the time when it first boots up. And so you can pretty safely put stuff in there that you need to be sidecar, like a side effect to your durable object. So in the constructor, it will boot up another durable object, and then they will bind to each other, basically. And then the second durable object is watching for cues in the database of the primary. So in the case of an agent, you know that, like, records are supposed to be written or there's a check -in period or something like that. So when an event takes place on the primary, the watchdog is watching to see, okay, after a certain amount of time, do we have anything? And then it can either perform some diagnosis or it can just fire another alarm to see if it's there. But that's basically what it is. It's a small percentage of time when it fails, but it sucks when it does, so. Hello, this is a little bit of, like, sidetrack question, but, like, you talked about the 128 MB limit with the workers, and for, like, you know, long-running, long-horizon agents, what's your solution to solving that problem? The 128 MB? The 128 MB is really fun, but basically, what you need to do is just stream everything. Nothing, like, you can't buffer anything. Like, even when you're reading from the database, the results of your entire version history to send back to the parent, back to the LLM, that needs to be a streaming process where you're streaming it chunk by chunk by chunk. Oh, the other trick that I didn't put in the slides here is a file system. So Sunil has a solution for this in Durable Objects where you can basically treat the SQLite database as files, and we did the same thing. We arrived at the same conclusion before we did that. But basically, each row is allowed to be 2 MB. So you can take, like, even a binary file and automatically slice it into 2 MB chunks, store it in the database, and then when you access the file, like, let's say on an endpoint, you know, it's a 1 GB video, it streams the results back from there. So it never overwhelms the 128 MB limit. Yeah. Okay, last question. When you have thousands of these agents running around, will there be any topology kind to show the interdependencies? That's a good question. I mean, that depends so much on the business application that somebody wants to apply this to, but in general, sitting on top of all these Durable Agents is a separate problem that is really beyond the scope of even this talk where if you think about every single Durable Object having its own database and its own information, well, then what's the aggregate sum of all of that? Like, how do I know whether or not there's a problem in the wider ecosystem? And that kind of comes down to another, like, MapReduce -type problem where you've got to decide what kind of information do I want to transmit up to an even higher-level coordinator? Do I want a separate logging system? And then how do I get information out of that? So many options to do that. I don't have a good solution. The way we do this is that we send small chunks of telemetry from each one of these up to a central worker that does the processing on it, but who knows if that's, like, going to survive, you know, a million users at the same time or something. Okay, I think I'm out of time, but I think, thank you very much. I'll be here. If you see me, come and talk to me. Thank you.