From chaos to control: Building a harness for personal agents

Hey, everyone. Thank you for being for the talk. I know it's also lunch going on, but you all decided to be here. So thank you so much for being here. Now, my talk is going to be around my journey of building my own personal agent on Cloudflare. FYI, I work at Cloudflare. I work in the developer education team, but I also do a lot of engineering. So the journey of my own personal agent started back in October when Kenton Wurda, the chief architect of workers, gave a really interesting talk about personal agent army. That made me wonder when would be the time when I get to build my own personal agent that would do all the tasks that I hate myself. And then a few months later, not even a few months, like just a few weeks later, OpenClaw came out. It came out, I think, as CloudBot, if I'm not wrong, but it took the whole personal agent ecosystem by a storm. And it wasn't because there was a new model, but because it had provided things that none of the products previously were able to do that. To give you an example, it worked wherever you were. So be it WhatsApp, be it Telegram, Signal, Discord, whatever messaging platform you used, you could access OpenClaw over there. It would also remember you. So let's just say you asked it to find a restaurant and you gave it an address. It would know your preferences from that request. And in a new session, if you ask it to find a restaurant again, OpenClaw knew what your preferences are, and it would use those preferences from its memory and find you a new restaurant for that new location. And lastly, it did tasks for you. It wasn't just you asking a question and it replying back. It did tasks for you. So if you asked it to, I don't know, like file a bug on GitHub, it would go ahead and do that. Either use the tools that it already has or write a custom script to do that for you. However, when I was using this, I kind of ran into a couple of issues. And this was like the very early times of OpenClaw when it was just launched. So the team was shipping updates almost every day, which kept on breaking my agent. And I was not happy with that. I would spend more time fixing my agent than the agent doing the work for me. The other thing was it needed a computer running 24-7. No, I don't want it to be that person who would hold a laptop and run around. That would be just hilarious if I did that in the city. So I needed a better solution for this. And there was nothing in that that could do that for me. So I wanted an assistant. I got a second job of kind of a DevOps person where I was maintaining my system, like fixing every time there was a new change in that. And that wasn't a fun job at all. Now, before I talk about what I did and how I did it, let's take a step back and try to understand what exactly a harness is. Now, the early days of chat GPT was you ask the model a question, the model would give you an answer. Fun enough, worked well, but wasn't really useful at that point of time. It was revolutionary, but did not do a lot of things. So a harness basically is a loop around that. A harness gives your model tools. It also gives your model the ability, not the model, but it also gives a runtime where you can save the messages that the user has posted, and it would also reply back to the user at the end of the day. So it would basically run a tool call, the tool would execute, send back the result to the model, the model would decide if it's finished or not, and if it did not, it would follow that whole tool call loop again until it comes to a conclusion, and then send a reply back to the end user. So this is basically what a harness is, and this is what I needed for myself. Now, I started building this on agent SDK. This was, I think, the 0.04 of agent SDK that I built my own agent harness. Again, I had to build it from scratch. I knew I wanted to deploy it on cloud because, A, I don't want a computer on 24-7, so it has to live on a cloud. Now, agent harness back then did not really have a harness loop. What it gave you, all the parameters, and you as an engineer would have to build that whole loop. So it would have an on-chat message handler, which would basically get triggered whenever you send a message, and behind the scenes, I would write a custom stream text, which is my own loop, where I would pass the tool calls and do all the processing myself. And this stream text method was roughly 1,500 lines of code. I would define my own tools. I would use the existing tools that are out there, stitch it into there. I would have the model in there. I would build a system prompt myself. I would also have a custom logic of pruning the messages, or in better terms, doing a compaction on messages. And I also have to decide when the model should stop its turn after a particular step count. Again, worked fine back in the day, but it wasn't the most fun thing I had to do. Again, this was 1,500 lines of code I had to maintain myself. So what I had to really build, I had to build the whole agent loop or the harness loop for this. I had to decide how the memory lives. So I thought, hey, I have access to R2, which is the object storage. Let the agent write the memory to a file and store it to R2. I also had to get the tools from different places, stitch it in together in my agent loop. I wanted my agent to do things on the background. And I was like, what would be the best way to do that? So I had to build a custom workflow engine on top of Cloudflare's workflows. If you don't know Cloudflare workflows, it is like a static way of defining a step-by -step function that the workflow engine can execute behind the scenes. And this is way before we had dynamic workflows. So I had to define my own custom way of having dynamic workflows in that sense. And lastly, I was also experimenting a lot with subagents. So I had to build a subagent runner myself as well. So these were a lot of things. If you are building a harness yourself, you might want to look at an agent loop, memory, how and when the agent gets the tools, maybe a workflow engine, and subagents as well. But I was like, what if there was something which would take everything away? Because again, I don't want to be the person maintaining the code all by myself. I was spending more time on building the agent than actually using the agent for my use cases. For me, I was lucky enough, Cloudflare came up with the answer, which was Project Thing. Project Thing is Cloudflare's own opinionated harness loop. It is very much inspired by Pi. And this was, again, very early. I think it was in April where I got access to Project Thing, where we announced Project Thing. And I was one of the early adapters of Project Thing. So Project Thing gave me the agent loop. It had a way of saving the messages and resuming the streams, again, because it was built on top of agent SDK. It gave access to tools like files. So I did not really have to have my custom memory tools for writing the files. In 2.2, everything came out of the box for me. And it also came with a really interesting concept of context blocks. Now, I can have different context blocks for different use cases. So I had a context block where it defined, or it told my agent, hey, this is the block where you should be saving all the memory for this particular user. I had a context block for the agent's own identity. So the context block helped me define how my agent would do a particular thing. And it also came with helper methods for compaction. And it also gave me tools out of the box to run subagents and extensions. So I had everything, by default, come with Project Thing. So this was the agent loop that I had kind of stitched myself. But with Project Thing, it kind of switched away in a more idiomatic way. Now, I had to just think about, all right, what model I want to use with my agent, what tools my agent can have, and then what context blocks my agent really needs. That's all I had to do. So instead of having those 1,500 lines of code, it was just a few hundred lines of code now for my agent. So this was, again, back when I was working on Agent Claw. I got highly inspired by Open Clause. I was calling it Agent Claw. But I wasn't really happy with that project. It was a lot of migration. The code looked ugly. And I wasn't like, all right, I need to start something fresh. So I started a new project called Yoap, which from day one is built on Thing. So it uses everything that Thing provides. And if there is something that Thing does not provide but is available maybe in the agent SDK package or on an external package, I could easily stitch in with that. Now, I'm going to walk through a couple of things that I had to, again, do manually with Thing. And it also evolved with the changes the packages shipped. So the first thing was the chat. Now, how do I define how I'm going to define the chat interface or chat session for a user? Should each user have a separate chat session? Should a chat session be separated from each other? I wasn't really sure what would be the best way to do that. So initially, I started with a router, which would basically be a durable object, which would create a list of all the sessions a user would have. And a durable object, a user would have just one single durable object, which would contain all the chat sessions. So if a user was switching between sessions, I used to basically have a pointer, which would point which is the current session and swap accordingly. This wasn't the best experience, because there was a lot of overlap happening. I would send a message in one chat session, but it would appear in a different chat session, and it wasn't a really good experience for the user at the end of the day. The good thing was, I think just a few weeks back, the Agents SDK team shipped a routing default with the Agents SDK. So now you have a routing agent, or think of it like a separate durable object, which would be taking care of that log of which chat sessions the user has. And you don't really need to worry about anything. You don't really need to also worry about how do you route to that particular chat session now. The routing agent would take care of that. And the SQL statement that you see is like something I'm really happy about, because that was like dropping my custom DPL. I don't need it anymore. No custom code. I can use what the SDK provides out of the box. So if you're building, again, on the Agents SDK, you might want to look at this architecture, because you often want to have a mechanism where you would store an index or a registry of all the chat sessions your users are having. The next one was the tools. Now, for my agent to actually do the work that I wanted to do, I needed to give tools. And again, previously, I had to stitch the tools myself. Not fun at all. And I think that was the time also when code mode was introduced. So I had to plug in my own custom implementation of code mode. So I would take the tools that I had, would run a for loop for that, and then let it handle that for me. So again, this is the most ugliest way of doing a code tool, because the number of tools that grows, it would break. If I added a new tool and I did not follow the standard that I had defined for my agent, this loop would break, which meant that I had to, again, maintain this particular loop. We shipped code mode in the runtime as a package. So what I had to do was simply define the code mode over here, give it the access to a dynamic worker, and now my tools can use the code mode runtime by itself so that I don't have to stitch all those things by myself. The next thing I was able to do with that is have a human in the loop flow. I am very skeptical of my agents doing any kind of write things. So if I am even asking them to add an event to my calendar, or let's just say just write an email, I want to have a look what the agent is doing before it actually does any kind of write access or even delete an event in there. With this, I was able to have that approval flow. So whenever my agent was trying to access something which they are not allowed to, they would send up an approval request to me like, hey, I need your approval for this. And unless and until I give that approval, my agent would not be able to do that. The other thing I added in here was like an approval timeout so that if I don't approve it within like five minutes, my agent would take it as a rejection and then maybe ask me again, prompt me again to approve that or just skip that and think about like another way of achieving that task or just be like, all right, the user hasn't given me the approval. Maybe they don't really want me to do the task. So with each release that came out with the agent's SDK with Think, I was able to delete a work around all the time. And today I am trying to figure out a crucial question. Should I still use Think or should I go with Py on Durable Objects? If you are aware about Py, Py is a coding agent, a coding harness which recently came out with Py Durable, which means that you can now natively run Py on Cloudflare and it gives you a lot of things out of the box. Think is inspired by Py. You can use Think if you want. If you want to have more customization, you can deploy Py. I know Open Code is the version two of Open Code is also compatible natively on Durable Objects. So you can deploy that as well. And we have another framework called Flu. Now, when I talk to people about these different frameworks, they often ask me, all right, you have so many options at Cloudflare to build and deploy agents. How do I know which one to use? And my take on this is how much customization do you really want to have in your agent? Do you really want to have a full customization for your agent? Go with agent SDK. It gives you all the building blocks. Use that to stitch whatever you want. And that's the best for you. If you want an opinionated harness that is built on top of agent SDK, Think is the framework for that. If you love Py and if you want to have all the features that Py ships out of the box, like a tree. So Py has a mechanism of starting a session from any of the checkpoints that you have. If you want to have a forking as well, you can use Py for that. Open Code, if you want to use Open Code. And for me, Open Code is more on the coding side. So if you're building more of a coding agent, Open Code, you can use that as well. And if you're building something very custom, you're familiar with React ecosystem, you want to have a dynamic way for your agent to get access to the tools, a dynamic way for selecting which model your agent should have, use Flu. Another interesting thing with Flu is you can kind of run that agent via CLI, because Flu comes with that as well. So you're not tied to just deploying it on Cloudflare. You can run it maybe in a GitHub action as well. So if you have those kind of use cases, I believe Flu is the best approach. Now, again, these are kind of my opinions on these frameworks. But you should be playing around with them and seeing which one works best for you. To give you a quick recap on the architecture today, so I have different interfaces where my user can interact with Yuke. There is a web application that the user can sign in and have their own personal agent over there. I have a mobile application built with React Native, and I wanted that experience for myself. So there's that. And I have also given Yuke its own email address. So if I want to send in an email and ask you to process something, Yuke can do that for me as well. And again, as I mentioned, there is a router agent, which basically contains the index of all the sessions that I have with Yuke. And then, based on that particular session, I can go into that particular session and talk with Yuke. Yuke also has access to agent memory. This is something we have in beta at Cloudflare, which you can hook it into your particular agent. So your agent remembers who you are, and the next time it needs to perform a similar action, it can go ahead and do that. And because it is using Agents SDK under the hood, I can have schedules in Yuke so that you can go ahead and schedule and check whatever it needs to do for that. I'm connecting external services using MCP, and it has access to Browser Run. So if I want Yuke to go ahead and use a website, it can use Browser Run for that. I have a few seconds, so I'm going to try and show a quick few demos that I have. So this is something I had. I think I did it on Sunday itself. So I asked Yuke, hey, I am traveling to SF. I am going to be jet lagged. Find me a way of how to fight this jet lag. So it went into my calendar, found out my flight details, and added things that I should be doing on my flight, when I should be sleeping, when I should be kind of having a coffee and stuff like that. And it could not figure out my return flight, because that wasn't in my calendar, apparently. So Yuke automatically scheduled a task to check this today itself. Like, hey, I'm going to check it on Wednesday at 11 o'clock to see if that flight detail is in your calendar. Apparently, I forgot to add it to my calendar. So Yuke does not have access to that. But that is something on my side. There's another example that I have. Let me try to play the video over here. I'm not sure if it's going to play over there. All right, let's... Okay. So now... Come on. Anyways, okay. I have another video that I can pull up real quick. Which is where I am basically using my agent to ask for a restaurant in San Francisco, because I was meeting a couple of friends. So I was like, hey, find me a restaurant in San Francisco that I can go and hang out with my friends. So let me zoom this in. And over here, I am asking it to go find out some restaurants. It goes in, it does a bit of thinking, it tries to figure out the restaurant. It's going to do web search in there. So if you see, this is basically the web search that it does. It is now reasoning around it. And it is going to use the browser in there. It is now trying to book a table using OpenTable. For me, I did not went ahead through the whole process of booking a table, because I wanted to make sure it is a good restaurant and my friend is going to enjoy eating over there. So I haven't done the whole process, but just like to show you the capability of what the agent has today. So this is what you can do today. Trying to go back to my slides. All right. So I started with building a lot of things that were kind of not really available with the SDK. But with the new features and new things that were shipped with the SDK, I was able to delete a lot of those things, which was fun for me, because I had less of a maintenance to do and more of usage. And I can actually start asking my agent to do stuff for me. I also wanted to show you it on my mobile app. But for some reason, I'm not able to connect it to it. But if you want to talk more about agent harnesses or how I am building my personal agent, feel free to find me in the speaker lounge, which is at level 2. I'm going to be hanging around that. Happy to answer any questions for that. And if you don't find me over there, I am over here throughout the conference. Just find me, and I'll be happy to talk around that. And one last thing, do let us know how you find this session. I've tried to just cover the architecture and give you an overview. But if you want to learn more in the technical depth of it, your feedback will help me learn that. And hopefully, the next time I can go in more depth for that. Thank you so much.