Durable AI workflows with Effect: Maintaining open source at scale

I don't think it would be that surprising for anybody in the room if I said that AI has radically changed what it means and how much work it takes to be the maintainer of an open source project today, especially if it's a popular open source project. On the one hand, it's made it much easier to contribute to your favorite project, which is great for open source in general. But on the other hand, this increased volume of contributions that we're getting as maintainers makes it much more difficult to keep up and generally it causes a lot of problems if you don't have the right tooling in place. So what I'd like to do today is talk a little bit about how we're addressing some of these challenges in our own project, which is called Effect, as well as some of the tooling that we've put in place with a specific focus on the foundational architecture for how we're building these tools. So a little bit about me. My name is Max. I'm one of the founding engineers at Effectful Technologies, which is the company that's kind of driving the open source development of Effect and all of its ecosystem packages. My background is actually in DevOps and systems administration, which I think is kind of how I got my interest in building robust, reliable software, just kind of a general interest in making sure things are as reliable as possible. My own brand of anxiety is thinking about all the possible ways things that fail. And I think that's what drew me to Effect about six or seven years ago when I first became a maintainer, because this is what Effect is all about. And I'm actually curious how many folks in the audience are familiar with Effect. All right, all my Effect enjoyers out there, welcome. But for those of you who are not familiar with the project, Effect is a toolkit for building production-grade applications in TypeScript. But all that kind of sounds very jargony and ephemeral. What do I mean by production-grade applications? Well, I consider any application that needs to be production-grade an application that has to handle all of the possible cases that can happen when your app has actual users. And those users can be humans or agents. You want to make sure you're handling all of the possible scenarios, whether it's operations that can fail, whether it's resources that need to be acquired and then cleaned up later to prevent memory leaks, whether it's coordinating concurrent tasks or even introspecting your application at runtime. And Effect is a set of tools that's designed to give you all of this out of the box. And more than that, all of the pieces of Effect are designed to compose very well with one another, so that instead of installing 30 NPM packages to build all of the production-grade features into your application, you just have a single install that you need to deal with and you get all of that for free with Effect. But I mentioned that Effect is also an open -source project, so I have another query for you all. How many of you all are open -source maintainers yourselves? Cool. All right. So then in the last six months, how many of you have seen a pull request that looks like this? Yeah, I mean, there's a pretty significant dichotomy here between the amount of time it took to generate this PR, this contribution, the amount of human time it took. This could have just been a single prompt, right? And there's nothing inherently wrong with that. There really isn't. Every PR or contribution, there's a signal behind it, whether it's slop or not. The problem here is that, on the one hand, the amount of human time it took to generate this is vastly different from the amount of maintainer time it's going to take to properly review this and decide whether it makes sense to include in the library at the end of the day. And so our initial approach when AI started taking off and we were being bombarded with a thousand PRs and it was really getting untenable, was to try to put processes in place to filter out the sloppy PRs or contributions or block things. And we see projects doing that now with trusted contributors, or some projects have even blocked pull requests altogether. But over time, and especially as models started getting better, we decided to take a slightly different approach and really lean into both accepting these contributions as well as using AI very heavily to kind of deal with the influx of contributions we were getting. And this is an example of a case we had when we first started trying to figure out what tooling worked for us, where we had 200 pull requests opened over the course of, I think it was like 50 hours or something like that. It took down all our GitHub actions, our whole project was a mess. But at the end of the day, it gave us a lot of insight into the types of tools that we found useful in order to kind of manage all of this. And really solidified for us that leaning into AI was the right thing. Because like I said, every single contribution we get, we now view as a signal. Something that didn't work for the user, something that we could have done better, a feature that was missing. And everything is in the service of trying to improve the library. So we did start off by evaluating existing tools that were out there, right? We looked at code review tools, we looked at AI auditing tools. We looked at all of the different things we could possibly leverage to kind of fit into that workflow that we found was working for us. But one of the things about automation that I'm particularly, I don't know what the word is, it's a sensitive topic for me, is if automation introduces friction, then it's worthless, right? If it introduces friction into your workflows, it's not serving you. So ultimately, we did the thing that every engineer does. And we had to make the decision between using some existing solution or building it ourselves. And I think you can sort of guess what we decided to do. We decided to build it ourselves, of course, because we're a bunch of engineers. And the application is called the Janitor. And it's really quite simple. It's just designed to put automations in place that help us deal with repetitive maintenance tasks. It's nothing complicated. It's a relatively simple app in what we want it to do. But I think the interesting piece here is the architecture we put in place to serve these automations, these workflows that we have. So we started evaluating all of the different tests we wanted to automate. And they all kind of had a similar theme. It was a sequence of steps that needed to happen, sometimes calling out to external systems to get information, sometimes running AI agents ourself in order to do different things. And if for some reason that sequence of steps failed, we didn't want to repeat any of the work we already did. We wanted to pick up right where we left off on another machine or whatever and keep going so that we could remain as efficient as possible. Because I mentioned that with the influx of contributions we're seeing, we don't want to repeat work if we don't have to. And this kind of process of picking up where you left off with a sequence of steps that you're doing, it has a name and it's called a workflow. Pretty simple, it's a workflow. And Effect has a system for defining durable workflows built into it. So we wanted to use our own software for defining our workflows. And Effect also has an actor system built into it called Effect Cluster. So we were very interested in using Effect Cluster and Effect Workflow, kind of dogfooding our own stack, in order to build this application. But we were also really interested in Cloudflare as the infrastructure provider because we really like the fact that Cloudflare has very composable primitives on the infrastructure level that you can pick and choose from for the different tasks that you need, which again fit our brains very nicely because that's what Effect on the software level is all about, composable primitives. The problem is that our actor system is designed to run on long-running servers. It's not designed for serverless or edge compute. But our community, which tends to be very vocal at times, had been pressing us to kind of figure out a solution to let them run Effect Clusters on top of Cloudflare's primitives, on top of workers and durable objects. So we thought, what the heck, there's a great opportunity for us to try to build that out and see if it works. And so that's what we did. So instead of focusing on the actual automations we put in place for our own repository because it's nothing exciting, pull request reviews, auto -labeling, like kind of all the things that you would expect of a normal maintenance task, I'd rather focus on the architecture we put in place, the foundation we put in place to build these workflows in a very reliable and durable fashion. So we'll start with talking about Effect Cluster in the traditional sense, running on long-running servers, and then I want to compare and contrast that with how we designed the system to run on top of Cloudflare and durable objects. In a traditional Effect Cluster, you've got one or more nodes, and these can be physical machines, these can be virtual machines, they can be containers, whatever. But this is the place where your OS-level process will run, your Node.js process, your Bund process, whatever. And that operating system process, the application itself, we call a runner. And the reason for this is because the runner is the thing that's actually going to run the code that you define for your actors, right? When you build your actor, the runner is the thing that actually coordinates getting the messages to your actor and running them. In Effect Cluster terminology, we don't call them actors, we call them entities because we want it to be fancy. And when you define an entity, it's something that you specify the protocol for it. You specify the messages that you want to accept for your entity, and you also specify the behavior that each one of those messages should trigger, just like in a standard actor system. You have messages, you can respond to them, you can trigger messages in other entities. There's also these shard things on the screen right now, which I want you to ignore for a moment. We'll get back to them and talk about what they are. But you'll notice that our entities are grouped into shards in our server full, I'll call this, design. In a traditional Effect Cluster, you also have shared storage, which is used for coordination of the cluster, and it also is used when you want to enable message persistence. Excuse me. Oh, I just realized my microphone is right here. In an Effect Cluster, messages are volatile by default, which means that if you send a message and it fails to be delivered, that message is gone. But you can turn on what's called at-least-once delivery of your messages just by enabling message persistence, and so then you get durability on the message level. You'll always get the message, you just may end up getting it more than once. And that's kind of at-least-once delivery in a nutshell. And persistent messages are stored in our shared storage as well. Runners. Entities can message between themselves, so on the same runner, or between different runners on different nodes. You can send messages to different entities. And you can also have clients that send messages into your cluster. You can even have clients that join your cluster as a client-only entity. And getting back to these shard things that are on the screen, shards are collections of entities that we use internally as a routing optimization. It allows us to do a lot to make sure that routing a message around the cluster is as optimal as possible. But one of the things it gives you as the user is the ability to address your entities without caring where the entity actually lives. All you specify is the entity type and the ID, and the message will make it to that entity in the cluster. With this sharding framework we have in place, you get location transparency by default, which should sound similar to folks who have used Cloudflare and DurableObjects before, because addressing a DurableObject is much the same. In an effect cluster, when a node goes down, we're able to deterministically reassign shards to other nodes. So if you're scaling up or scaling down, we can move nodes around in the cluster to try to accommodate that. There's one other thing I wanted to mention here, which is that when you have... you'll notice that there's no central coordinator here either. So in an effect cluster, you don't need a central coordinator. Runners know deterministically, using a consistent hash ring, where shards should go, which is pretty nice. So you don't have a central coordinator process that you need to deal with. But what if we didn't have servers? Well, I mentioned to you we wanted to build a cluster that worked with Cloudflare, and we had two options. We could either build a whole generic protocol that let us route messages around in Cloudflare, but also on other serverless platforms, or we could lean into Cloudflare as kind of like a platform and use its native routing primitives to do this. And ultimately, we opted for the second approach. So if you define an effect cluster with our Cloudflare integration, instead of every entity living in a shard, on a node, in a runner, an entity is just a DurableObject. And instead of having shared storage, message persistence uses a DurableObject SQLite database as its mailbox. You can still send messages into the cluster from clients. This is showing a worker sending a message to a DurableObject, but you can also do it straight to the DO if you want to. And you can send messages between DurableObjects as well with the cluster. And the nice thing here is that, from the user perspective, all the code looks the same for the protocol and the handlers. The only thing that changes is what infrastructure primitive you plug in at the very end. We call them layers in effect. So we have a special layer for Cloudflare. We have a special layer for running on a server, on a long-running server. So if we compare and contrast the two here, you'll see that with both of these things, we get location transparency. We just do it in slightly different ways. In our long-lived runners, we use shards to get location transparency. In the Cloudflare implementation, we lean into DurableObject routing. And the other thing that we get here is durability on the message level. But the handler for that message is not durable by default. Remember I said for our handlers, we want to be able to pick up right where we left off. So we also have effect workflows that you can use to define stepwise workflows where every step is persisted after it's completed, and the workflow engine will resume where it left off if anything fails. So if you want to map a workflow into the effect cluster, when you define an effect workflow and you run your workflows on top of effect cluster, the workflow itself, every execution becomes a new entity in the cluster. And when you define workflows, you define individual activities for that workflow, which are kind of like, you can think of them like save points. They're the idempotent steps of your workflow that you're interested in persisting over time. So if we look at this from a practical perspective, as we run through our auto -labeling workflow, we check if all our labels are still current, we persist the result, we record our plan for what we're going to do with labels, and then we record the results. And if we fail, eventually this workflow will boot up again elsewhere, and we'll read back the previous results, and then maybe it will complete successfully the next time. But the point here is that we don't duplicate the work that we've already done for each activity. So it's very similar to maybe other workflow platforms that you've used, but the definition of the workflow looks the same no matter where you run it. So you may be asking yourself now, why would I use effect workflows if Cloudflare already has them? Why would I use effect cluster if Cloudflare already has durable objects and workers? Well, one reason is to avoid locking yourself into infrastructure. I mentioned that your code will look the same whether you're running on a Cloudflare cluster, an effect cluster, or a serverful cluster. If you're already using effect, you can lean into the rest of the ecosystem for telemetry, for any other effect services that you may be using, any other platform packages that we have. Everything will kind of just work out of the box because, again, effect is designed to compose altogether. And because the code looks the same no matter where you run it and you're just plugging in a layer to tell workflows and cluster how to run at the end of the day, you can test your cluster and workflows in memory or in containers just by plugging in different layers depending on where you're actually running. So testing becomes pretty simple with effect. So I want to close with some of the future directions we're thinking of taking this system. Right now, it's doing very simple things for us. We have auto-labeling. We have code reviews built into it and a few other things. But it's doing nothing fancy. It's just kind of taking work off our plate. But if any of you have popular open-source projects, you'll know that the community is also a really big part of open-source. And especially if you have a large project, the discussions that you have about your project can often be quite scattered across many different platforms, across many different arenas, I'll call them. And we have the same problem. There's a lot of esoteric knowledge that gets lost kind of instantaneously between GitHub, Discord, Slack, whatever. Someone has to remember where it happened. So one of the things we're interested in potentially exploring with the janitor is using it as also kind of like a memory layer for ourselves and our community. So kind of funneling information in to the janitor from different systems and reusing this information when questions get asked in our Discord, for example. Or if we're trying to figure out why we made a particular decision about an API, instead of combing through Discord and Slack and GitHub, maybe we can use the janitor for this effect. And in particular, we're very interested in optimizing this to make sure that we're always giving the most current advice. Because again, if you have a popular open-source project, you know that it's constantly evolving, especially with AI. Contributions are helping this project evolve. And so we always want to make sure we're using the most current information. This is not a problem we've solved yet for ourselves. It's just an interesting direction we're thinking about taking the project. But one of the takeaways I hope you'll have from this presentation is that, especially in the era of AI, personal software, custom software that fits your own workflows is very much achievable. You don't have to use off-the -shelf solutions if you don't want to or if they don't fit your workflow. An effect can often help with keeping things highly composable and flexible for yourself and avoid locking you into different decisions that you've made over time. So with that, I'll close. Thank you all very much.