Fireside Chat with Brendan Coll, the creator of Miniflare

Hello everybody. Welcome. Welcome to Cloudflare TV. My name is Ashcon. I'm one of the product managers on the Cloudflare workers team and I'm really excited to be joined by Brendan, who is the creator of Miniflare, who's going to be joining us and talking to us about the recently announced Miniflare 2 and how you can test and run workers locally on your local machine. So Brendan, welcome to Cloudflare TV and thank you for joining us. Thank you very much. I'm really looking forward to this. So mind sharing with the audience a little bit about yourself, maybe kind of some of the origin story of Miniflare and what is Miniflare for people that may not know? Yeah, of course. So Miniflare is a fully local simulator for Cloudflare workers. So it's built on top of Node.js, which uses the same V8 engine as Cloudflare Workers, but Node.js and workers implement different runtime APIs on top of that. So the whole project of Miniflare is just to reimplement workers essentially in JavaScript on Node.js so that you can run workers locally on your computer without connecting to the Internet at all. Right, because when you deploy workers, that's running on Cloudflare's edge and Miniflare essentially takes that and allows you to run it in Node.js. Exactly, exactly, yeah. So Miniflare itself, the reason I created it in the first place was I was building my own workers application, as we do, and I was trying out Wrangler Dev, which I think was recently released at the time, and it was a little bit slow. I was using, it was a full stack application, so I was using Vite.js on the front end, which one of its selling features is like lightning fast hot module reload. So when I was making my front end, it was like I would type something and it would instantly appear in my browser, and then with my worker it was a little bit slower than that. So I wanted to see if I could improve that. I found a couple of projects, so there was Dollar Shave Club's Cloudworker, and there was another one called Cloudflare Worker Local, both of which are trying to do the same thing of running your worker locally on your computer using Node.js, and they were both really good. Cloudflare Worker Local pretty much got me through that project, but at the end of that I was like, I think there's some room for improvement here, and there are lots of new workers features that these aren't implementing, so that's why I created Miniflare. Yeah, that's really awesome, and certainly I think one of the interesting things we can talk into is some of the advantages of running things locally, because I think people can understand how Cloudflare runs workers on the edge, and how they're different from containers, but I think there's an interesting component to running things locally that maybe not everyone gets the advantage of, and it'd be interesting to hear kind of what are the things you can do when you're running a worker locally that maybe we couldn't quite do as well if it's running on the edge. Yeah, of course, so obviously the main advantage for me is the speed, because there is no Internet required, like you're just on your computer, and it's Node.js, so it's all optimized for that, but also because it is Node.js, you can leverage that entire ecosystem for all of its debugging tools, so things like the Chrome DevTools, that protocol is supported in all major IDEs, so you can use that to efficiently step through debug your workers, and you can use sort of its amazing sort of console logging, which is pretty prints all of your objects and stuff, which is really nice and really helpful for figuring out what's wrong with your worker while you're developing it. We also have some nice things that we have, so very pretty stack traces that we can get locally as well, which I'm sure the workers runtime will add eventually, but right now we can steal all of that from Node and not have to do any extra work for that. Yeah, certainly a lot of the, and what you mentioned about kind of taking time to develop your worker, I think there's a certain iteration cycle when you're first starting a project, or maybe you're making quick iterations, quick changes to a project, and you really want to see those instantly, right, we always try to improve the time to deploy and try to make that as fast as possible for the edge. Ultimately though, it is hard to compete with just running it on your local computer, and so even in scenarios where you may not even have Internet, you should be able to run and test workers. 100%, yeah, so it's very exciting. And I think one of the, so we talked about running it locally and some of the advantages there, what's the setup of the project? So for people that don't know or haven't read, the blog post for Miniflare 2 recently came out today, so you can go ahead after this and read all the details about kind of the transition from Miniflare 1 to Miniflare 2, but maybe Brendan, if you could talk a little bit about the difference between Miniflare 1 and 2, and what you learned from the first iteration going into the second one. Yeah, of course, so the first iteration was really like, it was probably my first big sort of TypeScript project, so it's massive learning experience. The big thing with the second one is instead of having it as one massive package that has all of workers implemented in one thing, I've split it out into about 20 different packages, so you've got sort of a package for KV, a package for durable objects, and all of that, which means that if you wanted to just import like a KV namespace mock for your tests, that's something you could do, and you wouldn't have to bring in the rest of Miniflare for that. You were also able to shrink the bundle size too, right? So the Miniflare 1 was how large? Oh my gosh, yeah, Miniflare 1 was 122 megabytes or something, and Miniflare 2 takes that down to about six. It was 88 megabytes, I'm sorry, but yeah, it takes down to about six, which there are lots of sort of big packages that I sort of included in Miniflare 1 without really thinking about, hang on a minute, do I really need this entire package for this feature? So strangely, the entirety, like Miniflare 1 included the entirety of TypeScript, which was a little bit of a mistake just for sort of common JS module support, but in Miniflare 2, we just implement that properly without TypeScript, and it's actually much better as well. And I think one of the other things to mention as well, and you kind of touched on it, was that it's not just a core functionality of workers, like defining a module, having a fetch handler, it's KV, durable objects, all the other kind of ecosystem features that you can use with workers, and as you said, you can even mock them locally. Do you want to talk a little bit about kind of the extensions you might say to Miniflare? Sure, so the stuff that Miniflare adds on top of the workers runtime? Yes. Yeah. So I guess that they're primarily focused on improving the developer experience and testing experience, so things like when an error is thrown, the error page that shows will show the stack trace and like the file source code, let you see exactly where it's thrown, that kind of stuff. Lots of the error messages have been enhanced with sort of suggestions for how you can fix them as well. And on top of that, we've got things like the sort of Jest environment, which let you use these workers APIs in your tests, while each test sort of has its own isolated storage that I'll show a bit later, I think. Which I think is a really big deal. So I mean, obviously, unit testing is a big part of kind of any developer's toolkit. That's been a challenge historically with workers. And so if you're looking to unit test workers, Miniflare's Jest plugin is a great solution. A hundred percent. Yeah. That's pretty much what it was built for. Yeah. Well, so we've talked about it a little bit. I think people might be interested in maybe taking a look at a demo. Yeah. Let's see if we can make this work. All right. So can you see that screen? Yep. We are all, you are live. We're all good. Fantastic. Okay. So let's start with a really simple worker. So we've just got a single file here, got index.js. And this is one of the new modules workers. So if we just had basic hello world type response, we can run Miniflare with a single command. MPX Miniflare will install Miniflare for you. Dash dash modules is because we're using the new yes modules thing. And live reload is for fun, which we'll show in a bit. So that single command will start up a worker's development server. And when we visit that in our browser, we'll see the response. So because we've enabled live reload, if we make any changes to this worker, they'll be instantly reflected in the browser, which if you've tried Wrangler dev before, that happens a lot faster. And you're not refreshing the browser, are you? You're simply just running control as you're saving it. Yeah. It's instantly updating. Yeah. So that's really exciting and really improved sort of developer velocity, which is one of the core aims of Miniflare. So we can take this a step further. The other thing about Miniflare is I wanted it to be as sort of zero configuration as possible. So if you've already got like a Wrangler tomo file, Miniflare should load all its configuration from that. So say we had some like KV namespaces in here. We will be able to access those instantly in our worker. So I'm going to update the code a little bit just so we can make use of KV namespace. So oh, dear. So this is a simple worker that just takes a count from a KV namespace. And if the URL is increment, it increments it. If it's decrement, it decrements it. And stores the new count and returns it. But you'll see here that we actually typoed counter namespace here. And Miniflare is telling us that, hang on a minute, I can't put into this KV namespace that doesn't exist. And it's showing us exactly where this error happened. So we can go into our code and change that and save. And it instantly refreshes and fixes our code. So that's really exciting. So now if we go to increment, we'll see that the counter increments each time, which is really exciting. So the problem with this is that say this was our actual application. And we just seeded it with a bunch of test data by like filling in like registration forms or whatever. And we put all our stuff in our namespace. And then we finish for the day. And we stop developing with Miniflare. And we come back. All of the counts are gone, right? So this data is just being stored in memory at the moment, which is not great. So with that, we can enable Miniflare's KV persistence, which is just a simple flag in Wrangler.toml. And now what we'll see is when we increment the count here, it's actually being stored in the file system instead, which is really useful for a to we're actually we're seeing this data on the file system, which is great for debugging, because we can see that we are actually writing to the KV namespace. And we could also just if we wanted to edit this, say, so we did 40 instead, that would be reflected in our worker as well. So that's really exciting. Now we've got all our work done. So let's say we're ready to deploy this. But first, let's add some tests to our worker. So we're going to make use of Miniflare's Jest integration, because Jest is quite a popular testing framework. So the first thing we need to do is we need to write our actual test. So we're going to import our worker from here, because it's just an ES module. And we're going to write our test here. So we're going to test that our worker increments the count in a KV namespace. So we're going to get the namespace, and we're going to put a sort of initial count of five in there. We're going to make a request to increment, run our worker, and then we expect the result of that to be six. A couple of things here, you'll notice we're using request in our test. And request is not included in Node.js by default. So the Miniflare Jest environment is injecting all of these workers runtime APIs into our tests. So we're able to use those as well, as normal. A couple other things, counter namespace, because we're using a modules worker, normally we'd get that from the env parameter. But we don't really have one of those in Jest. So we have to use a special global to extract it. So once we've got that, we just need to configure Jest to use the Miniflare test environment. So we'll do that in a Jest .config.js file. And then we should, in theory, be able to just run our tests. Yep, there we go. So we can see that our worker does indeed increment the count, which is great. So our worker had two functions. There's a couple of really interesting things here. So one is, as you said, the Cloudflare, the worker APIs are actually in the environment. So you can basically write tests just like you're writing at any other worker. And then there was not a lot of setup involved in getting that set up. There were two built on the Rengar Tomo file, having the Jest file, which was basically just like one property. That's really slick. Awesome. Yeah. Sorry, I kind of cheated a little bit because I installed all this stuff beforehand. So you raised an important point there that this Jest environment will automatically load your Rengar Tomo file as well. So it gets all of its configuration from one place. So you only have to change one file if you add a new KB namespace or durable object binding or whatever. So one last thing as well. If we wanted to write another test for decrementing as well, it would be a bit of a hassle if we had to keep track of all of the changes we made to KB in this test, sort of undo them all in this test because this test will run after this one. So the MiniFlare Jest environment does another thing where it automatically undoes any changes you make to storage at the end of each test. So this test will be run in a completely new storage environment. So there will be no count because we only put the count in here, which is why we're expecting this to be minus one, not say like five. And the great thing about this as well is you can use before each hooks or before all hooks, and MiniFlare will kind of copy the environment from the previous context as it were. So if you see data in like before each, that will be available in your tests. Wow. This is really great. So, and people can start using this today. Yeah, of course. So npm install MiniFlare. And then for the specific environment specifically, it's just npm install Jest environment MiniFlare. So it's not included by default, but it's just an extra package. Yeah. And in terms of the v2 launch, so v2 came out today. What are some of the other, so you mentioned the Jest, and we just saw the Jest unit testing environment. Are there any other interesting new features that you'd like to highlight about v2? Yeah. So one of the really exciting things is the ability to run multiple workers in the same MiniFlare instance. So this is kind of preparation for things like service bindings, which are hopefully coming really soon. But you could run your authentication service and your actual site in the same MiniFlare instance and have them communicate with each other. This is also quite useful just on its own at the moment for testing things like routing and workers, like sort of the routes field. And also the using durable objects from other workers. So by like the script name property. Other exciting things there in MiniFlare 2 as well. Durable objects sort of got completely changed after MiniFlare 1 was released. So we have things like input and output gates now and block concurrency while and all of that. And so MiniFlare 2 supports all of those features. That was pretty hard to implement, actually. Yeah. So input output gates were really interesting with DurableObjects. If people don't know, you should read up on the Cloudflare blog if you search for DurableObjects and input output gates. There's a really great blog post that kind of dives into concurrency in DurableObjects and how we handle it. And you are basically able to mimic our implementation into MiniFlare as well. So all the kind of nuances and whatnot, you can expect the same thing in MiniFlare. Yeah. I'm really keen for people to try that out on their DurableObjects. Like the workers that are using these new DurableObjects because I haven't had that many test cases for it. I have to just use the examples in that blog post and just some other things as well to see if I got that right. But it seems to be working. So let us know if you have any problems. Yeah. If people run into issues, should they just go to the MiniFlare repository, Cloudflare slash MiniFlare on GitHub? That would be absolutely fantastic. Yeah, that's the best place to put them. And if you've got any sort of questions as well, the MiniFlare channel on the Discord server, the Cloudflare Workers Discord server, is a great place for sort of support as well. Yeah. I think even the people in there were able to get a sneak peek at some of the V2 features and work. So if you're interested in learning about kind of the cutting edge stuff, both for MiniFlare and for workers features and new upcoming things as well, certainly joining the Discord is a great way to get a sneak peek into the future. 100%. This is really great. So I think one of the other things that we could talk about more broadly is I think when people think about an emulator or simulator, some of the older projects that you mentioned towards the beginning as kind of motivation for MiniFlare, they did a good job, but there was a challenge with matching the APIs of the Cloudflare workers environment. So Node.js will have some APIs that don't work in workers. We will have some custom APIs that aren't available in Node.js. I think inherently people have that kind of worry about an emulator that maybe it's not that accurate. But MiniFlare does a really good job. And I was hoping you might want to talk a little bit about how you approach that and what you tried to do to make that better. Sure. So I think just slight kind of my personality in a way, like being a little bit of a perfectionist, but just testing out like all of these edge cases in like Wrangler dev sessions, just trying to make sure that they all worked exactly as they should. And then also the fact that this is now an official Cloudflare project and we have sort of collaboration with the sort of Cloudflare runtime team as well is really exciting because that sort of means that if something isn't working, we can like ask them, hey, what should this actually do? And they'll tell us. I'm looking forward to a little bit more with that sort of getting some of their involvement in the development of MiniFlare. And certainly in terms of collaboration and having MiniFlare as an official project, one of the new upcoming things that the team is working on as well is Wrangler 2. And so MiniFlare already integrates with the Wrangler configuration. But one of the things we're looking with with the newest revamp of Wrangler is that MiniFlare is built in to Wrangler so that you can essentially have a seamless, you know, one configuration, one setup. And we're really excited. We're still working on Wrangler 2. You can try it out on our GitHub at Cloudflare slash Wrangler 2. But we're also really excited that MiniFlare is a core part of the experience because I think as is the demo and what we're looking at, it's really seamless and it's really easy to just get started. It's worth noting you don't need an account to try out MiniFlare. You don't even need Internet access if your worker is not using the Internet. So it's a really great way to just get started with the platform with workers if you don't want to set up an account. Yeah. And so in terms of what's next for MiniFlare, so V2 came out today. What do you see? What do you see as some of the interesting challenges or what's on your mind in terms of what you'd like to solve next with MiniFlare? What would be next in store for what you'd want to tackle? Yeah. So a big thing at the moment is if you have a worker that makes lots of fetch requests, just say like an external API, there's no real easy way, especially in testing, to mock those responses. So for testing, you want reproducible results and you'd like to sort of say, when I make a request to this API, I expect this response. So you can sort of fix that data. So there are a couple of ways we could do that. And I think I'm really going to look out and reach out to the community on this because I want to see what they would expect this feature to look like and how they would actually need to use it. Because, yeah, as I said, there are a couple of ways we could do that. So it'd be interesting to see what they think. Yeah, certainly. I mean, when we think of the bread and butter of workers, right, it's taking a HTTP request, transforming in some way, rewriting it, batching it, and then giving response. And so I think certainly even for the edge, we're starting to think about how do we enable you to easily test and mock certain fetch requests and be able to make that seamless. But certainly getting that running on Miniflare and just having, being able to test that locally and just knowing that this sub request or this response will be exactly what it is, I think gives developers confidence that when they deploy, when they actually make change to their code, you've actually tested and you have a pretty good indication that it's either going to work or not. Exactly. Yeah, so that's the other thing with Miniflare 2 as well, is that one of the other goals, which was really helpful with me having access to the worker's source code while I was at Cloudflare over the summer, was just replicating all of the exceptions that the worker's runtime APIs throw. So say when you pass the wrong string or the wrong type to KV or whatever, that kind of type validation there. So sort of going through and seeing where all of these validations are and making sure Miniflare implements those. And I think to your credit and all the work that you've put into Miniflare, I think that's why a lot of the community is really impressed and excited by it, because it is very accurate in terms of maintaining the APIs. And as you said, even some of the error messages are set to be very close, if not the same. Though there's some inherent challenges with that. And I think certainly it'd be interesting to hear maybe through your development of Miniflare, what were some of the challenges you ran into, either from v1 or from v1 to v2? So I think the biggest challenge was definitely the new durable object stuff. Because from my perspective, the only sort of documentation I had on that was the blog post, which explained how they were supposed to work. So trying to reverse engineer it from that and the examples and just the behavior that I was observing took quite well. But there are some places where I have intentionally diverted Miniflare's behavior from the runtime. So with the upcoming service bindings, for instance, when you make a request to a service and it throws an exception, currently, I think this is so experimental, so this is very subject to change, but it returns just a 500 response. So your worker doesn't necessarily continue to propagate that exception up to the top of the stack. So I think for the development experience, you kind of want to know that that exception has been thrown. So Miniflare actually just continues to throw that exception, because that's much easier for us to do, because we're running in the same process and we don't have to tunnel it through different machines that might be on different physical computers. So that sort of helps us keep the pretty error page at the top and just makes it easier to debug your worker. That's really great. Yeah, certainly all the features that have been added, particularly the stack, when there's an exception, having the stack trace in there, it's really things that we're actually considering bringing into the actual edge experience. I think that's one of the cool things about Miniflare as a project and being in the open source as well, because we can experiment with developer experience changes, making things slicker, smoother, and then take that and bring that to everybody on the edge and in the core workers experience. So we already have a few ideas in mind of things that we want to make sure to replicate on the edge. But yeah, we're really excited and I think it goes without saying that you've done an amazing job on Miniflare. Pretty much everyone from the Discord community to the Cloudflare staff, everyone is really impressed. So if people have not tried Miniflare yet, you most definitely should if you're interested in running workers or just trying it out. So for people, as we close off the fireside chat, and it's been so great having you on, and kind of your perspective as the creator. So I think maybe one last wrap up, which is so people want to get started with Miniflare today, what would you recommend? How should they get started? Yeah. So as we mentioned, Miniflare is now included in Wrangler 2.0. So if you've already got that installed, you're already probably using Miniflare without even realizing it. But if you want to just sort of explore it a little bit more in detail, just maybe try run it standalone and use some of its features. If you go to miniflare.dev, that's where the documentation is, and that sort of guides you through how everything works. Awesome. Well, thank you so much, Brendan. It's been great having you on. Congratulations on your first Cloudflare TV segment. I also think this might be one of our first live segments in the year. So that's also great as well. But we look forward to a lot more from Miniflare and you, and thanks so much. Thank you very much. Thank you.