TanStack Start on Workers: Server-rendering apps an AI just wrote

All right. Welcome everyone. Good to see you all. So I'm Fabian. I work with Lovable and we've been working on serving people web applications for about three years now. And really since the start we started building this on top of Cloudflare. And it's been a journey in the implementation and the services we've been able to use. So today I thought I would go into a bit of the technical details behind our implementation, how we serve our applications. So first off, just quickly about Lovable. We set out when we started Lovable to enable the 99% of people who do not necessarily have the knowledge how to code. And we heard Matthew talking here on the main keynote on stage about kind of this new era of creativity and creatives who are able to take an idea into reality. And that's really what we're all about at Lovable. And so our product is basically this chat box that you go in and you prompt and you'll get a web application or landing page. And then you can iterate on that and you can publish and you can host it with us. And today we'll dive into the technical details on the front end and back end hosting of this. Within our platform we also offer databases, the ability to send email, to receive payments, and a bunch of connectors around this. But we'll focus on kind of the core part of the front end and back end hosting. So just to put what we're going to go through into perspective of the scale that we're doing here. So since we started, we've created 70 million applications on Lovable. And every week we create about 1 million new applications. And every month these applications get over 1 billion visits. And I was just talking with our Cloudflare FD here in the front and I got the fresh number that also every day we have about 7 billion requests to these Cloudflare Workers that we're going to go through. And so I want to focus on this first number, the 70 million applications. So all of these applications, we need to be ready to serve them at any time, right? And this is way too many applications to have ready and kind of a server live to serve this, right? So we can store the code for these applications, but then we need to be very efficient about spinning up the compute to actually serve them. And that is really what this talk is all about. And so a quick history here in how we've been serving applications. So when we started, we had a simple kind of React single page application that would render in the browser. And this makes it very easy to build the infrastructure behind this because the only thing we need to do is route the request and serve some static assets. And so we save these assets in object storage in R2 and then serve them, right? But we've done this migration earlier this year to TensorStack Start, which is this meta framework on top of Vite. So we've always been Vite-based. But with TensorStack Start, we do server-side rendering. And we then need, of course, some compute to kind of handle that rendering. And we need isolation between the applications that we serve. And so this move, why did we actually do it? And the main reason that kind of made us really make this migration is that we wanted to have the best possible SEO, search engine optimization, for our users. And to do that, we wanted to give them server-side rendering SSR. The other piece is that before, we kind of had a separate backend serving layer to our frontend serving layer, whereas now we have this isomorphic setup where we basically have this one code base that gets deployed into a server bundle and a client bundle, which makes it very easy for kind of the AI to work with the code. And we also maintain being able to serve these pages. So in any TensorStack Start application, we can kind of define which routes should be static and which should be dynamic. And so we can still rely on kind of the ease of use of just serving static content when you actually don't need anything dynamic. But the dynamic parts is what is interesting, I think, technically here and what we'll talk about. So a quick primer, we're using Cloudflare Workers to serve this. And Cloudflare Workers, there is this JavaScript runtime where Cloudflare Workers uses WorkerD, which is the runtime, right? Workers is more the platform. And it's built on V8 isolates. And this allows us to really densely pack the compute of all of these applications that we're serving. And if you would compare it to something like a VM that would give you isolation, this has much less overhead. And so it will have much quicker code starts and be able to serve an application much faster. And so on the platform, on Cloudflare, you normally have the normal deployed worker that you kind of deploy via Cloudflare's control plane or their dashboard, regular deploy. And you make one deployment for one application that you have. Earlier this year, in March, Cloudflare released Dynamic Workers, which is really a necessity for doing what we're doing with it. And a Dynamic Worker is a way for a normal worker to spawn another worker. And so we'll go through how we're using this. And so all of this code that we have in these 70 million applications, they're stored in R2, which is the Soviet storage, and it's very scalable. So that's kind of the easy part. And then the question now is, how do we serve all of these applications? And if we look at how this actually works, the app starts just having the code in R2, and we'll cache it in KV. And then we have this serving layer that identifies which code is supposed to be served. And then we load that in, spin up a Dynamic Worker, and then that Dynamic Worker is ready to serve the application. And all of this happens in less than one second. So that's the cold start before we're able to serve an application from being totally cold in R2 to having this worker ready, which is quite good. And I think most of our users have no idea when a cold start actually appears because of this. And let's dive into a bit how this actually works. So we can imagine a browser making a request to a URL. And so you can have a custom domain on lovable, but here we just have the subdomain myapp .lovable.app. And this request goes to one worker, which is what we call our proxy worker. So all requests goes into this one worker. And we can derive which lovable application it is based on the URL. And from that, so this is myapp, is kind of the identifier of this application. Now we can go to our database, and we can fetch some metadata around this application that would include, first and foremost, where we store the code for this application, and then also some other metadata, such as which version is the published one, do we have some restriction on access to this, is it public to the Internet, or just in your workspace, and so on. And so this we can also cache in Cloudflare's cache that is co-located with the worker. So any subsequent requests here are very fast as well. And then the interesting part comes when, once we have all of this metadata, we have two paths that we can go down into. And one is this kind of path that we started with, where we serve the static file. So this might be serving like an index.html, a CSS file, and a JavaScript file. And in that case, we don't actually need to spin up any more compute, and we can serve that directly from our proxy worker. But if we have something server-side rendered, we need to spin up one of these dynamic workers. And that is only happening then for a subset of requests. And it's really this new development that we've made that allows us to have best-in-class SEO, and also performance in cases. In some cases, you need server-side rendering to have the best performance, depending on how you retrieve things from your database. And so just two quick concepts that play into this as well. So when we serve applications, we use React predominantly as our framework here. And normally, when you render React, you end up just having one div that you then execute JavaScript on to render up the React tree that turns into HTML. But there's an optimization here that you can do that happens when we do server-side rendering, which is that the browser will make a request to the worker. And on this worker, we will match that to a route in TAN stack. We will run the React code behind that route, and we will render the HTML on the server. And we'll also get the data behind this HTML, and we'll separate those two out, where that is called dehydration. And so we serialize the data, and we have the raw HTML. And then we can serve those two separately to the client, where the client will be able to render the HTML directly, and then hydrate it with the data. And before even any JavaScript is executed on the client, you will already have a rendered page. And then when the JavaScript executes is when you get interactivity. So you attach the handlers to the existing DOM. The other concept that TAN stack start, this meta framework, offers is, I touched on it briefly before, is isomorphism, where actually this code base, it is written in one way for humans, or in this case mostly AI. And then it can be executed both in the browser and on the server or in the workers. And the way this happens is that if you execute this function in a server setting, then it will run like normal, and you will have access to environment variables and secrets and so on. But if you execute this in the browser, you will of course not have access to the secrets and any sort of backend logic. And what will happen instead is that TAN stack start will, in its build step, it will kind of replace that code with just a request stub that will make an HTTP call to the server that actually runs it. But from the perspective of writing this code, you don't really have to care about this, and you're just able to reuse functionality across the browser and the server. And going back to the workers layer here a bit, when we serve something with secrets, we of course separate that from the application code. And so what goes into spinning up one of these dynamic workers is first and foremost the application code, but then separately from that, we bind the secrets into this new worker. And so the secrets, they're of course stored encrypted at rest in our backend, and then when we spin up one of these workers, we'll load them in into the environment of this new worker. And a cool kind of side effect of this is that we're able to know that if we update the secrets without touching the application code, we know that a request to this worker needs to be invalidated, and we'll actually spin up a new worker with the new secrets so that as soon as you update secrets, we're able to invalidate existing isolates that are live and be able to serve requests with any new secrets that you update without actually touching any of the application code or a new deployment step. So a question here for the room is, which app do you actually think is the biggest that is hosted on Lovable? I know the question, and I think, or I know the answer, and I actually hope that it will change in the future. I think it will change in the future, but today the answer to this is Lovable itself. And Lovable was not always hosted on Lovable because it's kind of hard to bootstrap these things. You have to build it first. So when we started building Lovable, we actually built it on XJS. That's just where I set it up in the beginning. And then we did this kind of large migration of rewriting our entire front-end code base to Tanzx Start to be able to deploy it on top of Lovable Cloud or kind of these Cloudflare Workers. And the reason why we do this is, I think, deeply rooted within our culture at Lovable, and it's around dogfooding. And we want to make sure that Lovable itself is served like any other application. And so we know that now since we're currently the largest application on Lovable, we know that any kind of scaling issues that we incur, we can fix before any of our users would hit those issues. And so it's a very virtuous loop. And so I'll give you some quick poetry here. This is some Swedish poetry, but for your convenience, it's translated into English. And this is Nils Ferdin, and I think it encapsulates kind of this culture we have around dogfooding. So I'll read it for you. My heart is thine, thy heart is mine, and mine shall remain till I die. My happiness thine, thy happiness mine, thy tears they are mine when thou cryest. And so this really encapsulates the idea of our happiness is our customer's happiness, but also our pain is our customer's pain. Or maybe the other way around, our customer's pain is our pain, right? And so we really want to make sure that any issues that our customers have, we actually feel as well. And so that we're always fully incentivized to fix these issues. And there's been a lot of things that we've learned from doing this that benefit both us and our customers because of this migration I mentioned. And so let's go into some of those lessons learned. So one of the first things is that when you're running one of these isolates, you're quite memory constrained. And so each isolate has a soft limit of 128 megabytes of memory budget. And this has to be shared both with the bundle and any concurrent requests happening to to this isolate. And so you have to be a bit conscious about this when you're writing the code. You have to make sure that anything that should maybe live in the client bundle is in the client bundle and it's not leaked into the server bundle. And we had a few examples of this when we migrated Lovable where we got some out of memory errors because we accidentally included like a TypeScript compiler, for example, and a syntax highlighter into the server bundle, whereas this should have only been in the client bundle. And so these learnings in how to make sure that you separate bundles correctly is something that we then encode into our builder agent for all applications that are built for our users, over a million of them per week. And the other part here is that the cold starts is really the expensive part. So once you have a warm worker running, then everything is extremely fast. But the cold starts is what is the slower part, even though it's less than a second. But you really want to avoid them if you can. And so we cache the worker bundles in KV for faster retrieval. And we also really keep the bundle on a diet, so to say. So we try to always minimize the size of the bundle because that just makes everything faster from loading and parsing and compiling. The last part here is that the fastest way to serve a cold start is to just avoid it. And we do this if we rely on pre -rendering, and we can just serve static files instead. And so this relies on actually rendering within PanStack then the bundle at build time or publish time. And then at request time, we simply have to serve this HTML and the CSS and the JavaScript and hydrate it, as I mentioned before. And so whenever we can, we use this strategy. And when there's benefits to using server-side rendering, be it for SEO or performance, then we do that instead. And so kind of closing this out, it is this virtuous loop that we have created for ourselves where learnings from Lovable, which again is on the same stack now as any Lovable app that is created, together with the learning from all of the Lovable applications built on our platform is kind of transferred into our building agent. And we make sure that all of these edge cases that you might encounter, that we actually take those learnings and we make sure to encode that knowledge. And we have a whole system around making sure that our agent can retrieve this knowledge efficiently at scale, but that it can actually use that knowledge to make sure that any new application or any new iterations on any Lovable application can avoid previous errors or previous problems that someone else encountered. And this just keeps on looping of us making learnings and making better decisions and scaling more and actually figuring out what we should improve, both again from Lovable itself, especially when it comes to scale, but then just all of the edge cases that we find within our user applications. So that is really it for my talk here. We've been very happy with our partnership with Cloudflare here and being able to serve these applications and actually quite lucky, I think, in the timing of the release of Dynamic Workers to be able to support server-side rendering on our platform. And it's really worked well, everything from the tech stack itself with 10-stack migration, we jumped on that train very early on and working closely with Tanner and team behind that as well. And so thank you so much for listening, and I hope you learned something, and I'll be around here for a few minutes also if we want to do some questions. Thanks so much.