100 billion profiles, one early bet on Cloudflare Durable Objects

Thanks for joining today. My name is Mark. I lead solutions here at Cloudflare, but actually my job is really easy. My job is to make Alex look good. So give it up for Alex. Thank you. Alex, tell us who are you? Where do you work? What do you do? Sure. I mean, Mark, we know for quite some time now, probably four years, close to four years. I think as long as you work for Cloudflare, probably, right? Yeah, I'm Alex Braunreuther. I'm chief architect at Contentful, now Contentful by Salesforce, got acquired just a few weeks ago. I actually use Cloudflare through my previous company that got acquired by Contentful, which is Ninefield, a personalization software that runs completely on Cloudflare and durable objects. That's why the title of this talk is 100 billion profiles. Yeah, and I'm doing software architecture mainly. Okay, that's dope. So you had your startup tiny. How many people were you at the time? Ninety? Roughly 30. 30? That got acquired by? Contentful. How big is that? 800. How big is Salesforce now? 85,000. Nice. Okay, good journey right there. Okay, cool. Tell us about that. So you have a personalization software and the talk title says like you have 100 billion profiles. Last time I checked, we have like 8 billion people on the planet. So what is a profile? What do you use it for? And what do you need durable objects for? So when we started building our Ninetales, we wanted to build a truly API first personalization software. So personalization software and experimentation at the same time normally work like this. You get some HTML divs, which you fumble into your website somewhere in a client, which is pretty annoying because it's slow. You've got flicker effects and it's simply not API first. So when you think about agents, for example, today, they could just work on raw JSON stored content or you simply don't have HTML. You work in apps or in a kiosk or wherever. And this was very important to us. On the other hand, performance was, to me, extremely important. So I come from the JAMstack era of web development, basically. Good times. Like squeeze out the last bit of performance you can get out of a site. And then you had a script on your site that runs for two seconds until you flicker something. And that was pretty annoying. So we wanted to build a solution that is extremely low latency anywhere in the world. And that's why we use durable objects for that. So basically, any profile that you got is one durable object. So that makes for a very nice scaling effect in the very, very low blast radius. So why is it 100 million durable objects or somewhere in this area, even if there are less people? Because our system works a little bit like a DNS. So you could now have a profile here when you open the Contentful.com website, you get a profile here. When you travel over to Berlin, for example, then it creates a new profile there and takes the data over to a durable object in Berlin. When now Cloudflare uses Salesforce, for example, the Salesforce server will probably inject some data somewhere in the middle of the US. Then it will generate a durable object in the middle of the US and then redirect the data to Berlin, basically where you are right now. So your profile travels with you, but it's profile partials that build up a system that talks to each other. Dang, that sounds pretty cool. So basically, my user profile travels with me around the world. And basically, you have a small database for everyone who visits your customer's website? Yes. So every user is one or even multiple small databases. Basically, they're durable object partials of their profile. And yeah, it travels with you because it has to be absolutely minimal latency as much as it can. When you started building on durable objects, so we met in 2022. Durable objects were super new. So we now have merged with durable objects on an X. Cloudflare, I can tell you, in 2022, we did not have merged with durable objects on it. I would take one afterwards. I'll get you one, actually. I'll get you one. Don't worry, don't worry. But why did you bet on it? I mean, AWS already at the time had like 35 regions or whatever. Why use durable objects? So the problem with a low latency system that is fully distributed is eventual consistency ultimately. So we have put our bets on workers in 2020 when we started. The problem there was like distributing your state to multiple regions is insanely complex because the system becomes eventual consistent. So let's take this traveling problem. So you would travel over from here to Berlin, for example. Now I have to synchronize. Let's say you have a Redis, for example, here and have a Redis in Berlin and I have to synchronize somehow over a bus. Then your profile also is available there. OK, when you travel, that's probably not that big of a problem because your plane is fairly slow and for sure a lot slower than the speed of light. But when you think about the problem that I said, OK, Salesforce server sits here somewhere and updates your profile, this leads to inconsistencies. And then out of a sudden, when you're on a website, you just flick around through different content because the consensus mechanism of this whole distributed system becomes very complex. And the interesting part here with durable object is they play very nicely to the problem because they are transactional in themselves. So basically, I like to think about durable object like the cap theorem. It's actually not possible to have all the three parts of the cap theorem. But as soon as you take the durable object as your atomic instance of something, it's basically in itself consistent. It's very available and you can scale it out to the get-no because at least we can because you do that for us, thankfully. And this is basically a perfect symbiosis of what we have needed at the time. OK, well, you can tell we have a serious talk here, a serious technical talk because the cap theorem was mentioned. So I'm really happy. Thank you so much for the checkbox of my bucket list. I appreciate that. I'll try to take that back. So you're saying, hey, globally distributed was important, but also making sure that my user profile is always consistent with what it should be, basically. Like strong consistency guarantees. But now you have like 100 billion small databases. I personally know how much of a pain it is already to manage one database. Now you have 100 billion of them. How tough was it to shift your mental model to say, OK, how do I go from, I feel like engineers, maybe like you bet on a new technology. Most likely there's not a lot of experts in the industry that were like, oh, yeah, I work with durable objects for 10 years. You know, like these CVE requirements, where it's like 10 years of durable objects. They're not there yet, right? So what was the biggest challenge for your team, maybe in the mental model of how to work there? Yeah, I think before we go into the mental model, like one factor that is extremely interesting there is like we were working with some of the biggest enterprises in this world, which have a ton of traffic. And one big problem for most of our competitors always was blast radius. So a classic personalization software works like this. You send in events, they go into a big data warehouse, BigQuery, Redshift, whatever. You have batch mechanisms that produce new profiles, put them into a cache and the cache is basically live. Let's say you get too much traffic and your radius cache runs over or gets too much traffic and simply blows out. So you have a multi -tenant system. One big enterprise kills your whole region. Or when you have sharded it to this enterprise, it kills itself. With the durable object, I don't care. If they basically come in and send a hundred thousand requests to a single profile, this profile is dead. But there are other few hundred million profiles, they don't care about it. They just run. So like I think since we run the software, we had never a single downtime, which is great. And like the scale you can put on the system. So I basically slept pretty much for every Black Friday and nothing works. I think your team needs really a mental shift for that. That's extremely important. The most important thing here is to not fall back into what you have learned. So an example would be I need to do GDPR deletion, for example, for my profiles. So I need to delete them after, I don't know, let's say 90 days of inactivity or something like that. I need to delete my profile because a customer has set up this. So how would you normally do this? So you would have probably a central index where you list all your profiles in a Postgres database. And then every day or every hour or whatever, you have a cron drop. Works great when you have a few thousand or a few hundred thousand or maybe a few million. But when you have like hundreds of billions of these profiles, now out of a sudden this query for this deletion can simply kill your database. That's pretty annoying. It sounds like it. So you basically now need to shift your mind when you think about durable objects. You don't manage all durable objects. Every durable object manages itself. So they have a great mechanism, which I think actually was the moment where I thought, okay, these things are brilliant. Our durable object alarms, because with this primitive, you can build pretty much any complicated distributed system problem here. So a durable object in our case is just like whenever it gets updated, it sets a new alarm for one year. And in one year, it kills itself. That's it. It's very, very simple. On the other hand, what the team then needs to learn is you always have to apply this thinking. So for example, there could be the case of a customer comes in and says, I need you to delete all my data. Now you're in a pickle because the Cloudflare APIs are actually slower in rate limits than our profiles increase. So basically, I think you have like, let's say you have 100 requests, which give you 100 profiles per second. When your profile rate increases by more than 100 profiles a second, then you cannot query them anymore. So the naive approach, which our engineers directly went was like, okay, let's index everything into BigQuery. And then we have a huge list and we can fetch this and go for all of them. Pretty annoying. You can think the opposite direction. You just have a helper database where you say, I have my tenant and my tenant says, I am deleted. And every day, every durable object has alarm, looks into this database and said, should I delete myself? No? Okay, I'm going to stay. And I mean, as basically a database read doesn't cost much because you can put a cache in front of this. And the durable object just wakes up and takes three milliseconds or whatever, like a very, very small amount of time. It's not very costly and they kill themselves. So you just turn the problem around. And this is, I think, the most important aspect here. Like, make it clear to the team to not take the obvious solution here. Cool. Before I open up to the audience, I have two more questions for you. But if you already want to think of questions, that's my signal to make you guys think a little bit. So I'm curious, you jumped on this really, really early on this primitive, most likely when it was, I don't know, better. How was the journey from jumping on the primitive early to where it is today? Did it get better? Yeah, for sure. I mean, if it wouldn't have, then we wouldn't be able to run it, actually. Because in the early days, betting on workers was really a hard bet, I have to say. And on durable objects, I mean, in the very early days, there were no locks, so there was no lock push. So you basically ran a worker and you could not run it on your local machine. You deployed it to the cloud and sent back errors via headers or something like that. So that's how we developed in the very early days, which was a bit crazy. Then another topic is how you do tracing. So there's the specter attack problem so that you don't have timing. So we have built a complete own system before the workers observability came, to aggregate locks into batches so we can ship them off to Grafana, for example, without killing Grafana. That's another problem when you are at the scale that you just kill downstream systems, basically. Or things like there was no event bus, or there was nothing like R2. I think KV was there at the time when we started, but that was pretty much the only thing. And as I said, I think it's actually great that there was not a lot of things there, because you have built the stuff the way you need it. Maybe I like that you create all these new primitives, but I think you don't need them. So you could build everything yourself pretty easily. I'm happy to hear. Now we have the options. You can still do the hardcore path like you guys did, build everything yourself, or use the managed features from Cloudflare now, I guess. Maybe last question before I hand it over to the audience. I know we met more than once. So you talked to Cloudflare. I think you were not always happy. How did the collaboration with the team work? What can we do better? I think it's very normal that you're not always happy. So that's for sure. Otherwise, there couldn't be innovation to move forward. I think it's very important to be vocal about your expectations and what you're building. So I think this was probably the best thing, that we have been always transparent with your teams. This idea that our system works like a DNS actually came from a solution architect from Cloudflare. Because this was a central piece to solve for us, and we didn't know how to solve it on Drupal objects. And we basically sat with a person for two days in a call until we were like, if we thought about this, and we were like, oh my god, that's brilliant, that's cool. But this only works by trusting you as a vendor enough that we basically get you early into our plans and really let us help us. So let you in. I think that was great. We have talked about so many things. I think we were always very blunt about what doesn't work and what we need. Can confirm, can confirm. Bluntness was present. And I mean, you delivered over time. So some things for sure, it's always nice to have every nice thing from the get-go. But yeah, that was the important part for us. Alex, thank you so much for sharing what it actually means to run something at massive scale in production. Now that you're acquired by Salesforce, I can't wait for your talk next year. 100 million trillion DOs on Cloudflare. That'll be great. So thank you so much, everyone. Thanks, Alex. Let's keep it up.