Scaling Discord’s voice on Cloudflare’s edge

My name is Jonas and I lead the audio video team at Discord, and I want to start by telling you a small story that will make sense in a bit. It is February 2025. It's 9.30 at night. I'm in my basement, sitting on my trainer bike. I thought about bringing a picture, but I don't think you want to see that, so I cut that out. But I'm in this virtual training thing called Zwift. I'm pushing up this hill. You know, I have music in my ears, really in the zone, and the music fades down. I get the Discord notification sound, and it's super annoying because I'm one of those people who do not have notifications for virtually anything. I have notifications for my wife, my very close friends, my CTO, and my VP. And my wife is asleep, and my friends are asleep because they're in Sweden, so it must be either of those two is the last one. So then sort of curiosity gets the best of me, and I was like, ah, do I really, like, stop my workout to figure out what's going on? So I get my phone out of my pocket, and it's a five-minute-long voice message from my VP at 9.30. And that wouldn't be so weird to get a five-minute-long voice message, but I happen to know that my VP is with Cloudflare in the East Coast, so it's a bit later for him, and I also happen to know that there may have been a couple of cold ones involved in the situation, so you kind of know that something is coming out of this one. So I'm like, okay, let's listen to what he has to say. So, you know, play the voice memo, and there's a lot of back and forth, and eventually he gets to the point. And the point he's sort of come to is, like, what if we took the Discord's voice engine and we moved it to run on the Cloudflare Edge instead of what we're currently running on, because we're currently running this on a bare metal cloud combo. And at the time when he said this, like, Discord is a heavy user of Cloudflare. We use their Ingress products, we use workers, we use their CDN, we use tons of Cloudflare already, but hadn't really been on or even radar to run this workload on the Cloudflare Edge, if that makes sense. So I was like, okay, you're crazy, and I was like, we are, you know, Cloudflare is mostly TCP, our workload is UDP, there's tons of reasons why this is not going to work. And, you know, you initially start with this, like, this is not going to work, if that makes sense. And today I want to tell you a story, like, how we ended up making this possible and why that voice message wasn't that crazy at the scene, right? So if you don't know what Discord is, if you have not used Discord, hopefully you have, but if you haven't, Discord is where people come to sort of talk before, during, and after they play games. We're pretty big at this point, we have over 200 million monthly active users, and the kind of things people do if you haven't used Discord is you kind of find a voice channel with your friends, and you go into voice channels and you sit with them for a long amount of time when you play games or just hang out. You can think of it as like a way to talk to your friends over long periods of times. So instead of like Google Meets or something like that, where it's a meeting that's 30 minutes long, these are very long sessions, they can go five hours, some sessions are days, we've had sessions go for months, so there are these voice sessions that can be very, very long. So we have to build our infrastructure to handle very long sessions, if that makes sense. When I say that I lead the audio video team, I'm not really talking about videos and webcams, because the sort of secondary thing people do when they use Discord is use our feature called GoLive, and GoLive is this feature we built many years ago that allows you to just click a button, and we hardware encode, we do a lot of accelerated stuff to sort of shoot that video feed from your game to someone across the Internet, your friends often, and we're doing it at super low latency because the sort of experience we want to give to people is being in the same room. So you can think of what we're trying to optimize for is like you want to feel like you're sitting next to someone when they're playing the game, if that makes sense. And that's the bread and butter of our application, right? We do voice and video, and people use this in all kinds of weird ways that we hadn't expected. When we built this feature, we thought people are going to use it for streaming games, but people stream homework, they stream web browsers, or mobile. A very big use case we have is people actually stream themselves watching Instagram so people can follow along in your feed. So these use cases I don't think we had imagined when we built this, but that's the kind of scale of what we're supporting with this today. And I think that's been true since we built Discord. You always start by building a product, and you have assumptions of how are people going to use your product, and then when you sort of shoot it into real life, you quickly learn that most of the assumptions you have are wrong. But the good part about our voice and video is that we've done this since we started the product, so we've done this for 10 years at this point. So we've learned a lot about what works and how are our users using the product, if that makes sense. So going into something like this, we could really look at the data to figure out what are we actually solving for, which is a much better position to be than like, hey, we're solving for unknown, if that makes sense. So how do we do it, or how does Discord work? And like, how do we do voice and video under the hood? As I said before, we use Cloud for a lot for the application, but for the voice and video, we are today a WebRTC shop. And if you don't know what WebRTC is, WebRTC is a protocol that allows you to sort of communicate between web browsers and applications. It was initially sort of funded and developed by Google and Ericsson, and they took a lot of telephony standards and they just shoved it into a protocol and then released to the world. So it's a bit archaic, but it's very stable, and it's virtually support everywhere. So the cool thing it gives us is we can support browsers, we can support apps, it kind of just works out of the box. And the most common thing I hear when I talk about a product is people are like, oh, of course, that allows you to do peer -to-peer. And this would mean that every client sends data to each other because WebRTC allows this. If you just set up a browser simply, it's like most people do, and people assume that we do it, but we don't do peer-to-peer. What we do is we use a relay. And we use relay because there's a couple of reasons we figured out very early on as we were building Discord. The first one is stability. If you do peer -to-peer, you often end up in these situations where you have asynchronous peering. So you might be able to send packets to your friend, but they cannot send packets back. So when you're trying to talk to them, they will hear you, but you cannot hear them, and you kind of get into all these weird issues. So you have to use a little server to control the network, if that makes sense. But the second one for us, which has been a selling point of what we're doing, is that we want to give users the privacy. Now you're thinking about, what do you mean privacy? A big reason, actually, that Discord got popular is that back in the days, 10 years ago, the alternatives that were peer-to-peer allowed people to discover your IP. And you think this is an edge case when I tell you, but what would happen in games is that you would play against someone and you would beat them in the game, and they would rent a small bot farm to DDoS you, so you would lose in the game. This would happen in reality. And especially with streamers that had sort of a larger audience, this would be a very common thing. So they just stopped using tools like Skype and other tools, and they sort of came to Discord because what we offer is this guarantee that when you use Discord, we keep your IP and you behind a wall, if that makes sense. So the problem if you build a relay and you're sort of naive about it is that if I stream video to the relay and the relay fans it out to everyone, that's 12 streams of video, and that might not make sense if not everyone is watching it. But you can think of, like, you have four people streaming video and then one person is watching it. Then you're fanning this video out four times 12 times. That's a lot of bandwidth you're wasting. And especially when you're trying to give away video for free to everyone on the Internet, that quickly breaks the bank. So what I think a lot of people, and we do, is we build what's called an SFU. It's a Smart Relay. It stands for Selective Forwarding Unit. And it's this sort of server that figures out, like, hey, who is watching, who needs the video? So if you're on a call with someone and you send video out, then the SFU knows exactly who needs that packet and decides in real time when this gets fanned out. And it's a much better use of bandwidth, but it comes at a cost in terms of processing because WebRTC is a very stateful protocol. So in order to know exactly what packets you need, you need to keep track of every packet you've sent and, like, where they are on the stream and SSRCs. So you're kind of trading the decrease in bandwidth for more compute that you have to spend, if that makes sense. You can't have the niceties of an HTTP CDN. And you might ask at this point, like, why not use a CDN? Like, there's a lot of companies that are trying to solve these problems. There's tons of commodity CDNs. There's technology like HLS and Dash. Like, we've solved this problem as an industry. I used to work at Twitch. Like, they've solved that problem. YouTube have solved it. Meta have solved it. So why would we not use a commodity CDN to solve this problem given they're optimized for video? And I think it's sort of important to go through this because it kind of makes sense later how we ended up going to the edge in the first place. The first one is that CDNs are often just optimized for, like, a one-to-many situation. So, like, when you build a CDN, what you want is a big fan-out. And the fan-out has to be economically sensible in the way you can give it to as many users as possible. Our usage is few-to-few. So the value you get from a fan-out is not really worth it to us. And we actually end up just paying more for that fan-out than we would gain. And the second one, which I talked about earlier, is latency. Like, what we are trying to give users is, like, a real-time experience. You know, even if you use something like LLHLS, then it's really hard to get under two seconds. Like, getting under one second or a sub -second is... It can be done in a test lab, but, like, in the reality, it's very hard to do with HTTP CDNs. So if you think about it, if you're sort of watching a friend and they scream, like, oh, my God, and then it takes two seconds for you to see that video, that breaks the immersion we're trying to give users, right? So we need it to be low latency to kind of give that immersion. And I think, you know, there didn't used to be good CDNs that did WebRTC. That has changed 10 years later, but, like, in 2015, if you really went and tried to find someone who was doing WebRTC CDNs, that didn't really exist. Most people point to you to the libWebRTC project. If you have ever worked with that and tried to build a server, you know that's a scam. But, like, it's one of these things where the industry has advanced, but even as the industry has advanced, it's really hard to make the control plane scalable across these different CDNs. So you end up having to build your own control plane anyway. And, like, when you build your own control plane, then, like, the value of the data plane is not really that good anymore. So the problem with WebRTC that I think we are facing and the industry is facing is it's really hard to standardize this, right? And we are solving this as an industry, like, media over quick. Side note, like, actually developing to fix this problem so we can commoditize low latency video and audio on the Internet. But it's not really there yet. And, like, for our case, since we've been doing this for 10 years, we're still on WebRTC. So I mentioned we run on bare metal. And our goal, even before we started this project with Cloudflare, has been to get off bare metal. And there's a couple of reasons we want to get off bare metal. You can probably imagine a lot of them. But I also will say that this was before the sort of AI thing happened, like, before stockouts happened daily. So in 2023, this seemed like a very sensible decision that might not be as sensible today as it was back then. But it was sensible then to say, like, hey, the public clouds have excessive compute that you can use. And we really wanted to make use of that. Not specifically because we believe that there will be value in getting better CPUs and RAM, but because we believe that if we could get access to better network accelerators and better network, we would be able to do a better job of serving our users. So we saw that, like, both, you know, AWS and Google had perfected how to sort of do accelerated networking from the Linux kernel all the way down through their host. And that was not as easy to get on a bare metal host, especially what we were running on before. So we wanted to be able to make use of this, and it seemed like public clouds would be where we could do that, if that makes sense. So when this voice message showed up, right, we were already in this mind of, like, we want to move to something different. We are prepared to re-architect our sort of, like, crafty bare metal thing to fit on these public clouds. But we hadn't really imagined Cloudflare being that choice, if that makes sense. And then this voice message shows up. And I use Cloudflare a lot personally. I built a lot of personal projects. And again, we use it a lot as a company. But we didn't see it as, like, a CDN. I think if you think about the CDNs, by the way, and you ask them, like, hey, can I have, like, low latency, they will happily sort of give you low latency, but they won't give you the low latency with WebRTC. So the problems that we came back to Cloudflare with after we started talking to them is, like, first of all, you know, if you think about what they run, like a worker runs per millisecond, but as I said before, a call can run for an hour, if that makes sense. It can run for many hours. It can run for days. So, like, how do we bridge this gap between what we are trying to do and what they are offering, right? You have durable objects, but at the time, durable objects didn't support UDP, so we couldn't run SFU. And specifically with the sort of data volumes that we're talking about, it wasn't really cost competitive for us either to run on something that durable objects. So we really had to answer the question for ourselves, why would we pick going with Cloudflare over an established vendor when we knew that we could get what we want? And there was something as we started looking at the data, because, you know, I would admit that I was sort of initially pretty dismissive of this idea, but I gave it a fair shot and we looked at the data. And when we started looking at the data, the thing that stuck out to us, that made us be like, okay, this might actually be a better choice than we thought, is that 75% of all Discord calls are local. They're within the same geographical region. And sure, if you meet friends on the Internet, you play World of Warcraft or whatever, you create a guild, you might end up talking to people that are not local. But for most users, they meet their friends in real life. They meet, like, people from school, people from clubs, et cetera, and they end up building connections where people live close to them. So with that in mind, if we look at the data, it's sort of, like, obvious to us that if we could do the work to be more hyper-local, we would serve this use case much better. And that's what's kind of, like, made us interested in going with Clover. Because if you think about the problem, right, if I am here and I sort of send a packet and it goes across the Internet for the Wi-Fi, the ISP, peers, and then eventually we show SFU, and then it has to go all the way back only to come out four houses down, that's a very long round trip for a packet just to, like, sort of come down the street, if that makes sense. So if we can shrink this distance, we can do a better job. But now you might be wondering, like, if we sort of shrink this zone, what do you get from this? Why would this matter to the user? Because hyperscaler networks, they're pretty good. If you've used AWS and if you've used GCP, you know that the networks are super good. But the thing for us that we know with our traffic, because we're not TCP, we're UDP, is that more distance means higher packet loss. So for us, like, if you're not familiar with it, I'll sort of bore, like, the thing with UDP is you don't have the sort of error correction and the negotiation back and forth, so you just kind of send the packet over the Internet and hope for the best, right? And that's how you get the low latency. But for us, because the more steps that is in between you and the SFU, the more chances there are for you to drop that packet on the floor. And we fix this problem in WebRTC by doing stuff like buffering, we do expand, we sort of paint over the audio, but we can't really fix the actual issue, which is that, like, higher loss means that we have to at least, like, higher latency, and it's bad for users. And you might be like, oh, it's fine if it's small glitches here and there, but what we've found working with our users is that it's fine if you're on a call for 30 minutes on a meeting and you speak for, like, five minutes because it's sort of easy to paint over that with your head, and it's fine if you're talking to your parents because you kind of know they're going to tell you to drink more water, like, it doesn't really matter. But if you're on a call for five, six hours, these small imperfections start smattering because they become friction, and that friction over time, especially looking at user sentiments, become a problem for our users. So we can literally see a correlation between user sentiment and packet loss. If packet loss go up by 1%, then your sentiment, you know, moves in another direction. So for us, like, improving the loss is the most important thing for us. So what we really wanted to get is, like, if we can have fewer hops, we can have less loss, and at least a way better cause. So the kind of idea was that, like, hey, maybe it makes sense to shrink the zones because it gives us something we cannot get from the big hyperscales, right? As I mentioned, they will happily tell you, like, if you want to get close to the user, you can use their CDN product, but their CDN product cannot run the SFU. And we really believed that Cloudflare was the same, and we talked to them, and, like, we kind of went back and forth, and they basically said, like, well, you know, we don't do what you want, but we kind of make it happen because we also want this to be the future, if that makes sense. So, yes, we worked back and forth, and it took a while, you know. We tried to initially figure out, like, can we use their existing concepts? Is there things that you do that we could use and re-architect our system? We thought about durable objects. But one thing that was very important to me throughout this entire experience is that we didn't want them to create something that was bespoke. If you've ever been on the other side of this, you're a customer and, like, a vendor, or if you've been on the side of building something for a customer, and you sort of build something that is bespoke, you know that the second that customer is happy, you'll start thinking, stop thinking about it, and, like, no one touches that for four years, and that kind of falls apart. So we didn't want to be on, like, a side-par car of, like, the sort of network. We wanted to use something that Cloudflare invested in and believed in. And we sort of told this to them, and they were like, you know, actually, we've been thinking of something that might fit the bill. It might not have what you need right now, but it's, like, sort of in the direction of where you want to go. And that's what today is known as Cloudflare Containers. Back then, I think it was called Cloud Chamber, but today it's known as Cloudflare Containers. And there was a lot of things they hadn't really imagined. I don't think they had imagined someone running this with UDP traffic. There was tons of other things. I think they thought about this as, like, sort of short-lived, not living for hours on end. So there was some sort of, like, friction we had to figure out. But, you know, to their credit, we figured it out with them, and we started deploying our traffic on these sort of Cloudflare Containers. And the cool part about this is, like, it literally took us three weeks from the voice memo to where we're starting to get actual traffic on production, right? So the voice memo, and then we had sort of, like, back and forth, like, will this work out? And February 18, we sat down with the team and said, like, let's do this together. They gave us access to the alpha, and we started working with them, and we got it done. And on February 26, what we tried to figure out with them was not specifically, like, will it work? We were trying to validate our thesis. Because at the time, we were like, are we crazy to do this? You know, is this going to work? It's one thing to sit on your bike in the basement and sort of ideate these, like, wild ideas, and then it's another thing to take them into production. So we wanted to figure out, like, is there a way where we can test this hypothesis in the real world without impacting users? So we wanted to find a region we could do this on. We needed a region that had its own language. So if it has its own language, we know that traffic are more likely to be in a certain zone and kind of spread out. You know, if you speak English, and you're next in our English-speaking country, there's a high certainty that that traffic moves further. But if you have an isolated language, it's a high certainty people don't know the language and just naturally don't speak outside of that. It makes sense. We want it to be a geographic isolation. So, you know, just naturally from a network point of view, we could get ice capture data in a smaller sense. And lastly, we wanted to have lots of Discord users because if we didn't have Discord users, we wouldn't get the data. So it had to be a place that, like, had high sort of penetration of our product and a lot of people in games. And there's one place in the world that fits this bill, and that is Iceland. Happens to be in Ireland. They have their own language, and they play a lot of games. So it's perfect. And the good thing is you might be like, oh, did we use them as a sacrificial goat? We did not because by the time Iceland was transiting all the traffic from Iceland all the way to Rotterdam, so they were already not having good experience. So what we thought is like, hey, we can actually test to do this and hopefully it would make their lives better. So it's kind of win-win. Like, we get the data, and they get a better experience. So we worked with them, and on February 26, we turned on Reykjavik, and as we had hoped, we saw a 9% decrease in ping and 11% decrease in packet loss, which was just huge, by the way. Like, it might not seem like much, but we would celebrate that we saw a 1% decrease of packet loss. The 11% was just like, huh. But the problem that quickly became apparent to us is that we saw a 2% ping increase for everyone else outside of Iceland that were talking to people on the calls. And it was a bit like, you know, in hindsight, it makes a lot of sense, but it was a bit of like a head-turner for us in the beginning. Because what we used to do is that the first person who would join a call would place the call. So if we have one person from Iceland join a voice channel, then the survey gets placed in Reykjavik, and then if you have three people from Germany, then all the other people have to go back to Germany to get that data. The problem for us is that the case improved for Iceland, but the average got worse for everyone else. So now we have this question of like, if that's the case, then like maybe this local thing won't really work out. So what we kind of figure out from that is like this coverage isn't placement, and we have to be very careful when we went forward to figure out like, how do we do this placement in a way that works for our customer? So instead what we did, we kind of went back to the thing, and we looked at what we could do, and we decided to do like Rotterdam and the surrounding region, because we believe that like, you would have a better experience if we did it in a place where like just when you weren't geographically distributed. And we had sort of also gotten like, you know, more confident that it was going to work from Trident and Iceland, so we felt more confident doing it. So we deployed it on Rotterdam, and it was great for almost all ISPs. We saw similar gains, but we started seeing these other weird outliers. There was specifically one French ISP who saw like over one second of latency at its peak. And if you're ever talking to someone and suddenly the responses come a second later, you can just imagine how bad that feels. And then we dug in and tried to figure out what was going on, and it turned out it's like not the servers. They were not doing well. You know, there was nothing with the software, and there was nothing with the network. And the thing that we then realized is that before when you have a hyperscaler, let's say you have USD-East-1, all the ISPs optimized to get into USD-East-1, if that makes sense. But if you have CloudFront and you have a massive distribution, then it might be that a small ISP, they optimize to get into the small one that is more local to them. They don't optimize to get into the one that is in Germany, if that makes sense, if you're France. So we kind of ran into this problem that we created more front doors that goes into our traffic, and we didn't do a good job of figuring out how to route the traffic backwards. So like, you know, if the traffic come into France, it would go across the CloudFront's backbone, but then it would be dropped on the floor like a hot potato and go back over the public Internet on the way back. So we kind of had this asymmetrical routing problem as we were talking about before. But CloudFront has solved this problem. It only turns out that, you know, for all the stuff they do, they have solved this problem. They have a great backbone, but we didn't get to ride on this backbone because we were sort of dumping UDP and they didn't really deal with this. So another thing we had to work with them to figure out is like, how can we do symmetric routing for UDP? As in like, how can they keep track of like where packets go in and out so they translate over their backbone so you get the same network quality for everyone without sort of getting into the public Internet problems because then you're just back to the peer-to-peer problem that we had before. And then the sort of second question we had to figure out is like, how do we run this, right? We said earlier that we didn't want to have anything bespoke and we were promised this magical land of CloudFront containers. So we said, let's go for this. But at the same time, we were rebuilding our software and we were rebuilding our software to fit on the public cloud because again, there was a lot of things. If you run bare metal yourself, you know that you can like ignore a lot of the problems and just like hope that they go away. But when you're on a public cloud, like stuff like maintenance out of your control does not go away. You just have to figure it out. So we had already started to rebuild our SFU to solve some of these problems. So lucky for us, we had built an SFU that was like built for the modern kernel. It has all this packet acceleration we were talking about. It was small and flexible. So like it could run on one core, it could run on four cores, eight cores, and it was a single threaded runtime per core. So we just knew that, or sort of we believed that it would scale linearly with cores. Of course, that turned out to not be true. And then one of the last things we wanted to make sure is that calls can move. So when we run bare metal, a call placed like, you know, we know that server is going to be up for a couple of months, but in our case, when we run on containers, they can come and go. So the SFU itself has to be willing and able to say, I know that I'm going to get shut down. I'm going to move this call over to somewhere else and move that traffic in real time without the user noticing. This was a huge undertaking by the team, but they managed to get this done, which unlocked us being able to run this on containers, if that makes sense. So we did all this and it went really well. And we started starting scaling it over the year. And by the end of the year, in December 2025, we had moved a lot of traffic to Cloudflare. Actually, it was almost 80% at that point. And I remember we had like an event in the office and we were celebrating. We were like, we're finally done. And we celebrated and we sort of, you know, we did a ceremony where we turned off the last bare metal pop. And we were all super happy. And, you know, we're like, we're going to celebrate this. And immediately, we started seeing issues. Like what we learned is like, the fun doesn't end when you sort of finish your migration. And it was really weird for us because we've been scaling slowly and it's been great so far. But as we kind of cranked the lever, like issues started popping up a lot. And it was really weird. So we tried to figure out like what is going on. Because what we started seeing was that like, when we were off peak, the traffic was fine. We were doing really well. We saw great improvements. But as soon as we went on these peaks, we saw latency spikes that were like, you know, very, very bad for our users. And it went so bad that like users started figuring out, because you can see in the application, if you're on a Cloudflare host and a ESP host, they started figuring out that the CF's host were bad. So they like started disconnecting many, many times to get another host. Which just made everything worse. So we ran into this like really adversarial problem with ourselves, if that makes sense. So what was going on here? We saw over 500 milliseconds of ping speak at peak. And at this point, like seeing this, I just literally told myself like, what have we done? Like, have we messed this one up? You know, have we sort of bet on the wrong horse, if that makes sense? But I think we went back to Cloudflare and we sat down with our internal team who did a fantastic job. And we were like, how do we figure this out? Because initially we were like, you know, it's probably us. It's probably not Cloudflare, but like we couldn't figure out what was going on and we couldn't be certain. We don't want to blame the vendor, if that makes sense. We don't want to be like, hey, you are the error here. So we have this problem where we're trying to figure out how do we figure out what is going on? We went back and forth. We added logging and we tried different logging methods. But it's really hard to see the issue. But then what Cloudflare came up with and then we implemented with them was really smart. So we have this health check packet that gets sent between us and back and forth that has a very specific signature. So what Cloudflare did is they basically wrote an eBPF filter to drone kernel that would timestamp this packet. So that allowed us to understand where is the latency coming from? Is it coming from the network card to the host OS? Is it coming from the host OS to the VM? Or is it coming from the VM to our app or then our app out? And of course, after they wrote this and we saw the statistics, it was very obvious that the problem was inside our app. So that was like, okay, great. It's our fault. You know, you feel a bit deflated. But to the credit of the team, they took this with great spirits and they're like, let's get to fixing this problem. And the good thing is it pointed us to where the problem was happening in our application. Because what we would see is that sort of one core would spike. And when the traffic sort of started scaling up, the core would spike and it was a very clear correlation. It was like the way we run it, like I said, like we have a single threaded runtime per core. But the old hosts we were running on bare metal, they were not the most modern. And they didn't have fast CPUs and they didn't have fast network cards. So when we had written SFU and deployed it on the old network, you know, it kind of just worked. But the problem we run into in this case is we would have a loop that would basically do like receive and then send and then flush. But the new networks were so fast that we just kept getting packets on receive and we never got to the send and flush. So we were kind of starving ourselves because the loop would never continue. So we hadn't really anticipated like how much traffic we would be able to push on these modern CPUs with these modern network cards. And it was kind of like a learning for us and we had to provide it. So we fixed this problem. And we felt very good about like, okay, you know, we got it. We're great. And we started to deploy it again. And we did two, two, two, two cores. Sorry, two cores scaling and it went fine. And we did four cores scaling, it was fine. And then we did eight cores and it wasn't fine. And 16 cores was absolutely not fine. And then we're back to the same problem of like, okay, what is the error here? But this time we were sort of like, you know, last time we blamed cloud first. This time it cannot be them. It must be us. So I knew the team was like, they really went through every single thing. They looked at every line of code. They went through, you know, all different test cases. And eventually we're like, I just, maybe it's not us, you know, we don't know. So we make the cloud first sort of head down and we talked to them and they took it seriously. To their credit, Cloudflare took it super seriously, as serious as the first time and they worked through with us. And what we learned looking into the metrics and kind of what like led us on to that was outside of our control was that we would see a core on the box. Sort of what we did is like, we sort of wrote a tool that will sample the cores. Some like H top, but like for ourselves. And we would see that like one core could be continuously pegged and then every other core would be fine. But this core would just like sit pegged, if that makes sense. Kind of similar to the thing before. But in this case, we were like, we know it's not us doing it. So like, what is pegging the core? And it turns out, as I said before, that Cloudflare Containers is built on Firecracker. And Firecracker is just like, it was built by AWS for Lambda, basically. It's this idea of micro VMs. You have many VMs on a host and they're all small. But we were kind of doing a macro VM. So they hadn't really intended for this kind of workload to be run the way it is in Firecracker. So the problem we ran out with is that in Firecracker, you sort of get an assigned core that takes all the network traffic in and then sort of fans it out to other cores. And as we were scaling the other cores, that core got extremely hot, just processing all the packets that have to go to every other core, if that makes sense. So we're kind of just running into this kind of like, it was a single pipe in and a single pipe out out of our containers. And that made it impossible for us to scale above four CPUs without sort of burning that core out. And once we kind of learned that, and we talked to Cloudflare, to their credit, we came up with a fix, which is like, we're just gonna do four core distribution and we're gonna do many of them, rather than like do one, because it kind of fixed the problem. And then longer term, the Internet, the packet steering, so they can now have multiple of these and figure it out, so we don't run into this issue. But those were like the kind of problems we faced when we were scaling this, but they fixed it. So we were super happy. And today it's a solved problem. We're good. I'm really proud of what we ended up here. We have just a few running in production. We managed to get this transition over and the results of this is really where I think we are extremely happy with where we ended up. We've gone into 80% of voice and video that runs on Cloudflare today. And if we take one region, Frankfurt, we saw a 34% decrease in ping and a 42% decrease in packet loss. I said before that we would celebrate 1%, so 42% decreases for us is like magic. Like we worked for 10 years to get this thing better and this kind of just like unlocked that for us in ways we couldn't have imagined. For Europe and Latin America, we saw between 20% and 60% decrease in packet loss. Again, crazy numbers for us. If you told me this upfront, I would have like dropped everything I did and worked on this immediately. It wouldn't have even been a question, right? And then the last thing that I think is very interesting, when we measure our user sentiment, right? We look at Santiago because we use a, the thing I told about before, the algorithm that pads over missing bytes, we call it Expand. It's like a thing that's built into Opus and WebRTC. We saw 40% decrease of this in Santiago, which means that we know that the user experience has improved. It's not just metrics, right? It's like the actual padding that we do that's kind of like weird. I should have brought a recording with me, but I didn't. But like the kind of weird sound you hear when you're losing packets on a voice call, that is down by 40%. So we know that this is impacting users and it's good. So this is what you can do if you run computer at the edge. Like, and we were super happy where we ended up here. But the thing that I think for us was so magical, it's like there are products that we couldn't even build before that we can build now. We've had ideas for years of like, what could we do with our client? Like we run a client on people's computers, right? What could we do with this client if we had a low latency network? And I think now we're in this idea, we're kind of building on two things that I can't talk about, but we're building this idea that like, it unlocks products that we just couldn't even imagine before. And I think that is what's so empowering for us and why we feel so happy that we did this. So the last thing I wanted to sort of leave you with, like if you're gonna do this, I think take time to rethink your traffic. Like if you believe one thing about your traffic and you really start looking into how your traffic really works, you might learn that you have the wrong shape of the optimization for your entire world, if that makes sense. And then as you do this, you kind of have to build the software for new configurations. I think the mistake that we run into often is that we try to take our old software and sort of slowly shift it. I think we should have done earlier is to be very clear about like, we have to take in your software and kind of build it from the primitives as we learned. And I think the third one, the most important, like you're gonna find a partner that is as crazy as you to do these things. And with that, that is all I have for you all today. Thank you very much.