How we built Cloudflare’s global network

I'm Emily Music. I am the Network Strategy PM here at Cloudflare. I'm joined by David Tuber, Director of Product Infrastructure also at Cloudflare, and we're here to talk to you today about how we grew Cloudflare's global network. We're going to focus a little bit on peering, why we placed our POPs where we did, traffic management, and if you're see some scary slides of things you hope you never have to deal with. So first, Cloudflare's network. You've probably seen this slide a bajillion times, legally required for infrastructure presentations. We're in over 300 cities. We're all over the world. We've got a global backbone, but that's not how we started. So we thought it would be fun to talk to you a little bit today about what things were like 10, 15 years ago. We just celebrated our 16th birthday. So a little look back is what you're in for. So a lot of you are familiar with how our network works, but just to remind you, we are an anycast network. We are focused predominantly on getting as close to our users as possible. Breadth is really important to us, and we run almost all of our services everywhere so that, like Stephanie said this morning in the keynote, no matter where you're located, you can get to what you need quickly. So in the early days, it may surprise you to know we actually didn't set out to build a CDN. We wanted to build a WAF, a powerful, performant WAF. So what we needed to do was filter traffic. We knew that we were going to need to do that on our own servers, and so becoming sort of a reverse proxy was sort of a prerequisite for us building a WAF, and a CDN was born. We wanted to be performant. We had to add caching, and so it wasn't our goal, but we needed it to get to our goal of security. So when we started out, we didn't have the Cloudflare money you see today. We knew that the public clouds were just not in the cards for us financially, and we realized really quickly that if we wanted to build our business, we couldn't sort of take on the fees associated with building on someone else's cloud. So what we decided to do was take the plunge, build our own dedicated Edge network, and this gave us more control over sort of the fees that we were paying in terms of bandwidth, avoid compute markups, all of the fun things you see today. And we also knew that we couldn't really operate and buy all sort of the specialized appliances that we need, so we really set out to offer what we could in terms of software. We moved towards this commodity hardware approach instead of the specialized hardware, and yeah, we decided to run all of these sort of WAF, DNS, cache, DDoS that you know and love everywhere. So today, Stephanie said, we launched in five places. We very quickly tried to expand into these other ones. The approach originally was just to get into these major IXs. You might recognize some of these from today or long ago. In phase one, we just knew we needed to get where it was going to be easy to interconnect. We knew that there was going to be traffic there for us to serve, and so our initial goal was to get into the hubs. Phase two was getting us closer to the end user. So we have a program internally we call EPP program, Edge Partner Program, and this is where we work with local ISPs so that we can get to their network directly. We co -locate in their facilities and make sure that we can serve their networks and interconnect everything, and it gets us closer to the user. So I know what you're thinking. You just gave us the recipe. We buy commodity hardware. We get into hubs. We may be deploying some ISPs. Why wouldn't we just build a CDN, right? I'm not here to deter you, but I don't know if you can see it, but it's under a lot of water there. This is a dramatic reading of one of my favorite emails. There is no security guard on premises. However, there is an aggressive dog that serves as an extra security measure. Whenever we visit the house, the landlord ensures his dog is kept inside before we can access the machine room. This adds an additional layer of security. Now, don't worry. We are a security-first company. These days, we do require at least three aggressive dogs to guard our infrastructure, and then my personal favorite, dear Cloudflare team, the power outage was caused by a squirrel damaging the fuse, resulting in a power interruption. Resolution, the damaged fuse was replaced. Can confirm the power supply to the rack is now stable and operational. Squirrel's fine. Don't worry about it. Perfect. So, now I'm going to pass it off to David Tuber. He's going to talk to you a little bit about how we are resilient. Thanks, Emily. So, as you can see, our network is very heterogeneous. So, we need to build our network to make sure that no matter where our POPs are deployed, no matter the network conditions or the number of dogs or the squirrel situation, that our network can work. And so, a lot of this is actually not done in hardware, right? You can have as many routers and interconnections, but if a squirrel gets into a data center or a data center catches on fire or something happens, then we can't deliver your traffic. So, what do we do? Well, we've got to build resiliently, but also there's a whole bunch of other stuff besides just resilience that we have to care about, right? Let me ask a quick question here. How many people here are running AI workloads? Hands up. Cool. How many people here are running, you know, Zero Trust sassy workloads? Cool. Some of you guys. How many people here are running CDN? Cool. That is a pretty even mix, and all of those look completely different on the network, and we need to be able to handle those. We need to be able to handle stuff like geopolitical fragmentation. Wesley talked about at the keynote. I need my data to be decrypted here and only here, not over there. How do we handle that? We need to be able to handle supply chain constraints. Fun fact, who here has tried to buy RAM in the last 12 months? It's pretty hard, right? So, imagine buying a lot of RAM for a lot of computers around the world. It's also pretty hard to ship to a bunch of different places like the Middle East. It's a little bit of a challenge right now. So, we need to be able to handle all of these problems and also the ones that are not on this list, and we'll figure those out when they show up. So, one of the ways that we solve this problem is just by architecting our data centers differently. Who here runs compute in US East 1? Show of hands. That is actually far less people than I thought. US East 2. That's actually slightly more. So, those sites are pretty important. It would be really bad if something happened to Ashburn, Virginia. It would be really bad if something happened to a Cloudflare data center in Ashburn, Virginia. So, let's go and build our data centers differently. Let's go make sure that each section of computers, each group of computers can fail independently of one another. Let's make sure that we have additional physical redundancies. We group our computers into logical groups of capacity called colos, and we have many of them in a point of presence. So, we call them multicolopops. This ensures that if something happens in Ashburn, Virginia, we don't take the entire popdown. That means we can still serve traffic in Ashburn, Virginia, even if a squirrel gets into a power generator, although I would imagine that's probably less likely to happen in Ashburn than other places. We run this in all of those major cities, and I'm sure that if I asked you to name some of the most important cities on the Internet, you could do that. Ashburn, Frankfurt, Singapore, London, Amsterdam. I could go on, but we don't have that much time. And all of these are running Cloudflare machines. And as Emily pointed out, and as Stephanie pointed out, and anyone who's probably going to talk who's telling you will point out, all of these can run any Cloudflare service all the time. And that ensures you get single-pass protection, single-pass performance, single-pass AI stuff on every single machine. And if any one of those machines fails, it's okay. We can just move on to the next one. And that data center architecture allows our single point of failure to be as small as possible. If we have more things that can fail, then there's more opportunities to serve your traffic. And that's thinking about things from a resilient level. And Matthew and Emily and everybody talked about how we do that. This is why. We want to make sure we can always serve you no matter where you are and no matter what's going on on the Internet. And a lot goes on on the Internet. For example, cables get cut. The North American fiber-seeking backhoe is a very hungry beast and will strike anywhere in North America and also in other places like underwater. And that will cause disruptions on the Internet. We need to be able to handle these shifts of traffic when they go from place to place because things just break on the Internet. And to do that, we've built autonomous traffic management systems that can detect that there is a problem and move traffic somewhere else so that that traffic can be served. Our principle is we want to serve you as best as possible all the time. That doesn't always happen all the time, but we want to make sure that no matter what's going on, we can serve you to the best of our abilities. And that combined with our resilient machine architecture and data center architectures allows you to be resilient. But it's not just about data centers. It's also about the network. You saw all of those connections and all of those long lines on the Cloudflare network map. We need to be able to make sure that for those paths, we can still make sure that traffic gets from point A to point B. So we have this product called Argo Smart Routing. And it's actually very simple. Who here has heard of Waze? I feel like Waze. It's Google Maps now for the younger people. Google Maps will tell you and Waze will tell you if there's a traffic jam or there's construction or there's an alternate path that will get you there faster. And Argo is essentially the same thing. If there's a problem on the Internet somewhere and your traffic is going to route through there, Argo will essentially tell our network, hey, there's actually a better path if you go this way. And so we'll send traffic that way. And we can detect a whole bunch of different things. We can detect latency increases. We can detect packet loss. And we combine all of that into a globally distributed route optimization engine that ensures that you're getting the best possible performance and the most reliable performance you can get based on the underlying infrastructure that we have. And Argo doesn't just make you more reliable, it also makes you a lot faster. We can make you up to 33% faster and sometimes even more, depending on how your network and how your use cases and where your users are. And we talked about fast. Let's also talk about reliable. This is a nice, fun little Greek mythology joke for those of you who either haven't seen Hades Town or didn't read mythology. Orpheus was actually the navigator of the Argo. And so Orpheus steers the ship to make sure that you get around the choppy water. So we have a metric internally that we like to call origin reachability, which is how often can Cloudflare reach your origins. And around 2020, that metric was about 99 .9%, which for those of you who run enterprise networks is probably not a very good number. We rolled out our first version of Orpheus in June or July, July of 2021. And when we did that, we gained an entire nine. Who here has worked on enterprise networks before? That is pretty hard to do, right? Gaining an entire nine. So we gained an entire nine just by rolling this out. And we're continuing to improve it and continuing to catch those edge cases and those new failure modes so that we can make sure that we steer around that choppy water so that Jason can chop off Medusa's head. Although it's not Medusa correctly. And all of this makes us not just reliable and fast from, you know, colo to colo, from site to site, but it makes us super fast on the Internet in general. Stephanie gave that metric. 95% of the Internet is within 50 milliseconds of us. That's a really good goal because it means that no matter where you are, you're getting the best possible performance. We're running workers and running applications and agents right next to you. And that's not like a really big deal if you're in San Francisco and the nearest data center is like five milliseconds away. But if you are, let's say, in South Africa or Jakarta, actually not Jakarta, but if you're in, you know, Bali or in Indonesia, or you're in Africa, or you're someplace very far away from where the major Internet sites are, having a worker and having your stuff run super locally is actually a game changer. And we've heard from a lot of customers that this ability to be close to users is actually a difference maker when choosing a provider. And we're pretty fast. We're actually much closer to our users than anybody else. If you've seen the Cloudflare blog post during any innovation week, you've seen a post like this, where we talk about how fast we are. We talk about how close we get to our users and how that changes our last mile RTT. And we're the fastest network in 60% of the top 1,000 networks. And we can't get this way without our network strategy. Get close to our users as possible through either major data center, major interconnection points, or by embedding in last mile networks. That gets us literally that extra mile. So summing up, whether it's snow or sleet or rain or fire or dogs or squirrels, Cloudflare has got you. And we do this for every data center in all of our 400 data centers around the world to make sure that you get the best performance no matter where you are. And Emily said, you know, just in case you were thinking about building this yourself, you can go build this yourself. None of this is actually anything wildly crazy. You can go and go rent data center space today. You can go get peering. You can go set up a network. You can go buy an ASN. You can go buy IP addresses. You can configure them. You can advertise them. You can negotiate with transit providers. And oh, my gosh, that sounds incredibly insanely hard. And it's so totally different than what you want to be doing. And so our hope is that you look at this and you say, this is hard. Because it is. But also at Cloudflare, this is our job. And so hopefully we do a pretty good job. And the reason we know this is that data center fire, literally we were back up and serving traffic the next day thanks to the hard work of our data center providers and also all of our network and infrastructure operators. So we've got a really good team that's working around the clock to make sure that you are getting the best performance you can. Thank you so much. Thank you so much.