Cloudflare’s Network Doubles CPU Capacity and Expands Into Ten New Cities in Four New Countries

Good afternoon, everyone. Thanks for joining us on Cloudflare TV. I'm joined by my guest, Jon Rolfe from our infrastructure team. We're here to discuss a recent blog post article Jon has written on Cloudflare's infrastructure plans, specifically regarding our capacity. Yeah. So hi, I'm Jon. As Peter said, I work on business operations strategy on the infrastructure team, specifically on the network strategy team. So in practice, what this means, because it's a super big title, I do a mix of like special projects, data analysis, and regular operational work, including obviously keeping an eye on our network expansion into new markets, new concepts, and all that fun stuff. Excellent. And my name is Peter Yoakum. I'm a solutions engineer here at Cloudflare. I work with our customers as well as our future customers, helping meet their problems and initiatives with Cloudflare products. Just to give a little bit of a recap on what Cloudflare's network is, Cloudflare operates one of the largest edge networks in the world. We utilize a property of Anycast to steer clients closest to our eyeball co-locations. We're within 100 milliseconds of 99% of the Internet connected world. We actually have a really way more impressive stat than 100 milliseconds, 99% within 100 milliseconds. So actually keep your eye out on our blog for that. And shout out to David Toober, the PM for our network. Oh, I actually forgot, there is one more introduction. This is my cat named Pomelo. She's very cute and she loves when I'm on video call because that way my computer gets hot. Anyway. I'm super glad you were able to bring your special guest on with you as well. Me too. So let's just jump right into it. Can you tell us a little bit about the article that you've written? So, I wrote a article on how we've added a bunch of new cities and doubled our CPU capacity over the last year. And I think there's kind of two halves to this that are equally important. Obviously, talking about cities is like really great, really fun. But quickly, I just want to talk a little bit about our CPU capacity expansion, which is mentioned just briefly in there. We doubled our CPU compute capacity, i.e., we can now handle two times the amount of customer load as we were before, which has really been instrumental to handling the huge surge in traffic and shifts in patterns of how the Internet works over the past 18 months. Which has been a huge logistical challenge that we don't speak much about. But it required an entire revision of all of our capacity plans. We normally were very data driven. We send servers to where they're most needed, when they're most needed. But it was kind of like a surprise. All of the adjacent networks that are really skyrocketing are different. All of the places where we had planned on seeing XYZ percentage growth instead saw Y. So, there were things like we had to divert shipments, and this was in times where shipments weren't even always going through properly. So, thanks to the awesome work of our logistics team there, along with our network engineering team, which allowed us to kind of make smart routing decisions that we weren't able to before, or even smarter network routing decisions to kind of balance this load across colas in a bunch of new and really inventive ways. Very interesting. And then, in addition to kind of the capacity issues. One of the things that we've done with our recent upgrades is we've added expanded our presence into more cities globally. Some of the specific cities I'll just run through the list here. We've got Tbilisi, Georgia, San Jose, Costa Rica, Tunis, Tunisia, Yangon, Myanmar, Nairobi, Kenya, Cheshire, Bangladesh, Canberra, Australia, Palermo, Italy, and then Salvador and Campinas, Brazil. I want to ask you about maybe some of these specific co-locations. Can you tell me a little bit about the Palmero, Italy location. Yeah, so we looked a little bit at that in the blog, but I think that's a great example because that's one of our more traditional deployments, I would say. It's a geographically and economically larger country, large population. We didn't have a colo in the south of France, south of Italy, which meant that a lot of Southern Plain traffic was being routed up to, you know, Milan or Rome, rather than being served locally. So I think that was kind of a, an obvious kind of slam dunk of Nupal. Yeah, absolutely. Tell me a little bit about say Nairobi, Kenya. Yeah, I like that example as well because kind of the almost the opposite, in a way. It's obviously super high population, but it's not a traditionally like super high revenue place in the same way, but it's one where we see traffic, and hence, we wanted to bring ourselves closer to the actual end users, because the Internet in terms of protocols and such, it's rather chatty. So, speed of light only travels so fast through fiber and bringing us closer to our end users wherever they may be creates an exponentially better Internet experience for those, for those users. Obviously building a bigger, a better Internet at Cloudflare is something that we believe strongly in. Do you think that this kind of message is compatible with that sort of mission statement? Yeah, I mean, I think it isn't the most super obvious kind of revenue and sales move. And I can't speak to the super fancy like mega business decision behind this, but on the infrastructure and network strategy, we're super data driven on building a better Internet for all, which, you know, we do work with sales, obviously, to make sure we can support deals and help out direct customers and goals. But as a team, we're really focused on helping to build better Internet, rather than just helping to build better sales, which I know you're in sales so I know this doesn't exactly make your life easier, but sorry about that. Not a worry. And my, from my position, I, I'm of course really enamored with kind of the mission statement of Cloudflare overall. That's why I work here. And when we serve a better or a bigger global audience. It's something that, you know, I can feel really proud of, even if it doesn't tie to my necessary goals, specifically. You did remind us about a couple of things in your article, you kind of reminded us a little bit about what Cloudflare's pop and presence looks like. You mentioned kind of as a little bit about as an asterisk note in there about our China network. Do you mind just touching briefly on that. Yeah, thank you for pointing that out. So the new cities I discussed in the blog post are only kind of our normal global network, which doesn't include our really incredible China network, which adds a couple dozen. I think around 30 right now, new cities in mainland China for enterprise customers, which is made possible from our great domestic partner, JD Cloud and AI. And I think it's not just special because it's adding new dots on the map, though. Obviously, I love seeing as good coverage as possible over here, but also because we've incorporated a lot of unique routing and peering strategies to handle the fact that the Chinese Internet works a bit differently. But I think in total, there should be a plan for 150 new cities with JD Cloud by the end of 2023. So, lots of new pops that new dots that will be added to the map, but that weren't necessarily covered in my post. Sure, absolutely. And to tie along those lines of what you were mentioning earlier and bring it back to kind of the conversation about revenue versus, you know, building a better Internet. Can you tell me a little bit about why it's strategically and why mission wise, we consider building a better Internet to be important for us. Yeah, I mean, there's obviously like you, I'm here for the mission, if nothing else, like I really believe in building a better Internet, which means like actual Internet rather than just highest revenue regions. But I think it's a real win for all of our global customers as well. Just like quickly shout out to Zendesk who we've done a bunch of case studies with and has been great partner to ours, like that's a company that wants to span the entire globe, as well as possible, and not worry about, oh no, if I want to make like an expansion into XYZ region, they don't have to worry about the network being able to back them up, wherever that is, and they don't need to like have talks with us beforehand. They can just work on whatever strategic initiatives they want, which in turn, makes for happy customers, it makes for a stronger network. It's a real win win. And I think that's kind of tackles both ends of it we were helping to build a better Internet, but we're also helping to build like really happy customers were able to kind of do their strategic projects without worrying too much about what's happening on the back end with us. Right, because as we're learning the Internet is a very, it's utilized by everybody world over. We've got tools we've got applications and platforms that that use Cloudflare that use the global Internet that are available to people, no matter where they're located, some days. So it is in some ways, a pretty unifying force and it's it's it's lovely to be a part of that. Tell us a little bit about what this means though from a technical perspective. What, what, what makes this such a challenging proposition for Cloudflare, and just anyone that wants to take on this task. So, on a software in that kind of technical perspective, it's, it's pretty simple. As I said, the Internet, it's a bunch of chatty protocols. So, latency adds up very quickly. Like a couple milliseconds times the number of requests that go back and forth between your phone or smart toaster or whatever in the Internet adds up quickly to make a either a better or worse Internet experience on a whole. And there's literally only so much you can do with with all the cool smart routing stuff we we have the speed of light only goes so fast in fiber, we haven't been able to tackle that one yet. So what we want to do is put as many as many dots on the map as close to all of our users as possible. Sure. And when we're evaluating and I think about kind of. What I've experienced since I've joined their little over three years ago we weren't at 150 cities at that point. Now, over 200, it shows that Cloudflare is really positioned itself for expanding its presence and and bringing the Internet to a bigger and bigger audience every year. Are you able to talk a little bit about the growth or is that something that we're not able to share. No, that definitely is. In fact, I didn't want to make slides but I do have this chart of growth it's totally public it's it's just mapping out spending our blog posts, but actually this is the issue with using it as slides. But, yeah, when you signed on in 2018, we know we're just, you know, one for these range and now we're up to 200 with a very clear path to 250 again not including our, our China network. And I think that's been able to happen, based on a lot of strong partnerships and a lot of hard work, but the reality is just looking at the chart what you can see is it kind of comes in fits and starts at times, which just kind of reflects our status as a business and our growth. And also just how the Internet has shifted. For example, you know, we had some really incredible growth between 2016 and 2019. And it's been kind of a little bit calmer since then. And the reality of that is just, we've gotten really busy, and we've, we work in a very data driven format so we deploy where servers are needed and that can sometimes cause new cities to be kind of sidelined. So I think that what you'll see, just like in all of the previous times which I am so helpfully blocking out, is that we kind of go calm, and then increase a lot in a very short period of time. I think you're going to see that over the coming year so hopefully exponentially but you know I obviously can't say that. But it's from 150 to 200 road to 250 and beyond. We have that all nicely planned out. Sure. And then this kind of ties in a little bit to what you were mentioning, you know, in both your article and at the top of the column. When we are looking at kind of these areas of calm periods where we're not necessarily expanding geographically and the cities. We're taking on these initiatives like actually improving the overall infrastructure, in this case, the doubling of the CPU capacity for colo. Tell me a little bit about sort of like, are you able to talk a little bit about some of the considerations for how we're, how we, you know, make our system robust or when we are data driven society company what we're kind of tending to look at with regards to plans for either addressing one road or the other. Yeah. Um, so that works in a variety of ways because there are a variety of definitions for capacity. So, the chart that I just showed was of cities, which is one way of thinking about capacity. But behind the scenes, there's also computational capacity, which just means number of servers or power of servers computational power, not physical power required to make it happen and also network interconnection and network capacity, which I wish I had pre loaded my slide, but that's been a complete exponential growth, since, you know, since 2010, when we started in lieu of a slider able to tell us a little bit about what that might look like. Yeah, so I'm just a few years ago. And I'm restricted to talking about public facing numbers but we used to be in the like, oh, we have like a couple terabits like really huge number you don't think about how large a terabit is in terms of pipe. And now we're most of the way. Most way to 100, you know, it's a really I have Have a sticker. Actually, thank you. Yes journey to 100 terabits per second. And what the slide shows is literally just exponential growth between those smaller numbers and 100. It really, it looks like the, the dream startup chart, the hockey stick that everyone talks about it, which I'm really proud of. Absolutely. It's definitely sort of that exponential growth growth that seems to be the most desirable sort of chart, you can go after. Well, so this has been certainly enlightening with regards to kind of the general plans or some of the complications that you'd mentioned to are able to tell us a little bit about that. I know that 2020 posed a lot of interesting time challenges for folks the world over. To the extent that you can are able to speak on that. Yeah, so there's logistic Logistic complications, which, you know, there have been a ton of articles about there's the global Various electronics electronic component shortages, which fortunately we were able we kind of were ahead of that and still have a good steady access to our next generation servers. But logistically, suddenly, everything that took a week and cost x will take three months and cost y or customs, which normally takes any given country, a amount of time will take four times that that's actually to quickly. To quickly go off of that, that can be why we have potential future cities that are potential future for a bit longer in progress for a bit longer than what we'd like just because they're trapped in customs. So red tape is a problem. Physical servers being constructed at only so fast as a problem. And sometimes the networks that we work with, because we work with, you know, thousands of networks adjacent networks. They can move at different speeds because they have increases in their load on their networks, which means that we can, you know, it's kind of first come first serve sometimes in terms of the The service we get from them. It's a Quite literally a lot of moving parts. There's quite literally a lot of moving parts. Yes. Absolutely. I know that unrelated to chips, but my share I ordered from my apartment. I think probably took six weeks last year to arrive. Yeah, it's when you combine that with also just the factories and, you know, even if the factories all go 10% slower, which isn't realistic, you know, somewhere closer months. That that builds up the entire global supply chain has really been upended which I think it's actually a real Real testament to our logistics teams and our hardware teams with hardware engineering and the folks who work with our suppliers. To make that even vaguely feasible. So I know this is this is pretty big, but yes, a lot of challenges. No, I completely understand and and and to the extent that I mean you've been involved in this as well. And you've got a lot more sort of visibility into the issues and very wonderful to have everybody on the team able to kind of help us navigate what as otherwise, as you point out, a lot of challenges. As sort of like Cloudflare, you know, prepares for creating kind of the future state of the Internet. What are some of the things that folks can kind of keep in mind or should kind of keep in mind when it comes to building a network. Is there, is there a simple answer. Is there any sort of Things that we should be that a global citizen that uses the Internet should be keeping in mind as they're looking to upgrade their devices or as they're looking to think about things in the future. What, what, what does the Internet here have to kind of offer when it comes to building a global future state. There was a really good blog post on this topic during our impact week. A couple of weeks back, but I think for end users. The most important thing is picking one network, you get at the last mile, because the discrepancies in terms of quality of service and interconnection. fluctuates wildly, even with, you know, big networks that you would assume would be well connected. Sometimes they make some really funky routing decisions. So even if you're in, in, say, if you're in Austin, Texas, and you could connect to a pop in Austin, Texas, sometimes, you know, Chicago or something like that. So the blog post we had kind of covered places with the most competitive in the places with the most competition in terms of Internet providers actually were the ones who have the best Internet service in general, which means less congestion so on. So I think We obviously are someone who really tries to Promote Promote healthy competition that space and and, you know, work with the various government policymakers in that space. But I think What anyone as a global citizen can do is be mindful of their networks and try to, you know, where we can push for change and reform in regulation in terms of like competition and what is possible because, as I said, you see some places where I live in San Francisco. Some, some areas have one just one ISP provider and they have terrible service as a result, whereas, you know, a few blocks over. If you have a few options you see way, way better service at a lower price. Absolutely. I think we're both neighbors. So I count myself as somebody who's lucky to have kind of an abundance of options, but I can understand and having lived in an area where that's not always available to me. Typically, what that can usually mean for my choices, usually that I have none. Yeah, which is true for a lot of the world. Yeah, absolutely. So there's been. Can you remind us just a little bit. You're on the infrastructure team. Can you remind us a little bit on sort of just the cast of characters that are evolved, maybe in adjacent to your team, whether just either at the high level of the teams or who's involved to help, you know, support both yourself as well as Cloudflare as a company and that initiative, are you able to speak a little bit about Yeah. So, as I mentioned, I'm on the network strategy team under our infrastructure department, but because of the wide scope of our departmental mandate. There are a lot of, a lot of different pieces, just to go over a few key ones. I mentioned obviously our logistics team, we have the data center and infrastructure growth, who work with our suppliers, make sure that there is literally space and power and planning for that. We have data center engineering Cloudflare has zero full time employees on site at our various cities, all of these. So, it requires a lot of wrangling and a lot of really clever planning with all of our various sites to make that happen. So you have data center engineering, you have infrastructure growth, we have network strategy, we have infrastructure engineering, which is not only hardware engineering but working on things like the automation and such to, to help streamline the process of provisioning Undisclosed large number of servers in a given year. And of course our partners on like network engineering, which is an adjacent team that helps us not only helps us provision and make sure that the pipes are working correctly, but also things like smart routing decisions between the various Between various pops to make sure that user always get the best experience. Absolutely. Yeah, they kind of keep all of the plumbing working really well. Yes. And actually, since we only have a few minutes left, I'm going to give the obligatory shout out if I may. So, on, on my network strategy team, which is awesome and we have open jobs and everyone who's watching this who finds any of this interesting should apply. We have a team that works with our global partners who enable ton of these dots to come up on our map to delay and more remote places. There's a lot of Like the Internet is is a series of tubes, obviously, but also a series of relationships and a lot of different plans and interconnections and we have some really great folks working on that. And then finally our kind of commercial team, you know, we prefer to have everything on a cost neutral basis because the networks we work with are getting just as much out of it as we are, but sometimes you need to do commercials. We have an opening on that team as well. So, Absolutely. That's a very good shout. Please visit our career section Cloudflare.com slash careers. See all of our open roles there. We're always hiring. So please check us out. Or if you're happy with, you know, as I said, we have a ton of ISP partners. If you work at an ISP and want your users to have better access to the Internet. Please get in touch with us. We have something called the edge partnership program where, you know, we'll cover the servers and such will send you units to embed in your network and just flip of a switch your users are having a better Internet experience. So that's EPP at Cloudflare .com Shameless self promotion. I just had to sneak that in. Sorry. No, get the peering up. I support it. John, thank you so much for taking the time to speak with us today and to review your blog post article. We'll have a link to that here for you all to review as well. Thank you so much again and have a wonderful rest of your week. Thank you. And Don't forget. Thanks so much. Take care of john have a good one. You too. Bye bye. Bye bye.