Cloudflare Deployment in Guam

and I think we're live. Hello and welcome to our program about Cloudflare data center deployment in Guam, a country archipelago in the middle of the pacific ocean.

With me I have David Antunes, network hardware engineer based in Lisbon, Portugal that wrote the blog post we did about this and Tanner Ryan, network engineer based in Austin, Texas.

I'm João Tomé, storyteller and coming to you live through the Internet from Benfica, Portugal.

Hello David, hello Tanner. Hello João, hello Tanner.

Hello and welcome. Just to get things started, why a blog post about Guam?

Guam is a small country of 168,000 people in the pacific. Why this in particular?

Okay, I can try to answer that since I wrote the most of the blog. So yeah, it's interesting because of the geography of the place as I just told you.

So Cloudflare has data centers spread everywhere so we cannot write about them all and eventually some of them are more special than others, not because of their importance but because of exactly the location.

So Cloudflare is a global network but at the same time we are trying to be the most local network as well so when I think this represents pretty much that concept because it's so remote.

It's not probably the most remote one.

I think the most remote is Hawaii which is right like more or less in the middle of the pacific ocean which is huge but I would have to say that this is probably one of the top five if not higher than that because it's as we wrote it's so far it's like 2400 kilometers away from the from east asia it's from my head 6100 kilometers away from the west coast of the US and as explained latency is directly related to distance in this case the Guam residents do feel the latency because of their geography.

So yeah it sounds like a good idea okay let's write about this because it really shows how committed Cloudflare is to make a better Internet for everyone and including Guam obviously.

Of course all in terms of network and this many people don't know that use Internet you're connecting with a data center and Cloudflare is all about connecting you with the shorter route so you don't have to go to another continent to get your specific website and that helps with the performance of those websites of your experience in the Internet in a sense.

So in this case when there it's an island in the pacific and we had for example this situation in January where in Tonga there was a volcano and because of that submarine cables were damaged and the Internet went was a profound outage for weeks they didn't have access to the Internet so in this case an outage but the experience of the Internet in the pacific if you're in a remote island could be very different from what we have in Europe or the US or in countries that have a better or more close data center so this makes a difference for those who live there a big difference.

So if we try to make a comparison let's why a blog the first question why a blog about Guam so having a data center in Guam from scratch is not the same as having a new data center in let's say mid France or middle of Germany because in Germany or in France or other European countries and I'm saying European because that's where I am but we could say the same about the North America, South America, even Africa, Asia as well.

We already have a tens of data centers in there so like in Germany we have Munich, we have Berlin, we have Dusseldorf, we have Hamburg I believe some more some probably some some others that I'm not reminding about but in this case it's not the same if we build let's say tomorrow a new data center in let's say Stuttgart okay the residents already have nearby locations that provide them a better service Guam is not the same so the alternative they have is Tokyo, Seattle, probably Australia so yeah it's definitely not the same and it certainly is having a good impact positive impact for the residents in Guam.

Here is the map of the blog you wrote Guam is really in the that vast ocean that is the Pacific and again latency is all about latency here one of the things I writing a lot of blog posts writing a lot of things I've noticed even doing a lot of interviews is how important having a quick Internet is for example in Africa there's a big latency in some regions in Africa and that means that people use less the Internet they it's a worse experience and because of that they do less using the Internet so when you have a bad experience using the Internet you do different things you don't do the same so here the latency improving the latency with a close by data center is also relevant there again if you also highlight the submarine cables here yeah so let's see definitely it's physics there's not no rocket science about it it's the speed of light it is what it is so if you are far and it takes more time that's always and if we go back in time and if we think about what the Internet was like I don't know 10 15 years ago the speeds were not the same as they have today and therefore we could not do the same as we do today because the technologies didn't allow so in this case it's not only that the technology is there it's the distances that is there now as well or that isn't there as we speak regarding the submarine cables you mentioned Tonga so we wrote about it in this in this blog post so Tonga was a special case was more more subjected to failures we wrote about this in Guam because Guam is a kind of a hub in terms of someone comes so plenty of cables landing in this region and from there being diverted to different locations so we get the feeling that indeed this is not so subjected to failures as Tonga was because of that diversity of someone comes going through Guam so Guam has a kind of a strategic role in that matter because many of these cables are going through it yeah this is the submarine cables that in a sense are the the way the Internet works really and in the pacific there's it's all about those submarine cables like we saw in Tonga and here this image shows well the how the submarine cables work for a specific archipelago archipelago here so it's all about connections sorry go ahead even with having all this connectivity connecting Guam there's still presence for failure failures or even capacity issues so if there ends up being a cable fault on one of these submarine systems it could take it out for weeks at a time potentially months and that would be limited capacity so as Internet usage grows over time it's more essential that we have that local traffic delivery just because we can be reaching capacity constraints of these existing systems or if a failure happens you can really reduce your available capacity which makes sense I think that shows a little bit we can go now to the how a data center is implemented because we have a lot of experience there but how is the process how can we explain a little bit of that process in this case in in in an island in the archipelago in the pacific so at Cloudflare we have varying types of data centers depending on the geography it serves some data centers are designed to receive traffics from various regions as well as potentially other continents depending on the type of traffic flows and then we have more data centers that are designed to receive traffic from various regions that are designed to serve that immediate area so it ultimately depends on the type of scale for the example of the guam Cloudflare in itself still needs connectivity to the greater Internet so if you have a website that's hosted on Cloudflare we still need a way to contact your server in order to fetch the content from your server but once we have a copy cached locally on our data center in guam we could actually serve that directly to networks so for the example of guam we are connected at an Internet exchange called mari ix and this Internet exchange allows us to exchange traffic directly with the networks that are in guam and immediate surrounding area so instead of a network sending a request to a path that ends up in tokyo or to los angeles or san jose they can prefer the local routes coming from our guam data center and what that allows us to do is stay off those submarine cables and have as many requests serve locally as possible so whether you're using Cloudflare as a cdn or you're using our authoritative dns recursive dns ntp services we try to serve as much as that content locally as possible on the island and only if content can't be served locally or it's not cached well we then have to leave the region and go fetch for the cache it's a matter of distance because if it goes under the submarine cables in this case and it's a long way the the distance is big and of course the speed of light is very quick but it's not that big because it's the distance is big the Internet your Internet experience won't be as fast as possible in the sense yeah yeah pretty much and um so when we think about latency and we're talking about loading web pages it's not it's not just like a packet that you you ask once and you and you get web pages composed by different objects so latency is counting every time whenever you need to download to download all the all the objects that compose a web page so it gets a kind of a multiplication factor um so it really also of course depends on how the web page is built so some heavier than others but that all contributes to increasing or decreasing uh how long it takes to to download the web page the latency thing one of the things that is interesting that i'm not sure if we covered on that submarine cables map so guam is also serving uh even more remote islands uh the mariana islands and um and i believe the marshall islands yes so the they are sorry the marshall islands and the fact that it's on micronesia that's correct so they these are relying on guam so if you deploy on guam as we did you are also giving the chance to uh these even more remote uh residents on those small archipelagos to to better serve to have a better Internet ultimately um it's a an interesting topic and uh it seems it is complex the Internet is the network of networks it's a complex uh way of collaborating for sure in terms of the technology but the experience of people is very easy and this has a real impact in the everyone's experience and that part of the world so i think this is a good use case in a sense to to show a little bit of that uh you also measure the performance right the performance impact for the region yeah we did we did provide a few shots about latency uh just before going there just a remark about what on top of what then i've said so when you do the cash locally obviously tenor mentioned this we are taking outload from the submarine cables and that also represents a benefit because you are consuming less in terms of your transit providers and what eventually they can charge you from for the bandwidth that you are taking so that's another benefit from having this this type of deployment um regarding the performance yeah we were a lot of some some very basic shots just to to give an idea and not make it very complex but basically well so when we started serving one particular as which is the one we are seeing in the screen nine two four six so we have in this first chart clearly uh when it was being requested if you scroll down before it was being served by seattle um so that's up to the as to prefer routes coming from there or there so for some reason it was usually it was going to seattle which is not the the closest location we had before before one and then of course when you when a turn up was done it stopped being served by guam it's pretty clear on on this chart the most important one is the which shows us the what what is called the the basically the latency so eyeballs are the the users so the people like like us uh this is around three time for that same as and the well the difference is huge so we are talking about the median the percent percent to 50 uh decrease from 136 milliseconds so just nine milliseconds so this is huge 93 this is all about distance again and the worst case is like percent to 90 the p90 in that chart so it decreased from 188 to 97 so like 50 to 48 percent to be more and more exact which is also huge um and this is the type of benefits that we are that we are achieving that we are delivering actually to to users to customers by doing this closer deployments closer to where they are we have another shot below if you scroll down so we are looking to a busy hour so the the the metrics for the old busy hour in that same location for the same as and again we see a massive 92 reduction in latency so going from nine 146 milliseconds which is a lot to just 12 which is nothing um 146 milliseconds latencies it's not good basically it's the it's not it is what it is um but it doesn't seem it doesn't seem a lot in terms of seconds oh second to second it doesn't seem a lot but if you're using the Internet you feel like a real world incredible difference yeah it's uh it's not a good experience 146 because because again we are not just downloading a single object if it is okay click and get an object okay maybe not that bad but a web page is composed by different objects so this adds up on top on top on top and and this reminds you an interesting story some i guess maybe it was a long ago like 2006 or seven previous job i had we were discussing okay latency is that so important this was like i think 2007 and this was about the mobile network the technology that was new by then it was called hsupa 3g stuff like really old now we are already in 5g why i'm talking about this because during this discussion someone reminded on don't know who but it doesn't matter someone okay let's have a focus group and let's put some uh people regular people in a room with different laptops and let's configure those laptops with different settings so that latency provided by the 3g network can be higher or less and let's see what the people have to say and the impact was huge i don't i don't remember the figures which is long ago but i remember okay yeah this laptop is really cool what what is it different okay it was not only the the the throughput that they could could achieve it was also also about latency and some of the laptops have the same throughput than others but the latency was much better because they are using hspa and the others weren't and you can see that on the feedback we got from this focus group it was really cool yeah i remember that that story and by the way go ahead sorry and the difference was not near close to the one that we are having with this opponent so we are talking about going from 146 to 12 we're talking about i don't know but maybe 60 to 30 or 60 to 40 around that maybe so yeah we should do a video clip of people experimenting like the 146 milliseconds experience and do what you usually do on the Internet with this and with a more more short type of latency and just hey what do you think of the experience just receiving that feedback even if you're not a technical person you can really feel it it's it's a very bad experience definitely we we also showed the impact on Internet traffic so http requests in one so in a sense this reminds me of let me share the the blog post here this reminds me of something that john grant cumming our cto told me when we were discussing this blog post that was hey you should check http requests because usually when we put a data center there people will use more the Internet because it's a better experience so they will use more that that was the theory and we we saw that right yeah as as the graphic shows so pretty much there is a increasing trend on that that slope a positive slope and i guess as as we as we wrote as more as are being added because we haven't covered all the asses that participate in this particular Internet exchange it will get better definitely yeah so of course as you just said so if we don't have a good service we don't use it if we have a good service to use it more and more it's pretty much it and we have noticed this go ahead sorry uh we have noticed a trend too in these more remote regions that when we land data centers uh there could actually be more networks coming onto our network because of those local performance benefits so in the experience i've seen almost all of them have been uh free customers but if let's say you have developers or small local news websites that are within guam now that they know cloud infrastructure is present they can be backed behind the security performance reliability of our global network but still have that local service that they've always been looking for so there's been opportunities for networks to really speed up their website and it hasn't always been available so especially in these remote regions we do want to make them available exactly it's like the ecosystem if you improve one part of the ecosystem other parts will go there and will benefit from that improvement so and a lot of different stakeholders will improve because of one in this case data etc but of one deployment in a sense yeah yeah that's a good description yeah um and that's why we have also here the edge partnership program so for those isps interested in hosting Cloudflare cash uh they can uh get in touch with our edge partnership program page right exactly to do that to to benefit from and let me show our global network uh right yeah so if you are an Internet provider or you are a mobile telephone company or even a regional Internet provider that sells Internet to other networks you're either going to be directly connected to Cloudflare through an existing Internet exchange or a private network interconnect or one of these data centers or you're going to have to transit through one or more networks to get to us so with the Internet depending on where you're connected to uh the local connectivity infrastructure may not be there so for regions especially um within central africa there's not a lot of local infrastructure present uh for data centers and Internet connectivity which results in a lot of the traffic either going down to cities like johannesburg cape town durban or those requests will have to go up towards europe whether it's amsterdam milan frankfurt and so with the edge partner program we can help Internet providers start these infrastructure ecosystems by giving them the opportunity to actually plant network infrastructure down in their network not only to help improve the performance for their customers but to also help bootstrap a local connectivity ecosystem for networks so for the examples um georgetown and guyana comes to mind they're looking to start an Internet exchange to have more local Internet traffic delivery within the country and so Cloudflare getting us into the country was the first step into kind of kickstarting that Internet exchange and so oftentimes um to build an Internet exchange you need a reason for Internet providers to actually build out their network and connect there and so through the edge partner program that's one way that we can help kickstart these local infrastructure initiatives yeah it's really important and i think it uh it's a lot about how the Internet works and how it is built in terms of it remind me of wikipedia in a sense because we hope and we are trying that it's getting better more data centers more connections with different exchange providers or isps so that connections are making the Internet faster if you remind i don't know if you remind us the experience of using the Internet in late 90s was so different from what it is today and it's all about how the Internet is built it's improving and we're also helping making our part in that improvement but everyone has to be together to for those improvements to be reaching everyone in the sense right definitely so even today with um our network uh we are deployed in over 250 Internet exchanges meaning that if Internet networks want to exchange traffic directly through us and skip those intermediary networks they can do that and it's great for them because it does reduce the latency in a lot of senses but even then as you have to hop through various networks to get to us it introduces points of failure every little router or switch or fiber optic cable you run through it could be something else that breaks and so the more that we can stretch our network out into the last mile to have these networks connected to us we can help offer a better performance to these Internet providers who in exchange can really deliver the best possible Internet experience to their customers you explained they're peering at Internet exchanges for example but what about peering over pnis what does that mean what why is that relevant so Internet exchanges are a common switching platform you could think of it as a large switch that we would be connected into and various Internet providers can connect into and by forming a peering agreement we will enter into a agreement most of the time commercial free free of charge that we can exchange traffic so if there's a network that is on an Internet provider we can connect to that content over the switch likewise if there's content on our network that Internet provider can get our content and access our services directly over that switch depending on the scale of the network sometimes there could be a lot of traffic exchange between us and some of the larger networks that they may only have access to a small port on the Internet exchange and so by getting a pni it's as if you are taking a physical cable and plugging it directly into our network so you're bypassing the risk of the Internet exchange switch failing and you can have that direct one-to-one connection with our network makes sense in terms of what's next we also we are always increasing our network in several ways with data centers or with the call center offices and other type of things we we have in place what's next i know we we cannot announce new data centers per se but just explain a little bit of what we are doing next yeah so um the facilities that were present at for a pni and the facilities that were present at for Internet exchanges we cover a large portion of the Internet but the truth is not every network has the infrastructure to get back to those locations and so we still want to offer uh infrastructure opportunities to connect directly to these networks and that's why we have the edge partner program and so networks who may be in a secluded geography or lack the infrastructure to connect with us at one of these facilities can apply to the edge partner program and it's as if you are getting a mini Cloudflare data center directly in your name so we would send the necessary network gear and servers that would be installed within an Internet provider and whether you're you know in the middle of the pacific ocean or you're somewhere in a remote region in asia or europe you would still get local connectivity because we can serve and cache as many of those requests from within your network makes sense uh and we're almost out of time any final thoughts in terms of why this area is relevant and exciting both david yeah uh well well to be honest i'm when i wrote the blog i was have i had a different role than the one i have now so i'm not so involved as i was in deploying new and or expanding the existing data centers uh so it was exciting because yeah you really get the opportunity to to work with completely different people different cultures different ways of working and uh and you get the sense of mission because you are really okay it's of course it's called there is a company and but uh you you get to you get the sense that you are actually doing something good for for a lot of people not just for for the complete culture but also for the the residents and for the people that are using the Internet so it was quite exciting yeah and it still is the new rule is a bit different uh so i don't think it makes sense to talk about it uh just that the aim is to do it even better and better and more efficiently and yeah quite quite and working with tanner and working with the other teams because we all work very well together and uh with that sense of mission that okay it needs to be done and let's get together let's uh work around the challenges and yeah tenor your perspective there 30 seconds i agree with everything that david said uh just to add we've had networks who have come to us so we've deployed infrastructure inside and they said thank you for giving us a better Internet it was slow before it took forever they were frustrated their customers and their customers customers are frustrated so it feels like we're giving them a brand new experience yeah and that's you you see the impact there the feedback there for sure uh we're almost out of time let me just say anyone who wants to read the blog post you can visit on blog.Cloudflare.com thank you tanner thank you david see you thank you