Internet disruptions overview for Q2 2022

Hello. Welcome, I think we're live. Welcome to our Internet Disruptions Overview segment. I'm João Tomé. And with me is David Belson, our head of Data Insight, who wrote the blog post where we review several Internet disruptions that we saw in Q2. Hello, David. Yes, well. We, just for people to get a sense of what Q2 brought in terms of disruptions. There's some exam related disruptions, of course, preventing cheating on exams. There's fiber cable issues. There's a lot of different things going on, right? Yeah. The standard compliment, I don't think we saw anything too out of the ordinary in terms of clauses or drivers in the second quarter. That's was more on the key three already with the Rogers outage in Canada. But that we'll talk about that, though, later. Exactly. In terms of starting the way that Q2 started, in terms of disruptions, the first one were was related to Comcast. So in the US. In late April, there was a fiber cut in Florida that took out the Comcast service for subscribers across about 20 cities there. So pretty significant but localized impact. And that actually wound up not only impacting the local subscribers but through the radar data, including some of the graph we published in the post. We saw that the outage actually impacted the mix of mobile and desktop connectivity or the mix and mobile desktop clients that we saw across all of Comcast across their ASN during the outage, basically causing desktop traffic to drop slightly and mobile obviously seeing the corresponding increase as people apparently shifted their connectivity types. And then a slight mix, a slight shift in IPv4 and IPv6 connectivity. Comcast has historically been strong in supporting IPv6, but in general, we're seeing about a 40, or at least at the time we're seeing around 40% V6 adoption from their autonomous system with the outage. For some reason the IPv6 adoption went up and before adoption went down. So before adoption, sorry, IPv6 adoption went up from about 40% to about 50, a little more than 50% and v6, sorry, V4 adoption dropped from about 60 down to about 50. So it's interesting to see just such a localized impact. A localized outage had sort of a full, full impact across their full network. Exactly. So sometimes like when there's a disruption other than usual, things are disrupted also, like the mobile versus desktop here and also the IPv4 adoption. Right, because we see a drop in traffic and ultimately the traffic mix, the residual mix of the profile and mix of the residual traffic changes which can impact things like v4, v6 or mobile and desktop usage. Exactly. In this case, there were many subscribers that were affected. 2000, because it was about 20 communities within Florida. So certainly not the connectivity impact did not impact Comcast's. It was not across its full multi million user subscriber base. Of course it was focused on that area per se. If it was not, it would be different, of course, going on in terms of the other outage of fiber or cable issues that we saw. South Africa, going on now to Africa also had an issue here. Yes. So that was mid-May. On May 17th, the Telecom S.A, which is one of the big providers there, thankfully publicly admitted, tweeted something that said, Hey, we had a fiber cut and it happened at this time. And we were able to go into our graphs and take a look at their autonomous system and look at a handful of regions in South Africa. We saw the biggest impact, as you can see in the graph there in a region called Gauteng where we saw a brief slight drop in traffic starting at 0:00-6:00 UTC or 8:00 local time and that persisted for about 7 hours. So, when we look at, actually I take that back, so when we look at some of the other regions as well, you can also see the impact there. Thankfully, Telecom S.A. was forthcoming and saying, Hey, we had this issue which enabled us to understand why we were seeing the drop in traffic, but they didn't provide any additional information about what happened, what caused that fiber cut. Yes. Sometimes that happens in these cases and some operators sometimes are slow to say, Hey, something is happening. - Sometimes... - Unfortunately. Yeah. When they're proactive about it, it helps us. For those of us in the industry that watch these events, it helps us to provide context to our users and to our audience to say, okay, Here's a graph, here's here's what caused what you're seeing in the graph and here's what caused that fiber cut or that whatever. Of course. And Venezuela, going back now to South America, Venezuela also had a fiber cut related disruption, right? Yeah. And we see that kind of unfortunately, kind of frequently with CANTV. They seem to have some fairly frequent connectivity issues. And in this particular case, we believe it was a fiber cut or it was reported it was a fiber cut that impacted traffic for about almost 24 hours, which is interesting. I think a lot of times in Venezuela of the the disruptions we see, there are generally shorter length and less and less significant scope. This was almost a near complete outage in the state of Falcón. And again, we see these probably at least once a quarter, if not if not more frequently. Yeah, it's frequent. In this case, for example, we have also data that people can see. For example, here with CANTV, there's the ASM, there is where we show the trends that people can see not only yes, but even people can go and check this specific ASM in this case. Right, from the operator itself. Exactly. Going back to a big one, this was a big disruption for sure. It was a lot of countries that were impacted from Africa to Asia to Europe, and it involves submarine cables. Yes. So this one was really interesting because we saw essentially a near simultaneous impact to traffic across a number of countries. And when we see that, I think we might have even discussed it on the Q1 overview. But when we see simultaneous traffic impacts across a number of countries, one of the first things we start to do is say, okay, what do they have in common? And in many cases, depending on where they're located, obviously, but in many cases what we start, at least what I start to do is look at submarine cables and say, okay, here are the nine countries. Three, five, 12, whatever countries that were seeing traffic loss. Are they all connected to a similar submarine cable? So in this case, based on what we what we looked at, we were able to determine and what others in the industry published. It went up that there were two cables, the Africa, Asia, Europe One, or AAE-1, and the SMW-5. I think it stands for something. I can't remember what cables, both of them sort of cuts, which is ultimately what drove the impact that we saw in the in the traffic graphs. The interesting thing is that usually the submarine cable cuts occur undersea, given where the cables are located. In this particular case, though, the two issues, the two cables had issues on land. So both transit and we talked about this here, both transit Egypt as a way sort of across that part of the world. And apparently the issue occurred when they were over land. That also obviously led to the traffic returning, in essence, in just a few hours. So it was a big disruption in terms of time. Exactly. So in many cases, when you see a submarine cable cut and we saw this, you see this often in the South Pacific submarine cable issue. We saw with Tonga, for example. Tonga. Exactly. It's like, okay, first we got to figure out where the hell the cut is and then we've got to get a boat out there and then we've got to reel it up and splice it and drop it back down. It's like you're talking often weeks, days at best, usually more like weeks or months. And then of course, you know, it depends on the weather and all those issues. In this case, I think they're able to basically say, okay, we know that it happened in you know, this building for whatever reason. And they were able to address the issue within a matter of hours. So that was one of things it pointed at. Pointed at being not subsea. And we can see here our charts that show the disruption per country. For example, Ethiopia, Somalia, Tanzania, Pakistan, Mozambique, Kenya, India. A lot of countries were disrupted, although it wasn't a complete outage, there. Wasn't a complete outage in many cases. These country, the main providers in these countries will have redundancy. So they'll have either other submarine cables they can they can travel over or they have other terrestrial paths that they can take to the Internet. It was interesting, I believe, when I was looking at this, I believe it was Ethiopia isn't actually connected directly to that cable, but when I looked at their connectivity, they were getting connectivity through, I think one or more providers that are in countries that are connected to the impact of cables. So that may be one of the reasons that Ethiopia saw the impact of traffic. That shows us the Internet connectivity between countries, between sometimes even cables sometimes doesn't seem direct, direct impact in the country, but there's a relation there, so. Absolutely, yep. And then on Canada, not the Rogers one, but a funny one. Yeah, this is right. Yes. So we normally see the enormously backhoes taking a terrestrial fiber. In this case, it was a little more of a chain reaction where according to a published report or a number of published reports, a beaver, as they do, gnawed their way through a tree, that caused the tree to fall. And of course, it took out power lines, as well as a fiber optic cable belonging to Telus, Yes, that was the offending critter or probably not that one specifically. But and because the damage to the fiber optic cable ultimately wound up impacting a number of Telus customers, including Citywest, which is a local utility company. So we can see within the graph here that traffic for Citywest subscribers locally was out between 18:00 UTC on June 7th through just after 03:00, 03:00 on June 8th. So about nine, little more than 9 hours. But with a very an impact, very constricted to a specific city in this case. Yes, but the topic at hand was very Canada, let's say, like that. Yes. - Yes, it was. - In Portugal, that wouldn't happen with a beaver at least. And let's move on to preventing cheating on exams. This is a practice — very interesting I think, at least for me, thinking that a country or a government can think of taking the Internet down to when there's a reason for people not to cheat. This is an interesting concept. Yeah. So this has started about six or seven years ago in the Middle East. And, generally, what's been happening is that they will often take the Internet down nationwide as the tests are being distributed, I guess, to prevent somebody from getting hold of an early copy and figuring out what the answer is and communicating them out through social media channels or whatever. And then oftentimes I think what we see is, is sometimes the governments will either leave the Internet down through the exam period as well, or like the sort of the full country exam comes back up. And then mobile connectivity often will go down during the actual time of the exam. So different countries have different mechanisms for implementing it. And again, we even see that like in Syria where we saw these four outages throughout the end of May and June, those were aligned with the dates of the tests. But one of the interesting things with Syria is that their implementation of the Internet outage is it's not — what do we call it — it's more asymmetric in nature. So requests can get out, but responses can't get back in. So one of the artifacts of that ultimately is that we see. DNS requests from the country spiking as the Internet is shut down there because these are now, say, social media clients or other sorts of applications that are trying to resolve some hostname. You're not getting response back. So they try again and they just sort of get a little more aggressive about retrying and trying. Yeah. So it's interesting. We haven't seen that really in other countries. I found in terms of a chart, this is a very interesting chart for me because most people don't realize, but this drop down, very specific drop down, means that the people in that country don't have access to the Internet, or at least most people for sure. Correct. And that's in terms of a chart, the impact of looking at the chart, that's a real world impact and kind of incredible as an experience. Your reaction there speaks to just how ingrained the Internet has become in our lives. I think, thankfully, in Portugal, that concern of that happening is probably significantly lower. In Sudan, we saw daily shutdowns for across 10 of 11 days. These were not complete shutdowns. So they occurred nationwide, but clearly some amount of connectivity was left in the country. We often see that where government agencies or maybe things like hospitals or banks or whatnot are often left with some measure of connectivity. So it doesn't drop to zero. It depends on the country. For example, in Portugal, that would have a disruption in the whole country because there's a lot of the economy is based on services. In a country where the economy is not based on services, they can do that without any bigger costs. For example, you mentioned Algeria. They did it differently because of that. They saw it has an enormous cost to the economy. Right. So I think hopefully Algeria took a slightly wiser approach. I mean, the best approach I think arguably is not doing this at all. But in Algeria's case, we believe that they were doing a little more of a content blocking approach. So restricting access to certain sites or certain types of sites instead of doing a complete nationwide Internet shutdown. Make sense and we don't have it here, but it's interesting, also, if someone wants to mention to us, is how effective this is. I'm curious about that. But we don't have the... Reports that I, yeah, there hasn't been a lot of... I don't think there's a lot of follow up research on that. Like, is it? Yeah, I don't know that the people really have determined that it's a truly effective mechanism, but the government seem to think it is. So they keep doing it. I think the final one to touch on here briefly is Kurdistan or the Kurdistan region of Iraq. They started a four-week period of twice weekly Internet shutdowns across three governorates, three areas in the Kurdistan region. You can see here basically that traffic drops to near zero across these three regions on Monday mornings, on Thursday mornings. And this continued, it started out at the end of end of June and continued into the first three weeks of July. Exactly. So I think we generally start to see these more in the summertime than during the year exams. In most of the north atmosphere, countries are in the summertime, or at least the main ones, the national ones probably. So yeah, and these are less like, hey, I'm taking a social studies test or a math test and more like these are school placement exams. So if you don't, if you don't do well on these exams or don't pass them, they have the ultimate consequences for effectively the rest of your life, or pretty significant. Which is obviously the driver for cheating. There's also government guided shut downs, not only preventing cheating exams, but there's another type of control of communication. We've seen this even in previous wars, around elections, rallies, protests, and also in Turkmenistan in this case. Yeah. So Turkmenistan is known to be sort of a instable or instability within the government. So in this particular case, there was criticism over a recent presidential election and the government, I guess, chose to direct the shutdown of Internet connectivity there for over a day and a half. So nearly 40 hours, which we could see, which was obviously very obvious to us within the country level graph as well as well as within graphs. For two of the major telecom providers, there were traffic effectively drop to zero. And also in Pakistan, right? Yeah. Pakistan is different. Disruption was less significant, at least in a country level. We understand that was related to protests that were led by the country's former prime minister. It was pretty limited in scope. It was pretty brief, only 2 hours. And if you look at the national graph here, you can see it's almost unnoticeable. But when we look at graphs for Lahore and Karachi, two cities within Pakistan, you can see it a little bit more significantly. So there's the right side of the graph there. There's the dip. Yeah. It's a bigger dip for. Those and then a bigger dip where you look at some specific network providers there as well. And then closing the quarter, we had the blackout communications and also Internet shut down in Sudan. Also protests in this case, it was on the news, those protests. And then Internet went down actually before it was on the news, the Internet went down. Right. And this is, of course, Sudan was also going through the exam related shutdowns over the course of the quarter. So they really were challenged with connectivity. So we saw in Sudan, we saw a I'm trying to remember it was on June 30th. So we saw a near complete outage and then traffic returned for a brief period and then it went back down again. And I know how we ended the quarter on that. I'm trying to remember when it came back up, we can see across a number of the ISPs there the outage and the brief return of traffic and then another outage. My recollection is that it ended the disruption ended pretty shortly thereafter. Yeah, I think so too. But actually, we can check radar live. Probably 30 days. We'll be able to check the 30 day view. Right, right, right. Exactly. But moving on, there was also more infrastructure types of days. It was on July 1st. July 1st. Okay. That's what I thought. I thought it was a pretty quick resolution, just not it was after publication time or after submission time. But yeah, in Puerto Rico... Puerto Rico has a notoriously unstable power grid. So this particular case, due to a fire at one of the generation plants that resulted in widespread power outage across the country, or across the territory, and ultimately no power impacts end user Internet connectivity. So that led to a multi day drop in traffic that we observed within the region. So starting on April 7th and then traffic had largely returned by April 10th. And the local energy, local power company had said that by April 10th they and restored power to nearly all the customers so that we can we can almost track the restoration of electrical service by the return of traffic from Puerto Rico. That's interesting. And even growing, it starts to grow probably when the electric power starts to be more effective, in a sense, be more restored. And their power grid is so challenged there that this is probably at least, at least once a quarter we see a power outage related issue, Internet disruption in Puerto Rico. Makes sense. We also had in this case, Swisscom had a problem related to the same topic in a sense, infrastructure. Yeah, well, they said that they were doing maintenance work. So it wasn't thankfully, again, at least they were they were upfront and proactive saying, Hey, you know, we did some maintenance work and that caused a disruption in services. They didn't provide any additional details, unfortunately, but the work, the maintenance work they were doing ultimately did, ultimately it was observable within the traffic graphs. Makes sense. And then Iran, it was a different topic in this case. Yeah. So Iran, this is almost is this really is effectively government directed. But in response to mass protests around shortages of bread and water, they cut off connectivity in the Khuzestan Province. And we could look at a number of the network providers there to see the impact of this action. So we saw this drop in traffic that started mid-morning on May 6th. And then they also did something a few days later on a round cell also within Khuzestan. And then there were some near complete outages in May that we observed sort of a national level, but more specifically on a particular set of providers. So we don't know... So the issue here sorry, the issue with the multiple provider outage was that most of them shared or all of them shared telecom. So they'll share a comment upstream provider that had an outage. Unfortunately, we were never able to determine what caused that outage because the telecom, the upstream provider had the outage. A number of the networks that were downstream of it also lost connectivity. For sure. We don't have a lot of time. We also have had some more news about Internet traffic in Ukraine. In this case, even recently, we had a follow up on this related to Kherson. Right. Right. So actually our — so yeah, so during the quarter — so during May, one of the things we saw was an outage in Kherson. And then what we also determined at that time was that when Internet connectivity came back in Kherson, Kherson Telecom, which was obviously one of the main providers, it was no longer using Ukrainian upstream. They were now using Miranda, which is a Russian telecom as their upstream. So a lot of the traffic from Kherson telecoms inside was now flowing through Miranda and then further upstream through Rostelecom, which is another Russian provider, not only less than a few days that shift in connectivity, but then we observed it happening again at the end of May. And I believe that the connectivity now for telecom is still to this day. So now at the end of July, two months later, it's still routing through Miranda. I checked yesterday. Exactly. So no difference there. This was the course we did at that time. Yet we covered it. And we can see the colo. So the data center was going through the Moscow data center in that period. We'll do the data center selection based on where we see the traffic coming from and because it shifted. We said, okay, observing from Moscow made more sense and serving it from elsewhere. For sure. We have 2 minutes and not in the quarter, but in this case, actually, let's just also show the this tweet we did related to Kherson. So there was a disruption also this week, so suddenly yesterday out of Ukraine. So it continues with disruptions there, right. In this case, a big one, an outage. But we also had the, not in the quarter, the Rogers outage in Canada. That was a big it had a big impact in millions of people in Canada. Yep. We'll definitely cover that in next quarter's review. But that was the end of the first week of July, apparently due to some routing issues, some internal routing issues. Rogers Autonomous System, ASA-12, essentially fell off the Internet, there was still a little bit of traffic getting through, I believe, but they were down for, if I remember correctly, something in the order of 17 or 18 hours. It was a really, really surprisingly long Internet disruption and. It was 7 hours, 17 hours with outage, and then it start to go back. But it was like 22 hours when traffic was at least at half. So it was a day of impact for sure. Yeah. So definitely a big issue there. And what it did was laid bare, just the reliance on Rogers in Canada and just how many systems were unavailable and how many things just didn't work. Banking and purchase, ecommerce and whatnot due to the outage. We're ending. So it's almost done. Let's just say to people that they can reach us. Reach data, our data on radar, radar dot cloudflare dot com. And of course, follow us on our Twitter account. So you should have Twitter. The Twitter account is at Cloudflare radar and we'll be back in a few months with the third quarter summary.