Cloudflare Radar Bulletin: 2022 Q4 Internet Disruptions

And I think we're live. Hello, everyone. I'm João Tomé and welcome to our Internet Disruption Overview for the fourth quarter of 2022. With me, I have, as usual for these Internet Disruptions Overviews, David Belson, our Head of Data Insights. Hello, David. Good morning. Good morning. Good afternoon for me and good evening for those who could be watching on demand. We had, it was not, the last quarter of the year was not a very busy one in terms of disruptions, right? Right. Yeah. Comparatively, it was, it was, we seem to track fewer disruptions than we saw in the prior three quarters. Exactly. And the blog post you wrote that was published last week, essentially tries to highlight all of the different types of shutdowns, outages, disruptions that can happen and did do happen all over the world. How should we proceed in terms of showing them off? Should we start the way the blog post is set up? Yeah, I think we can follow the post. I think we, you know, I've organized the post in terms of the different causes, I guess, of the disruptions. You know, there was nothing sort of new and unique there. I think a lot of the old favorites, as it were, in terms of why Internet traffic dropped in various countries. But yeah, I mean, I think that the first one we can start with, obviously, is the government directed shutdowns. These are, unfortunately, you know, still a favorite tool of the more authoritarian governments. You know, in Cuba, for instance, they've seen shutdowns in the past, generally related to local demonstrations and local protests. And in September, there was, after Hurricane Ian hit the island, you know, the authorities there obviously scrambled to do their best to restore power and to restore Internet service. But, you know, some of the citizens felt that things weren't coming back to normal fast enough. So they organized protests around that. And in response to the protests, unfortunately, the government chose to shut down the Internet for several hours. So there was a shutdown that we saw that lasted, I think, for about eight hours at the end of September. We covered last quarter's report. And then there was another one that happened or a similar shutdown that happened on October 1st. So unfortunately, Cuba is no stranger to Internet shutdowns. This was just another in the long line of such events. Exactly. And in this case was a few hours in terms of time that the disruption happened over overnight, mostly? It was overnight-ish. I mean, so it was interesting that it happened between 1800 or 7pm local time on September 30th and just before three o 'clock in the morning local time, October 1st. So it, you know, a lot of times we've seen Internet curfews, which we'll cover in a bit, that generally will last more sort of overnight. You know, they'll go from like 7pm to say, you know, five or six a.m. So the timing of this one was sort of interesting. But yeah, it wasn't really a curfew as we've seen in other places. Yeah. And next we have Sudan with the October 25th first anniversary of a coup, right? Yes. Yeah. So again, Sudan is another government that has a multi-year history of shutting down the Internet around civil unrest and protests and whatnot. As you mentioned, this was the first anniversary of a coup that derailed ultimately the country's transition to civilian rule with military rule. And in this case, the government instituted a shutdown on October 25th. And that happened basically during the workday. So between about 9.45 local time to about 5 .50, 5.40 in the afternoon local time. So, you know, odd that it was during the workday, but I guess if that's when, you know, the protests were happening, then I guess it makes, the timing makes more sense. And, you know, as we can see from the graph, it was, you know, almost a near complete loss of traffic or near complete loss of connectivity. So the same thing in terms of loss of connectivity in Cuba. So a major disruption in terms of... Yeah, absolutely. I think the difference here is that, you know, not only would this have prevented communication, you know, around and about the protests, but obviously it was going to disrupt any sort of business or government related connectivity as well. So, you know, maybe that sometimes we do see a little bit of connectivity speed by, and I think that's what we see here, that it's not, you know, completely flat to zero. So maybe the case that there were some limited networks that were left available that were, you know, potentially government related or business related, you know, maybe the banks or the government offices or whatnot might have still had a little bit of connectivity left. Exactly. And next we have Iran, of course, that was also covered in the last quarter's blog post extensively because of the protests there since September. This is the third quarter international disruptions blog post. But another set of disruptions there happened, right? Yeah. So this was, as we were talking about earlier with the curfews. So there were, you know, in relation to the nationwide protests surrounding the death of Mossamini, the government there had started implementing these Internet curfews. So basically shutting down Internet connectivity on the three major mobile network providers effectively overnight. So I think, you know, connectivity was on during the day. People could do work, school, whatnot. But then when the protests flared up overnight, that was when they shut down the connectivity to prevent communication about the protests and prevent sharing of video, audio, text information with the outside world. So we saw those continue into early October and then a couple of other similar disruptions during mid -October. And then later in October, we saw another shutdown in Sanandaj. So that one was about 22 hours. And then in December, another one in, I guess also in Sanandaj that was on and off about a week. Or it was, sorry, it was shut down for about a week. Exactly. So, yeah. So in a sense, these are governments that are doing these shutdowns in order to try to Internet connectivity, not to be able to, Internet to be used in this type of protest and this type of situation, right? It's a bit different. Yeah. So, I mean, it's similar to the other ones we saw, you know, in Sudan and Cuba. Ultimately, I think governments have recognized, hey, you know, this is a way of preventing communication, preventing people from organizing, you know, protests and communicating while they're there. But I think while, you know, maybe even more importantly, I think they're seeing it as a mechanism for preventing communication with the outside world of, you know, what's going on. So, you know, is the military involved? Is there, you know, are there abuses being committed? You know, and, you know, obviously those are the videos that, you know, audio and whatnot that the government doesn't want getting out. So I think that's why they do this. You know, really interestingly, after the, you know, the disruptions in Iran that were taking place over Q4, there was some publications where there were a number of publications that looked at or, you know, published information on the economic impacts of these shutdowns to both business and government. So, you know, one case, the mobile operator, Rytel, sent a letter to the government basically saying, hey, you know, these shutdowns have decreased our data traffic by about 50%. You know, it's obviously that's causing them to lose money, you know, because they're not getting paid for that bandwidth usage. And that basically, you know, it continues to lead to bankruptcy. So, you know, they don't have business. Right. There's a mobile provider there saying, hey, you know, we can get paid by the bit. And if you won't let us serve bits, we can't get paid. That's going to be a problem. There were other, there's another local newspaper that provided a tangible view of the impact where they, I didn't do the conversions, but they, you know, they, in terms of the daily loss of revenue, but they said, you know, more than 41% of companies have lost 25 to 50% of their income during this period. And about 47% have more than, have had more than a 50% reduction in sales. So, you know, it's really pretty significant, you know, in terms of the real economic damage that these shutdowns ultimately cause. Exactly. And, but there were more focused on mobile networks, right? Which is a specific situation there. So mostly when people are outside their homes, in that case. Yeah. I think around, I believe is, I mean, they have good connectivity in terms of multiple providers, but I think they are also probably a very mobile centric society. So that's, you know, by coming off the mobile connectivity, you're really doing damage to people's ability to, you know, interact on a day-to-day basis. Of course. Makes sense. And next we have a different type of disruption, power outages. And with, there were a lot, but let's start with the Bangladesh one here. So in this case, in Bangladesh from October 4th, there was a grid failure. So apparently the government or part of the government had said to the power providers, Hey, you know, you need to distribute the load across multiple facilities or whatever it was. Those instructions were not followed. And that ultimately caused a part of the grid to fail within the country. So you can see in the graph here that caused a pretty significant drop in traffic. And that, that disruption lasted about seven hours. You know, this is a little bit different, not a little bit different, significantly different from the shutdowns. And I think, you know, in this, in the shutdowns case, the government is saying to the providers, Hey, you know, basically stop routing debts. And in the case of power outages, infrequently, this is more a function of Cloudflare not seeing traffic from the end users. So the end users, you know, they can't get online, you know, their computers, their phones or whatever don't have power. And then their local connection points, you know, the wifi routers or whatnot, or sometimes even the cell phone towers will often lose power as well. So it's, it's really, you know, it's not that the bits are getting routed. It's just that the systems, you know, that are generating the bits routing the bits that don't have power and ultimately are offline. Of course. So the Internet there doesn't work at least apart. So there's a drop. It's not also like sometimes not completely is partial. Right. You'll see, you'll see some bits still continuing to flow because it's not a complete nationwide power outage. Oftentimes the outages are more, more localized, more regional. And Pakistan is another situation here. Yeah. So, so actually in this case is another good example of the sort of more regionalized, more localized issues where in this, in the case, this was, this occurred in Sindh, Punjab and Balochistan, or Balochistan, I'm not, I'll pronounce that. But in this case, there was a fault in the transmission system across the power grid there. And this was reportedly due to faulty equipment and substandard maintenance. So you know, again, we're seeing sort of activity or lack of activity by the people, you know, that ultimately results in power outages that ultimately result in a loss of connectivity. And in this particular case, we saw a 15 hour outage that started about 9.30 local time and traffic had dropped about 30% lower than it was at the same time the previous week. So very clear on the graph there, you can see where it should have been on the orange line or where it normally would have been and where it was represented in the green line. And 30% is already a lot. So from the usual levels, those will have a real world impact for those in that regions. Yeah, absolutely. Yeah. If you lose power, you lose connectivity, then you know, you're out of luck. And then we have Kenya. So the explanation here was that it was a system disturbance. So various parts of the country had lost connectivity. You can see here on the left side of the graph, it was not a, you know, not a significant outage. It fell slightly below where it was the previous week, or excuse me, the previous day. But this was about a, almost a six hour outage. So, but more limited impact in this case. Exactly. Also power related in this case. Yes. And another example, go ahead, sir. No, no. I say, and then, yeah, another, another one here was in the US. Another very localized issue was in Moore County, North Carolina, where this was being termed domestic terrorism, and where electrical substations were targeted by gunfire. So you take out the equipment in the electrical substations, and then obviously it takes out power to the local areas. So in particular case, there were, we trapped a loss of connectivity within two communities within Moore County. And you can see there was sort of the, the connectivity, the loss of connectivity is fairly clear, where the graph drops, you know, it's basically near zero on Sunday, December 4th. And then in Pinehurst, it sort of stays fairly low for a few days, and then comes back on Thursday. It looks like it's kind of back for good there. And then in West End and the Green Line, that stays low for a couple of days. And then Monday comes back to life. So you see the peaks there on Monday, Tuesday, and then it dropped again after Wednesday. So not clear if they had to take the, you know, facilities down again for maintenance or what happened there. But clearly it took a little while longer for it to fully resolve. And it shows us that it's not only like in Middle East or in areas where there's a war going on, even in the U.S. or in other countries, energy problems happen, and those have an impact on the Internet for sure. Yeah, absolutely. And next we have Ukraine with the war. Ukraine we've been covering since Q1, you know, almost a year now. I think, you know, earlier in the year, I think we've talked about the past, a lot of the outages were due to the local fighting and to just a lot of, you know, there's some traffic rerouting and things like that. But what we saw in the fourth quarter is a lot more attacks on electrical facilities and some stations, power stations and whatnot. So obviously you're targeting, the Russians are targeting the power facilities in the various cities. And that's causing a lot, that damage is causing a loss of connectivity. So like on October 20th, we saw power stations in Kyiv being targeted, resulting in power outages that lasted several days. Another one in November, we looked at, it was more significant, we looked at it at a country level and saw disruption there that lasted almost a day and a half. And then again, on December 16th, you know, a significant enough attack that caused about a 13% drop in country level Internet traffic. So looking at traffic across all of Ukraine. And then, you know, but if we looked at it at a network level, we were seeing more significant impact. So TrialM saw a 70% drop in traffic related to the attack. Kyivstar saw a 40% drop related to the attack. So again, you know, obviously you have to have a network impact, even if we're looking at it at a country level, you're going to see those impacts at a network level as well. So again, these attacks by the Russian military on the power facilities in Ukraine are ultimately having an impact on connectivity and Internet traffic in the country. Exactly. And one of the things, first, it started mostly in October, as you were saying, especially October the 10th, I think that was the day when targeted attacks to an energy infrastructure started in a more coordinated way and in a lot of different energy infrastructures. And one of the things I've been surprised, first, is these types of attacks have been recurrent. So we've been seeing different types of drops in traffic in all of the countries, some of the regions. Because of that, even last week, we did a tweet about Odessa, for example, which had a problem also for a few hours. So it has been recurrent even already in January 2023. But in a sense, I've been surprised of how resilient the energy infrastructure and the Internet with that has been in Ukraine. We know now that they are doing in Ukraine programmed shutdowns of energy, of electricity, the night to save energy. So in a way, using some of those methods, they have been resilient, although in some periods, there's a real lack of electricity or the Internet or both. So it has been interesting. And next, we have cable cuts. Cable cuts, yes. So I think we saw one notable one in the fourth quarter in the Shetland Islands. So this was actually sort of interesting because of the timing. So this happened on October 19th. And just after the damage to the Nord Stream natural gas pipelines, I can't remember the name. And so there was some concern about the damage to this cable being sabotaged. But I think after further investigation, authorities ultimately stated that it was due to errant fishing vessels. So somebody dragging an anchor or somebody doing something where they shouldn't have been. Which is common, right? It happens. Unfortunately. In this case, part of the challenge was that the Shetland Islands, as you can see in the map, are off to the north of Scotland. So the cable is, I think, probably their primary mechanism for Internet connectivity. So connecting back to the Scottish mainland. So in this particular case, we saw traffic drop late in the day, 11 p.m. local time, on the 19th. And a little more than half a day later, so the afternoon of October 20th, it came back online. So clearly, they were able to identify where the cut happened and get a ship out there to repair it and get the repair done quickly. And that may be the case because it's probably spanning such a short distance. So a lot of times the challenge is you've got a sub-oceanic cable that goes across the Atlantic and the Pacific or some much longer distance. So not only do you can get a ship out there, you need to figure out where the cut is and haul it up. So there's a lot of potential time that can be taken up with repair. And this one was fairly quick. Exactly. And like you were saying, it's on the news mostly because of the situation. The proximity to the Nord Stream. Yep. Exactly. And the war in Ukraine. Actually, there's more news on better patrols on that area in terms of warships trying to protect. There was a lot of, all of a sudden in the news, there was sort of a lot of concern about the safety of submarine cables. And it was one of the, I mean, because they carry, I think the, you know, last I saw was about 95% of international or intercontinental Internet traffic. So obviously there's some significantly well-placed concern about it. Of course. And next we have natural disasters. Yeah. So earthquakes can frequently cause damage. So not only do they cause infrastructure damage, but they also cause power outages. And, you know, so it depends on where they are, you know, in proximity to Internet infrastructure. So in this particular case on the Solomon Islands, there was a magnitude 7.0 earthquake there on November 22nd. And what we saw was an 11 hour disruption. So in this particular case, they didn't lose all of their connectivity. Just it dropped, you know, about probably 25, 30% compared to the prior period. Exactly. And also technical problems are one of the different types of things, right? Yeah. So technical problems is sort of that catch -all of like, Hey, there was a problem, but we're not giving you specifics about what it was. I think the fact actually, in some cases they do give specifics, but it's sort of a, not a very specific, like, Hey, there was a fiber cod or whatever. So in Kyrgyzstan, on October 24th, there was a three hour outage, which we obviously observed in our traffic data. The country's ministry of digital development said the issue was caused by an accident on one of the main lines that supplied the Internet. So it does sound like a fiber cod of some sort, but they didn't really provide any additional details on what it was. Exactly. That's also common sometimes. In Australia, it was related to one specific ISP, Internet provider, Aussie Broadband. Yep. So Aussie Broadband on October 27th. And what they said was that a config change was made and got pushed out through automation. And that ultimately wound up, I think, screwing up connectivity for users that were downstream of it. So they had to have the users there and like reset their routers to get new configurations. And it can be kind of messy. This was, I think, primarily, not focused, but it primarily impacted customers in Victoria and New South Wales. We saw impacts in both regions, in both states. Exactly. Another example is Haiti. So in another part of the world, in this case. Yep. In Haiti, down in the Caribbean, this was a longer outage. So it was about 14 and a half hours. We saw on November 9th. And again, this was one of those where it might be fiber cut. They said due to intermittent outage on one of our international circuits, our network is having problems. And when we looked at submarine cable map, we see that there are two submarine cables there. Bahamas Domestic Submarine Network and then FiberLink. But there was no indication of which one might have been causing problems or which one might have been having issues. So not a lot of information. And then the official unknown. So no information at all. Right. This always frustrates me where you can observe an outage. And unfortunately, even with social media these days and whatnot, providers are not often very forthcoming about what happened. A lot of times they don't even acknowledge the fact that there's a problem. So that's unfortunate. But in this particular case, Wide Open West, which is an Internet service provider in the U.S., had some sort of issue where customers across multiple states were impacted. It wasn't even like it was close by. So in this case, we're looking at Alabama, which is down south, and Michigan, which is sort of the northern center of the country. And both of them saw traffic dropping for about an hour on November 15th. And then Cuba. Yeah, and then Cuba. So again, like we talked about earlier, they had the shutdown due to protests early in the quarter. And they've seen fiber cuts in the past. They've seen power outages in the past. In this case, they saw a seven -hour disruption on November 26th, dropping to as much as 75% below the previous levels. You can see there it wasn't a complete outage, but it was pretty significant. But no excuse was ever given. Exactly. And also SpaceX Starlink service. Yeah, SpaceX is interesting because one of the things with SpaceX, because it has sort of this global footprint, in contrast to a lot of the other providers that we've discussed, the impacts of a SpaceX outage can be felt globally, ultimately. And it can be pretty significant, especially as more and more geographies come to make use of their service. So this is at the end of November. This was really only about a half-hour, 45-minute outage. But we saw traffic drop to at least near zero very briefly. They didn't acknowledge the outage, at least at the time, nor did they provide any specific reasons for getting something to happen. So it'll be interesting to continue to watch SpaceX going forward, especially as they grow their customer base to see if these outages are more or less frequent. Exactly. And sometimes those happen and also have an impact. So it wasn't a busy, very busy quarter, but a lot of different types of disruptions. Yeah. Yeah, absolutely. And again, we summarized at the end of the blog post, I think one of the things we saw with Ukraine was that connectivity has become a casualty of war there. The attacks against them are becoming weaponized, essentially. So either we can impact the country by targeting the connectivity, targeting the power. And then there's the government shutdowns, obviously, that we saw in relation to the protests, the natural disasters like the volcanic eruption in Tonga earlier in the year, and then the one we just mentioned. It was a year ago, actually, yeah. Yeah, it was. Yeah, it was one of the first things we covered right after I joined. And then submarine cable issues, obviously. But yeah, we've seen a lot of stuff over the course of the year. We summarize those every quarter in the blog posts on blog.Cloudflare.com. We review them in the Cloudflare TV segments like this. And then for folks that want to kind of follow along in real time, they can follow us at Cloudflare Radar on Twitter, and potentially seems to be mastered on. They can also look at radar.Cloudflare .com. And on Radar, there's also the Cloudflare Radar Outage Center, where we plot and provide more information on the outages as they occur. Exactly. Here it is. So, a good sum up, and our time is almost up. So, thank you so much for this, David. Thank you. That's a wrap.