Cloudflare Radar Bulletin: 2023 Q1 Internet Disruptions

Hello, everyone, and welcome to our Cloudflare Radar special about Internet disruptions. It's an overview for Q1 of 2023. I'm João Tomé, based in Lisbon, Portugal, and with me I have David Belson. Hello, David. How are you? Good morning, João. Just for those out there, you're based in Massachusetts, let's say like that, right? I am. Just north of Boston. Exactly. Near Boston. This quarter, in terms of disruptions, we had a lot of different things, for sure. The war in Ukraine is still there in terms of Internet traffic disruptions, but also an earthquake, like really incredible stuff in terms of impact. Yeah, definitely around the gamut. Saw disruptions all over the world and for a whole wide variety of reasons. Exactly. Let's dig right into the blog post you wrote this week. It was out this week. The same week we also had a radar-related blog post about the DDoS trends, the DDoS report also. Yeah, both came out. The disruptions report came out yesterday and the DDoS report came out, I believe, the day before. Here it is, Internet Disruptions Overview in Q1 2023, and also the DDoS Threat Report I was mentioning before. Let's start in terms of overview. Where do you want to start? Like the blog post, government-directed? Yeah, let's take it from the top. Last quarter, we also did one of these for those who want to search the blog post or the Cloudflare TV segment. And last quarter, we spoke about Iran lengthy because there was a series of government-directed Internet shutdowns there, right? Yeah, those started, I believe, in late September. So we've hit that on the Q3 report or summary as well as the Q4. The ones that started in September were largely in response to protests over the death of a young woman there while in police custody. Those shutdowns have largely trailed off, those wide-scale shutdowns. Although we do, there are some reports that they're kind of regularly occurring on Fridays in a couple of small locations. But actually in January, we saw a shutdown in Iran across two major cell carriers, but it was actually for the old favorite to prevent cheating on academic exams. We've seen this in Iran before. We've seen it in a number of other countries where the government shuts down generally mobile Internet connectivity for several hours, sometimes over the course of several days or weeks, in order to try to prevent cheating on academic exams, try to prevent kids from sharing information about the questions or the answers or what have you. Exactly. We have done blog posts about that before, even in the summer with Sudan and countries in Africa, which sometimes is common. Yeah. So in this case, we saw basically a three-hour outage or a three-and-a-half-hour outage in a couple of provinces on January 19th. So not a full outage, but definitely visible impact of traffic. Exactly. Moving on to Mauritania, a clear impact. Yeah, that one was interesting. So there were four prisoners that had escaped from prison. And basically during the manhunt, for some reason, the authorities there decided to shut down Internet access or shut down connectivity, excuse me, across three major carriers there, across Chinguitel, Mauritel, and Mattel. And that disruption actually lasted for several days. So it started on March 6th, and then it seemed to have been resolved. Traffic started to return primarily on March 12th. They captured three of the escapees, or sorry, excuse me, three of the escapees have been reported killed, and the fourth was detained. So I guess the authorities decided that things were back to being safe and returned Internet connectivity to where it was. I think this perspective gives us, first, Mauritania is not a country that has possibly as many services as a European country, for example, because disrupting the Internet because of this in other countries could possibly lead to big problems in general. And next, Internet is really important even to catch, in this case, fugitives. Yeah, it just seemed odd that this would be a step that they would take in this case, you know, or in this situation. My gut tells me, like, you'd think they'd want to leave it on so that people could share information about, hey, you know, share tips and information and whatnot that could help potentially lead to their capture. But I guess they felt that in the public safety or something, they decided it was best to shut it down. Still in the government shutdown type of outages, India also present here, right? Yes. A region in India. Yep. So India actually is, you know, one of the biggest offenders in terms of Internet shutdowns. The folks over at sflc.in track Internet shutdowns in India, and obviously the team at Access Now through the Keep It On Project does as well. And so in this particular case, this is one example of that, where mobile connectivity in Punjab was shut down across four of the providers there, Cosmotophone, Idea, Barney Mobility and Reliance Geo. And it was down for about three days. This was related to protest -related violence. So we do see that fairly frequently, is that the governments will shut down Internet connectivity or often mobile Internet connectivity in response to some sort of protests or protest-related violence. This idea is to help either prevent more organizing from going on or to help prevent more importantly, for them at least, prevent audio video of what's happening there from getting out into the rest of the world. Exactly. And in this case, it was a big outage in terms of not full, complete outage, but a clear disruption there. Yeah. You can look at the graphs and you see that most of the traffic is really a fraction of what it was previously. Exactly. This is a recurrent in India, like you were saying, in these regions, mostly on the region level. Yes. Totally very localized. And next we have cable cuts, which is something that happens often, depending on the quarter, sometimes a little more than others. It could happen for all sorts of reasons. And in this case, this initial case is Bolivia, right? That was impacted. Yeah. So it's interesting. Cable cuts, we use sort of as an umbrella term. And I think within here, we talk about cable cuts in respect to both terrestrial cables, as well as some marine cables. So the first one we saw through the quarter was in Bolivia, where CometCo, an ISP there, reported that international fiber links, problems with international fiber links, excuse me, were causing problems with Internet service. So one of the good things, I guess, is that in the case of cable cuts or cable related issues, a lot of times the ISPs will be a little more forthcoming about problems and say, you know, they'll put something up frequently on social media saying, you know, hey, here's an alert. We're seeing degraded service because of problems with a submarine cable or Internet. In this case, they were kind of vague and said international fiber links, what have you. So, you know, unfortunately, in this case, they weren't clear whether it was a terrestrial connection to a neighboring country or if it was a submarine cable that was, you know, several hops upstream. Bolivia itself was landlocked and not connected directly to any submarine cables. So I think that, you know, it could have been either in that case. But regardless, we saw about 80 percent traffic loss for about eight hours. That's a lot in terms of hours in the country. Definitely points to probably the need for more redundancy there as well. It's interesting because in some of these cases, it's not clear what happened. Some countries reported better. Others are not very much. Don't give the details. Yeah, that's always the frustrating part from a research and analysis perspective is you see, you see it's changed in the graph, you see it's changed in the traffic data. But when you go dig in, a lot of times you just come up empty because nobody's saying anything about it. And in this case is also Anguilla is one of those other situations where a subsea fiber break happened. Yeah. Yeah. So in this particular case, it was a Facebook post from the government, which is interesting. So it wasn't the providers themselves, but the government said, hey, there's an issue. It's affecting these two big service providers in the country. They did note that the issue, though, was due to a subsea fiber break, as they said. So when you look at submarine cable map dot com telegeography, you can see that the only submarine cable that's connected to Anguilla is the Eastern Caribbean fiber system. So that may be the one that had an issue here. You know, we can see from the graphs that traffic dropped pretty significantly, both at the country level as well as across the two referenced. Well, so a country level is certainly across AS2740, which is Digicel, which the government referenced. And then AS11139, which is cable and wireless, which is actually Lowe, the other one the government referenced, did see a slight drop in traffic. But, you know, if you compare the two, you can see it's nowhere near as significant as the other one. So clearly they have some additional redundancy somehow. Exactly. Moving on to another part of the world, Bangladesh in this case. This was a brief disruption, right? Yeah. Yeah. Very brief. A couple of hours. This one was due to the Internet's old enemy, BACO. So with the ISP Grameenphone there said that the outage was caused by fiber-constituted road maintenance. So that was that was the old joke. Historically was, you know, what's the Internet's first enemy? BACO, because you'd always have those problems. Somebody's doing road construction. The cables aren't marked. You know, you rip them up and out goes the Internet for some state or city or whatever. Somebody cut the wrong wire in a sense. Yeah. It's probably unintentional. You know, you're digging and you just, you know, you don't know that there's cables there and you bring up a scoop and go, oh, like that shouldn't have been there. Exactly. This one was a brief one. And then Venezuela was a bigger one in a sense, right? Yep. So Venezuela saw a couple of issues on February 26th and 27th and then saw a much longer issue around 17 hours, 17 and a half hours on February 28th. And CanTV, which is one of the local major local providers, said that they had an outage in their fiber optic network. And then the timing of their post, their tweet was coincident with the outage of the saw. So you can kind of tie those together. Exactly. And like you say here, Venezuela is no stranger to Internet disruptions. Even in 2022, they happened more than once. Oh, absolutely. Yeah. So it's interesting that CanTV will frequently see Internet disruptions on their network. Venezuela, a lot of times we're seeing Internet disruptions due to power problems, due to power outages. In fact, there was a post this morning from the folks over at IOTA, the morning that we're recording this, where they observed some disruptive connectivity across a number of Venezuelan states. So I haven't had a chance yet to go look at our data to see if we see it. But interestingly, CanTV didn't post anything. The Venezuelan power company didn't post anything. So there's this open question of what's causing them. And so these things are ongoing. Actually, it's not on this blog post per se, because this is related to Q1 of 2023. But last week, there was also a big outage in the UK in one of the main ISPs there. Right? Virgin Media also wrote a blog post about that. Yeah, that one, however, wasn't Fiber related. So they haven't, as far as I know, they haven't publicly shared information on what caused that. But my understanding from seeing folks posting information they reportedly received was that it was more of an infrastructure issue. They talked about re-seeding the cards. Basically, Virgin Media network engineers were re-seeding Fiber cards or something to that effect to help fix the issue. So, yeah, so last week we saw basically two outages. And this one, we saw the earlier one that's sort of in the middle of the graph there. And you can see the right side of the graph, traffic going down again. We published the post as the second outage was in process. And then we also showed within the post part of the impact is that because Virgin Media's TNS servers are on their network, that prevented access to VirginMedia.com and a lot of those sorts of resources. Exactly. So they were using their own services. That's why even their website was down. So there's no access to the website, no access to the status page. Even to communicate with customers, right? What's that? Even to communicate with customers, they didn't have their own page. Social media maybe, but their own page. Right. Let's continue with the power outages part now. This is another category of disruptions, power outages. And Pakistan is the one next, right? Yeah. So this one was a countrywide power outage, which we saw across Pakistan in January. This one had clear impact. The reports were that the outage impacted over 200 million people. And as such, caused a significant drop in Internet traffic. The interesting thing with power outages is that they are frequently manifesting themselves as Internet disruptions from an end-user perspective. So it's not that the Internet infrastructure necessarily becomes unavailable. In many cases, the infrastructure is housed in facilities that have backup power, have generators and batteries or whatever the appropriate thing is. But because end-users, subscribers, people lose power, that's why they wind up going offline. So Pakistan has a long history of these wide-scale nationwide power outages. It doesn't look good for their energy infrastructure in that sense. No, I mean, and that's the challenge. I think a lot of these countries have energy infrastructure that sort of just works. It's not strong, it's functional, but it's not overly redundant. However, I mean, I say that sort of sitting in the US and ours isn't spectacular either. I've had experiences where it's a bright sunny day out and my power goes off. And more to the south in terms of the American continent, Argentina also had. Yep. So again, another large -scale power outage. This one was due to soaring temperatures. So I think you have higher temperatures, it gets hot out, people put on the air conditioner. A lot of times, the grid can't handle all that extra load or they're just not ready to handle it. So the outage is there. I wind up dropping Internet traffic by about a third. And that lasted between 1630 and 1930 local time. So fairly short, about three hours. And invisible certainly in a number of states there. Africa, also Kenya here in this case. With a nationwide power outage in this case. Yeah. So again, the Kenya power company posted a customer alert talking about what happened. Which is, again, it's great to see the transparency and the communication there. They said that there was a system disturbance. I don't know exactly what that means, but they had some issue obviously. And that resulted in a power outage that lasted about five and a half hours. On the news, worldwide was the earthquake in Turkey, in a sense. And we saw a clear impact there too, right? In this case, after the earthquake happened on February the 5th. Yep. Yep. So very much so. So yeah, it was a big earthquake. Lots of death and injury. But when we looked at Internet traffic from a number of the regions within Turkey, you could definitely see an impact. We looked at the traffic, or you looked at the traffic, where it was the week previous. And you could see it was significantly lower. So in this first region, it was about 94% lower. So really just almost gone completely. In Osmania, about 88% lower. In Hatay, 76% lower. Kilos, 63% lower. So previously, prior to the earthquake, we were seeing pretty consistent week-over-week traffic. We knew where it should have been. We knew where it was. And then once the earthquake happened, we'd see the very clear impact. Traffic dropping almost immediately when the earthquake occurred. And then you see it sort of slowly, very slowly, in some cases, trying to... Very slowly growing, I guess, as people get some power back, as the infrastructure is restored. But it took really several weeks. And as we call it in the post, a month later, traffic volumes had largely recovered in most of the regions. Although it was still much lower in some of them. We discussed this, actually. We did a bunch of tweets monitoring the situation over the weeks. And first, my surprise was these are regions with a lot of people, millions of people, in the south of Turkey, near Syria. And what I was more surprised was that, of course, the earthquake was so intense that a lot of people died. Houses, thousands of buildings were destroyed. And in several, at least two or three of these regions, traffic never recovered to previous time. Because the destruction was so big, people moved from two other regions. There's such a massive amount of infrastructure damage, and that just takes time to rebuild. Weather. Weather also could have played a role. Q1 was a really active cyclone quarter. It was sort of interesting to see that the weather -related outages that we observed were related to cyclones. The first one was a cyclone Gabrielle in New Zealand that was mid-February. That impacted traffic in a couple of states in New Zealand, a couple of regions in New Zealand, Gisborne and Hawke's Bay. That both lasted several days. Traffic dropped there on February 14th, and then had returned more or less around February 19th. Destruction that these massive storms caused. And the same thing for Vanuatu, right? A cyclone. Yep, Vanuatu is a small archipelago of islands in the South Pacific. And here you can see with Cyclone Judy, when it struck on February 28th, it dropped traffic by about 80%. And then took about two weeks for traffic to recover to early February levels. So again, just having to fix the infrastructure damage. And in this case, it was another cyclone in Malawi. So a different region of the world. This one's interesting because it was reported to be basically the longest-lasting, most powerful cyclone. And it hit Malawi over a weekend in March. And my recollection is that this actually went back and forth a couple of times locally. Where it was sort of bouncing between Malawi and I'm blanking on the country next to it. But it did a lot of destruction damage over there. But interestingly, the Internet impact seemed to be kind of more nominal. We can see obviously a drop in traffic over the course of the 13th and for several days afterwards. But interestingly, it didn't drop traffic to zero. So Malawi's infrastructure may be a little more stormproof. It may be a little more resilient. But with a cyclone like that, I would have expected, I think, a little bit more of an Internet impact. Absolutely. Let's move on to technical problems. We don't have a lot of time, so let's try to go over these ones a little bit faster. South Africa, what happened there? Yeah, so South Africa, RSA Web, their major provider there, they've had issues in the past. Initially, they posted about a problem that they said they were observing in their system. And through a series of tweets, they ultimately talked about the different services that became unavailable. And what they were doing to fix it. So you can see within the graph here, traffic dropped early in the morning, UTC on February 1st. And it largely recovered later that day. But RSA Web ultimately said that full restoration of services across all of their platforms took about a week. Exactly. Another example here is Italy. Usually, you were saying it's not only in non-European countries that outages happen. Just before, we were talking about Virgin Media in the UK. And this was a multi-hour Internet disruption in Italy on February the 5th, right? Yeah, in Telecom Italia primarily. So this one, they talked about a quote-unquote international interconnectivity problem. So not a lot of details there. So you're kind of left wondering what exactly happened. But it falls under the heading of technical problems. And here, you can see a little bit of volatility within the traffic graphs at a country level and certainly at a network level. You see at a network level, it's clear to see the traffic drops and then ultimately recovers several hours later. Exactly. And here is the Italian telecom here. Tim represented. Moving on to Myanmar. It was a fire at an exchange office, right? In this case. Yeah. So we've seen that before as well. I think something similar happened in Iran last year, where basically fire causes damage to infrastructure at a central exchange, at a core infrastructure point. And that ultimately disrupts services. So they said that they had problems with the call center, fiber Internet, mobile Internet, telecommunications. So when you look at the two autonomous systems that belong to NPT, you can see very clear multi-hour drops in traffic there. Exactly. And also in Congo, there was a problem in Congo Telecom. So again, they're great about being transparent about it. They issued a communique, they called it, but really only cited a network incident. So they didn't provide any more details on that. But regardless, you can see that this network incident caused a problem with Internet connectivity that wound up creating almost a complete outage at a country level. And when you look at their autonomous system there for MTN, it caused basically a complete outage for a little more than six hours. Congo Telecom, the outage was probably about the same length. If we're just looking at the graphs here on different scales. And moving on to Lebanon. Lebanon was interesting. Yeah, because here is basically they're saying, hey, there's other issues. And basically, the network's going to stop working if the generators run out of fuel. Ogero is the main upstream provider there. And I think they were due to strikes, worker striking. So there was a lot of sort of moving pieces here that ultimately likely were related to the outages that we observed for several hours. And this is not a very common disruption situation, right? Cyber attacks that we're seeing. Right. Yeah. So we've seen a couple of times in the past. I suspect also the providers probably don't like to really admit that they've suffered the impact of an attack. But in late January, LG in South Korea talked about suffering two brief DOS attacks. And those, you can see on the graph, basically caused two very brief disruptions to traffic on their network. Exactly. And WAM also related to a cyber attack in this case. Yeah, same thing. They talked about a cybersecurity incident and their servers being attacked. But ultimately, what happened was it impacted traffic enough. The impact or the outcome of that attack dropped traffic enough on their network. It was almost a complete outage and certainly enough to be visible at a country level as well. Exactly. Last but not least, I was mentioning at the beginning the Ukraine war. We did a blog post about the Ukraine war, a whole year of Internet disruptions and resilience. There's also a lot of resilience discussion on that blog post. This is some of the disruptions that happened earlier this year because disruptions continue to happen. Focus on Odessa in the south, right? Yeah. Again, this goes back to the shift in strategy, I think, that Russia chose in late Q3, early Q4. It shifted towards attacking electrical power infrastructure within Ukraine. As we talked about earlier with the power outages, the disruptions that we're seeing in Internet traffic in these various areas of Ukraine are basically a manifestation of the power issues that result from destruction of infrastructure. I think the good thing ultimately is that these disruptions tend to be fairly short -lived, multi-hours. Thankfully, the ones that we've been tracking here at least have only been multi-hour disruptions and not complete outages or multi-day or multi -week issues. Exactly. In this case, Kharkiv also impacted. A few regions actually also impacted over March inclusively. It was a full quarter of differences in terms of different disruptions in this case. Definitely busy. Definitely busy. Before we go, anything we should highlight? Of course, there's Cloudflare Radar that anyone can browse through information. Then there's the Outage Center where disruptions usually are there for people to see as they happen. The latest one was Starlink related, right? I'm sorry. Yes, that was the last one we posted. Exactly. That was the other day. There's also the Cloudflare Radar Twitter account and also presence. Follow us on Twitter. We're Radar at Cloudflare.social and Mastodon, which we try to post there as well. Just follow us everywhere you can. Exactly. It was great, David. That's a wrap.