Originally aired on October 31, 2022 @ 2:00 PM - 2:30 PM EDT
The third quarter of 2022 saw a large number of Internet disruptions taking place across multiple countries with a variety of underlying causes. David Belson and João Tomé discuss some of the major disruptions observed through Cloudflare Radar and their underlying causes.
Additional details can be found in the related blog post: Internet disruptions overview for Q3 2022 .
Hello and welcome to our Internet disruption overview for Q3 2022 program. I'm João Tomé, I'm based in Lisbon, Portugal, and with me I have David Belson, our head of Data Insights that is in the Boston area, USA. Good morning, I am. Morning, afternoon or night, depending on whatever it is. In this show, we want to highlight the blog post you wrote about the outages that happened on the third trimester of this year. A lot of things to talk about. They're ongoing, so there's new outages and shutdowns and problems in terms of the Internet, for sure, after Q3 that we are monitoring on our Cloudflare Radar Twitter account, but this one is focused on what we saw in Q3. Yeah, absolutely, and it was sadly kind of a busy quarter for Internet disruptions. In our post, we try to be comprehensive, but it's definitely not exhaustive, so there are certainly shutdowns and other sorts of outages that we didn't include for a variety of reasons, but I think it was definitely a very full quarter for those types of events. Exactly. In terms of events, and going to your point, we saw very different ones, which it doesn't happen all the trimesters or all the quarters, for sure, from government -directed shutdowns, exams, protests, natural disasters, power outages, cable damages, technical problems, so it was the full scope of things that could happen wrong. Definitely around the gamut in terms of causes, yeah, absolutely. Exactly, and in a sense, there's some worries in terms of the war in Ukraine, if undersea cables are safe around the world. We didn't see much, even on the news, in terms of disruptions that seem like related to that, but there was some disruptions in terms of undersea cables. Yeah, so we can even start there, which is great. One of the major issues we saw with respect to cable damage in the third quarter was in Pakistan, but that was actually more a function of damage to the cables that resulted from flooding and heavy rains and whatnot in the area, so just simply where the cables are located, where they're buried, the flooding and the heavy rains ultimately wound up disrupting those and damaging them and ultimately disrupting Internet service for folks locally. But to your point about the concerns about subsea cables, that really came up a lot in September and October after the Nord Stream natural gas pipeline was damaged or sabotaged. We took a number of calls from the press, and I did a where there's a great deal of concern now about the safety of subsea cables, and the press wanted to, and government, frankly, wanted to better understand what's out there, how reliable are they, and more importantly, how redundant are they. So the severing of a single cable will take some countries or some locations offline. In fact, we saw that early last week in the Shetland Islands in Scotland, where one of their cables, or they were primarily connected to the Scottish mainland through a submarine cable. So apparently now a fishing vessel caused that damage and took them offline for a few hours. But if you're talking Europe to the US or Europe to Africa, Europe to APAC, or continent to continent in essence, you're going to have a lot of redundancy on those routes and a single cable is generally not going to cause a catastrophic event. I was actually working on a blog post about that that I really need to finish. But in terms of cable damage, we've probably seen a little more activity there in the past quarters, but it's definitely something now that's getting more visibility out there. For sure. And earlier this year, we had the Tonga situation. That was a clear disruption. Another single point of failure situation there. So there's more dependency on that specific cable. So the full country was impacted in that situation. Absolutely. You're definitely going to see that in more of the remote Pacific Islands. Oftentimes any island country or island territory is often heavily reliant on submarine cables to reach the global Internet. And for better or worse, sometimes they're reliant on a single cable. Or what we see sometimes as well, like in Tonga, I believe is that Tonga was connected to Fiji and then Fiji was reliant on cables to other places in the Asia Pacific region for Internet access. So it was sort of the stack of single point of failures, if you will. Exactly. Exactly. Makes sense. Let's dig in a little bit more on the government impact in terms of shutting down the Internet. Yeah. So, unfortunately, we continue to see that as a frequent cause of Internet outages and shutdowns. The government-directed shutdown is basically the government is trying to generally prevent something. So, for instance, in the third quarter, we saw the government is trying to prevent cheating on exams. And this is something we've seen over the last 10 years or so, where the governments were taken to shutting down the Internet, oftentimes nationwide for several hours, while secondary school exams are going on. So we saw that in Iraq. And they did that, I think, twice a week, like Mondays and Thursdays from the end of June into July. So we saw the outages there very clearly. And then we also saw it in Assam, India, where India has unfortunately been a popular source of Internet shutdowns, sort of kind of flies under the radar there. But there are many, many localized shutdowns in India. So another one here was Assam, where shutdowns happened on the 21st and 28th of August to prevent cheating on exams in Assam. Exactly. So this is like focused on the region. India has a lot of that. So they focus a lot on specific regions. So there could be, it's a big country, there are a lot of people, there could be different outages at different times, depending on the regions in India. Yeah. India sees a lot of very localized Internet outages or Internet shutdowns, in contrast to a countrywide Internet outage that we've seen elsewhere. So one of the other things we saw from a government -directed perspective in the third quarter was Internet shutdowns due to protests. And that we saw across a number of countries in the quarter. But the one I wanted to focus on specifically was in Iran. That's actually ongoing. So that's something that didn't stop. Right, right. So it's sort of shifted. They've shifted their approach a little bit. But in Iran, after a young woman was arrested and died while in police custody, there were a lot of protests around the country. So initially, after her death, there were a couple of localized outages in Sanandij and Tehran. And Tehran is the capital, so it has a major impact there, for sure. Yes. And what happened after that was, for several weeks, I think about two weeks, there was effectively an Internet curfew that was taking place, where three mobile providers around Cell, Rytel, and MCCI were shutting down connectivity between 4pm and midnight local time. So basically in the evening. And that happened from, I think, the 21st or 22nd of September for several weeks into early October. And then they sort of shifted tactics a little bit, and the daily shutdown stopped. And they shifted to a weekend approach, where they were shutting down the Internet on Saturdays and Sundays. And then I don't think they've done that over the last couple of weekends, so now mid-October. With the same pattern, right? Yeah. So what we saw was, earlier in the month, they were doing that over the weekend. I think this past couple of weekends, they didn't shut down the Internet. But now we've seen a little more targeted shutdowns. For instance, yesterday, again, it was the 40th day since that young woman's death, and there were protests sort of marking that. And so there was a multi-hour shutdown in Saint -Étige again yesterday. But also, as a note here, as you're showing, not only were they shutting down Internet connectivity across those three mobile networks for several weeks, but even countrywide, we saw them blocking DNS over HTTPS and DNS over TLS. So when we looked at our traffic data for those capabilities, those services, we saw the traffic from Iran effectively go to zero. And then what they also apparently did is started to block QUIC and HTTP3. HTTP3 runs on top of QUIC. So you can very clearly see it on September 21st for Iran, the percentage of traffic we saw over those two protocols dropped from about 23% down to effectively zero. For those who don't understand the ins and outs of the protocols, what does this mean, really? Sure. So historically, HTTP, the protocol that the web runs on, leveraged TCP, trust for control protocol. So that has historically been the primary underlying protocol that many Internet protocols run on top of. And QUIC is something that has developed over the last several years. It was finally ratified as a formal standard earlier this year, along with side HTTP3. So HTTP3, it doesn't leverage TCP as its underlying transport, it leverages QUIC. So in essence, by blocking QUIC, you've effectively blocked HTTP3 as well. But in the context of Iran, this means that it's about what type of usage is being done? Yes. So the other thing there is that it is easier to implement blocking on the TCP -based versions of HTTP. So HTTP1.1, HTTP2. And our conjecture, again, it's simply a conjecture, is that some of the techniques that were effective previously could not be applied to the QUIC slash H3 -based traffic. So they took a more heavy -handed approach and chose to just block QUIC traffic outright. Before we go into other things, I think it's good to notice, because this situation is ongoing for a lot of days. So this is something that has a real-world impact, not only in a week, but in a lot of weeks. And you were saying before of this Internet curfews that were happening after 4pm. And I remember discussing this with you, actually, at the time, that we saw also on the news, the protests were happening more at late afternoon, not in the morning, for example. Right. So I think what they're trying to do is block Internet connectivity, especially, again, the mobile connections, so that if you're out there, you're planning to protest, you're out there protesting, it becomes much harder to organize, it becomes much harder to communicate. And probably more importantly, it becomes much harder to use the Internet to share pictures, video, audio of some of the atrocities that are unfortunately also happening, part and parcel of these protests, government crackdowns and the violence and whatnot. Exactly. That's also important, the communication part, but the sharing what's happening part, even with external players, for sure. Let's continue to natural disasters. There were a few also this quarter, this past quarter. There were. Yeah. And so there were a couple of earthquakes, both, oddly enough, 7.6 magnitude. One happened in Papua New Guinea. And obviously, the physical damage that caused by the earthquake ultimately impacted a submarine cable, which caused a disruption to connectivity there. Again, not surprising, with all the movement, both on land and undersea, that will happen, of course. And again, going back to Tonga, the volcanic eruption and the resulting earthquakes damaged the cable there as well. But in this case, it was Papua New Guinea in July. And then, excuse me, not July, September 11. And then a little more than a week later, we saw another one in Mexico. So again, another 7 .6 earthquake. And again, in this case, it was a bigger impact, actually. Traffic dropped 50% in those states that were affected. So quite a big impact. So not a complete outage, thankfully. Some limited damage. But thankfully, they were able to restore connectivity fairly quickly as well. In these two cases, the outages or the disruptions only lasted a few hours. Exactly. Although, yeah, if you think about it, in the case, for example, of Mexico, many people live there. So this is impacting a lot of people. Even if it's not a complete outage, it's 50%. So a few hours, it takes a toll depending on the country, if they use more the Internet or not, they have more people or not. So the toll is different for different countries and different use cases. Absolutely. And then we also saw with natural disasters, we saw several hurricanes. And we had two particularly significant hurricanes that came up from the Caribbean and made their way up the east coast of the US and into Canada during September. The first one was Hurricane Fiona, which made landfall in Puerto Rico on September 18th, ultimately caused island-wide power outages. The power grid in Puerto Rico is notoriously finicky anyway. And when it gets hit with a hurricane, that does some significant damage. Thankfully, Luma Energy, which is the local power company there, they actually have done a really good job historically of keeping customers apprised of the progress they're making on restoring power, using Twitter very well to communicate. So they continued to post updates about the number of customers that were coming back online. And we could use that also as a checkpoint in terms of looking at our traffic. And what we saw was that it took about 10 days for traffic to return to the expected levels, to reach the levels that they were at before. Exactly. They have their own... Actually, this is something interesting. So these companies are using Twitter and live updates to show what they're doing. So it's also an Internet way of communicating. Interesting, for sure. It is. It's one of those things where it's sort of like, okay, great, you're communicating on the Internet about something that maybe the people that most need to know that information can't see because they don't have the power to power their Internet connectivity. I shouldn't be laughing. It's not funny. But at least they're communicating to those that are interested. There's ways of getting that information. But there's a good viewpoint there that, for example, if they're communicating and it's getting on their local news, may that be radio or TV. That's true. That's true. The information is getting through, even if they're communicating in a way that people aren't able to connect directly. But if like... It shows my bias in how I consume information these days. They're talking about radio and TV. Yeah. Most of us do that. But it's interesting to see that the Internet is helping even mediums like radio also getting information. Absolutely. Absolutely. Yeah. And then still with Hurricane Fiona, it hit the Turks and Caicos. So again, on the Caribbean, with the damage it caused there, that took about a day for traffic levels to recover. So clearly, the impact was... Clearly, the damage impact were not as significant. Exactly. In Puerto Rico, it was over 10 days. That's a lot of days of impact. And then it came up the East Coast of the US and ultimately really hit Nova Scotia, the Canadian province. And again, really what it did is it caused a lot of damage. It caused power outages, which ultimately then disrupted Internet connectivity. We saw lower traffic volumes from certain places because end-users had no power and then really ultimately couldn't use the Internet. So what we saw there is about five days of lower traffic volumes related to Nova Scotia. So again, with Nova Scotia power, working quickly to restore service, and again, using Twitter to communicate about it. Exactly. And especially the first day of outage is really clear. Very, very clear. And then it picks up. And it's interesting also to see that people are in some of these areas that are impacted, and sometimes they don't have power, but they're still able to connect via mobile devices. Right. That's why we don't see a lot of damage. It didn't sort of clear-cut the island. Exactly. So it's pockets of damage, areas where there's more or less damage. But again, like you said, the mobile connectivity, in many cases, people will find a way to charge their cell phone. And the mobile towers and the telecommunications infrastructure is generally a little better protected and more often has backup plans with respect to generator power and getting fuel in the case of emergencies and things like that. Sure. And then we had Hurricane Ian. Hurricane Ian, which made landfall in Florida almost a month ago now. And there were a number of cities that were really, really impacted, and that we saw traffic volumes drop significantly from. I've been monitoring those basically on a daily basis. It's now literally a month later. And I need to double-check again this morning, but traffic is, as you can see here, is gradually returning, but still has not fully reached pre-hurricane levels yet. So this is taking quite a while to restore here. And in this case, I think it's not only power outages, but I think there's a lot of wireline Internet infrastructure that was damaged here. So the poles come down and it takes down the power cables, it takes down the fiber optic cables, it takes down the copper for phone lines. So I think that's one of the reasons we're seeing a more gradual return to pre-hurricane traffic levels. And then we also have a focus on power outages, a few. Right. Yep. So the natural disasters, like we mentioned, ultimately caused power outages, but there are several others that weren't from those sorts of events. There's one in Iran. We see actually power outages in key data centers in Iran. I don't think this is the first one we've seen this year. And I think that unfortunately seems to be a more common occurrence than we'd like to see. Generally, those are fairly brief disruptions. Venezuela, unfortunately, sees very frequent power outages that impact a number of the states within the country. So we saw another one, it's highlighting here, another one of those that we saw in August. And then the thing I want to focus on here as well is Ukraine. And you've done a great job of capturing a lot of the Internet disruptions that have been happening in Ukraine over the last, geez, what is it now? Eight months. And what we're seeing now is more targeting of power infrastructure within Ukraine. And that targeting is obviously causing power outages, or it's causing ultimately reduced generation capacity. So citizens or customers are being asked to reduce their power usage. So those ultimately we're capturing, those sorts of events and issues, we're capturing as power outages across various parts of Ukraine. Exactly. And it's ongoing. I'm showing here our Twitter account. Yeah, that's right. This is the last couple of days. Yeah, there's clear impact in some regions. Actually, I was checking today and today is even worse. Today, the outages are really, it's not outages, but the disruptions are even bigger than yesterday. So it's ongoing. Larger drops in traffic volumes. Yeah. Yeah. So this is something that we have definitely been seeing a lot more of, I would say, over the last couple of weeks. Exactly. And this is actually, this is all related in the sense that in this case, air attacks in Ukraine are causing power outages, because some of those are to energy infrastructures. Because of that, that affects, of course, the Internet for most cases. Like I was showing, it's not sometimes not a complete outage. Sometimes it's 25%, sometimes it's 50%. We saw cases like in Kursyn that was like even this week, more than 75% of impact of lower levels compared to previous week. But it's really ongoing, especially in October and also in a few parts of September that we're showing here. Right. And then we talked about cable damage earlier. I think the one big event that we can probably close out on, I'm not quite sure how much time we have left, but is Rogers. It was on July 8th, starting early in the morning. And that was surprising, because Rogers is one of I think the two or three largest telecommunications carriers in Canada. So they had... Millions of customers, yeah. Yep. Millions of customers, millions of end-user customers cut off, but then also impacts to businesses where credit card systems went down and banking systems went down. And just a tremendous reliance on ATMs, so it's really wreaked havoc on a not insignificant portion of the financial infrastructure in Canada. So I was actually at a wedding when this happened, but the team here did a great job of digging in and trying to figure out what the heck was going on. And I think saw over the course of whatever it was about 17 or 18 hours that it lasted, fits and starts of like, hey, we see a little bit of traffic coming back. Hey, we see some BGP announcements. And then it would kind of all die off again. But I think that ultimately it came back by the next day, they were kind of back online. Yeah. After two days, they recovered back to usual levels, mostly. Yeah. But I think they had brought a lot of it back up within about 18 or so hours. It was. And my recollection is that the post-mortem indicated that it was like a configuration issue within their network. And that, as usually happens, things that shouldn't have happened when they made a configuration change happened. And then it was this sort of collapse from there. And it was hard to recover from. So we did a blog post on the Rogers outage. I was showing that just now. Yeah. So I think to really close it out, we also launched the Cloudflare Radar 2.0 last quarter, or excuse me, at the end of last month, at radar.Cloudflare.com. And then within that, we launched the CROC or the Cloudflare Radar Outage Center. And within that, we are tracking all of the outages that we cover on Twitter and that ultimately we compile into the blog posts. So you can look at the map around the world and see where have we seen Internet disruptions over the last week or two weeks or 30 days, both looking at it from a map perspective and then At the table. And then the table perspective. And then links to our Cloudflare Radar Twitter account, for sure. That's also helpful. Yeah, we covered a lot here. And let's invite everyone to continue to check Cloudflare Radar data trends, but also the Outage Center, the domains ranking, we have that also too. In this case, in this topic, the outage map, I think it's a really good way to show, to see what's happening in the world. What's happening where, yeah. Exactly. And of course, the Cloudflare Radar page is also a good point to be aware of and to follow if you want. Absolutely. So I think we have a good sum up here of the blog, the Qtree blog. We continue to monitor some of the ongoing outages and there's a few. And so we invite everyone to follow us on Twitter if they want to check them out or in our Cloudflare Radar. So that's a wrap. Thank you, David. Great. Thank you. Let me just stop share for us to say goodbye. Bye-bye.
