Radar Bulletin: Q4 2023 Internet Disruptions

Hello, and welcome to Cloudflare TV. I'm David Belson. I am the Head of Data Insight at Cloudflare, where I head up the Cloudflare radar team. And joining me today to talk about Internet disruptions that we observed over the fourth quarter of 2023 is my friend and former Internet Society colleague, Dan York. Hey, welcome. I'm glad. Thank you for letting me on here and bringing me on all this kind of stuff. I'm with the Internet Society, Director of Internet Technology. I've been there for 12 years. I've been online since the 1980s, though. And so before all of this kind of things, and I've worked in voice over IP and voice over IP security, all those different kinds of things. And I come to you today in the context of our Pulse monitoring platform that we have, which is really our platform for telling stories about data and looking at things out there. So we'll get more into that as we go through this, I think. Yeah, absolutely. And I was lucky enough to be a part of the Pulse platform team as it was, I think, forming as an idea. And I think that we had launched the first version of that when I was still there. Yes, yes. Maybe slightly afterwards. I forget. In full disclosure, yes, we worked together while you were here and doing that. And if people go to pulse.Internetsociety.org, they'll see that we have a collection of different things that are all focused around there, which brings together data from multiple trusted sources, one of which is Cloudflare Radar, also like IOTA, UNI, Google Transparency Report, and like 20 other sources of different things. We focus on shutdowns. We focus on emerging technologies like IPv6 and DNSSEC and TLS. Resilience is an area that we've really focused on lately. We have what's called the Internet Resilience Indicator or IRI. We have now country reports that you can look at, and we'll reference a bit of that as we go through some of this here today. But all of these tools, all of these measurement mechanisms allow us to be able to talk and tell stories using data and be able to help policymakers, journalists, and others to understand the health of the Internet, the availability, and really how the Internet is evolving. So shutdowns, kind of what we'll talk about here, one part is just part of that larger picture. And we only focus on government-mandated shutdowns, which is a little different from your disruptions report where you look at- Right. Exactly. And I think that's actually, this is a great segue into one of the first topics, which was basically a high-level review of the causes of the disruptions that we saw in Q4. And just kind of trying to break them out, you highlighted government-mandated shutdowns versus kind of other, or the other outages. I think there's this question of intentional versus, I was struggling with what to sort of categorize the others as, the others like cable cuts and power outages, weather and maintenance, things like that, sort of intentional versus force majeure or acts of God or unavoidable. Right. I mean, look, we saw that particularly this past year in 2023 with all of the weather -related shutdowns, right? Whether it was extreme heat in some areas or extreme cold or extreme storms or floods, we had like kind of every possible thing you could have in terms of natural disasters this past year. And so many of those caused Internet disruptions, right? Which when you actually look at many of them, they were actually about power outages as well, because this- At the end of the day, and I think there's actually an interesting thing to point out as well, I think is that the disruptions almost come in sort of two flavors. There's the widespread power outages, which will take down the end user's ability to connect to the Internet. And I think many of us in the industry see that as a loss of traffic and ultimately disruption. But then there's the other side of it, which is a lot of times the infrastructure will stay up. So the routing infrastructure, the data center infrastructure is still humming along, but in many cases that will go down and cause an outage also. Yeah. But the weather events, it tends to be more, I think, end user-related. Oftentimes, or the distribution, right? The data centers may have their battery backups and their power systems and everything else that can keep them up and humming, but you can't get the Internet access out from there. When you looked at the hurricanes and the typhoons that have hit different parts of the world these past times, that has wiped out the physical infrastructure, the cables that bring people, the connectivity there. And so you can't get Internet because you don't have cables. You don't have any infrastructure. Right. And I think we see that in the recovery times also. And I think in some cases when it's a power outage driven, if it's, sorry, if it's, I'll call it a basic power outage, recovery times are often hours, days. I think what we've seen in Acapulco, what we see in Lahaina, Hawaii with the infrastructure damage due to the hurricane Otis and the wildfires out in Hawaii is that it has been, like you said, tons of infrastructure damage. And so you've got new cables, new poles, new buildings, houses aren't even there. Right. Yeah. I mean, you've got to do something on that line. I was just, you and I are both here in the Northeast United States and we've seen all sorts of flooding happening this whole year and just in the past weeks and things and just whole areas gone away. And that's all around the world we're seeing that. I think it's going to be very interesting. And part of our focus at Pulse as well has been around resilience and around what kind of resilience you have. And this is going to be interesting. For the last couple of years, I've been working on some work around low earth orbit satellites, like SpaceX's Starlink and stuff. And the resiliency story is fascinating. I'm also a volunteer with the IT Disaster Resource Center or ITDRC here in the United States, which does a lot of response after hurricanes or natural disasters and helps provide connectivity. They could take, I haven't done this because I'm not in, but out in California, people who are responding to wildfires can drive a pickup truck out to where the wildfire is. They've got a Starlink dish on the back that's powered from the truck or whatever. And they could just set that up and go and be able to go and have wifi for the local first responders or the community or whatever else. That's powerful in a way to go and have a resilience story that we haven't really seen. Now that Starlink, there's also OneWeb, Amazon's Project Kuiper, there's a whole bunch of these Leo systems coming on, which will be interesting when we think about that disaster response, what they can do if you can get power to them. But so many of these weather stories, again, just come back to power. And also you mentioned in the post, the military action, right? Again, if you're bombing out, if you're destroying the infrastructure, there goes your Internet connectivity. That's actually a good segue. We can jump around a little bit to segue into talking about Palestine and Gaza. One of the things that we observed when the conflict started there in early October was that there were about eight or 10 or something Palestinian ISPs or autonomous systems as we see them, that basically went completely offline and updating, looking at things yesterday, all but one of them are still offline. That's now three plus months later. So I think as a result of the potential, the damage from the physical conflict has had a cyber impact. Yeah. You've destroyed so much of the physical infrastructure there and in Ukraine and all of that. And was the one that was coming back up, Paltel? I think it was a- It was not. No. So Paltel actually wasn't one of the original few that we looked at. I only wrote down the AS number on my notes. I knew I should have written down the AS number. There we are. But it looks like it came back up recently. We started to see traffic from it recently. But with Paltel, I think they've in the Gaza State, one of the major providers in the Gaza Strip, they have more or less stayed up throughout the course of the last several months. But what they've seen is either, they've seen and actually been very communicative about the outages on their network as they're occurring, either because of lack of resources, like they ran out of fuel in their pops or their towers or whatever, running out of fuel. So basically the infrastructure becomes unavailable, power goes out. Or in some cases, I think more recently, there's been more infrastructure damage they've talked about. But they have seen, I think, 10 outages to date at Paltel since the end of October. There was one that I think was actually took place over the last couple of days, I think ended yesterday. And the one before that lasted about seven and a half or eight days, I think. So that was the longest one to date. But one of the things though, I think an important point is though that even when you start to see that the connection is being restored, it doesn't mean that the shutdown is over or the disruption is over. It just means that that local telco is working very hard to try to restore the connectivity and they've got partial service up. You see traffic sort of coming back. Right. And you may see BGP connections restored or pieces like that, but okay, connection's up, but they're still working on bringing that back up. So yeah, that kind of disruption is, again, you're going back to power, physical infrastructure, you just can't get the connectivity to where people are. The cell towers are out, whatever it may be. Right. And the impact there really is that you've got people now in a war zone who are unable to communicate with loved ones elsewhere, outside of the sort of their immediate vicinity. And the rest of the world's potentially having trouble seeing what's happening there, because it's hard to get video out or audio or text out. And aid workers or people who are in there trying to assist in some way have no way to coordinate or no way to communicate back. I mean, yes, they are finding ways, they're using older technologies, they're using short mesh networks, they're doing other pieces like that, but they don't have the Internet connectivity, which allows them to actually be able to function in their role right there in some form. Right. There was an article I saw the other day talking about using, I guess, eSIMs from neighboring countries, where I guess you can try to connect to a tower from a provider in a neighboring country, I guess if you're close enough to the border, I guess. Yeah. And there's been a burgeoning, I've seen a number of those reports around people who are really trying to provide eSIMs to people who are in there so that they can be able to connect. Again, you've got to be in proximity, you've got to be able to go and do some of that. It's a challenging piece but there are options around that in some form, but they're minimal. It's really about that type of thing. Yeah, absolutely. Absolutely. Your big story though, that you led off with in the report was kind of the shutdowns angle around Iraq. Yes. I was just going to bring that up. So yeah, so with Iraq is interesting because I think over the last, I want to say, seven years, I think it started in 2016-ish. We've seen Iraq and other countries implementing Internet, intentional Internet shutdowns around exams, academic exams. Generally, these shutdowns occur for three, four hours a day over the course of about a week or so. And they're ostensibly to prevent cheating on the exams. I guess people get copies of the exams ahead of time and they can share pictures of them on WhatsApp or whatever. I think the government ultimately is trying to avoid that, but taking a very heavy-handed approach and shutting down the Internet across the country. Right. And you look at this and you say, the Internet at this point is woven into every fabric of our lives. We're using it for everything. We're banking, we're communicating with government, we're sending messages, we're communicating, collaborating, creating, doing all kinds of stuff. We're ordering food, right? We're doing everything on this. And the crazy part is many of these companies, Iraq included, are trying to grow an online, a digital economy. They're trying to do that. And then they're just like saying, shut it down for a couple hours. Now, one of our people at Pulse, Hannah Crichton, explained to me one time what happens in some of the countries. They're shutting it down during the time of the distribution of the exams. They're going from the Ministry of Education site out on trucks, et cetera, out to the test sites. And so during that distribution time, they're going and shutting down the Internet. So somebody can't just take something off the truck, start to photograph a bunch of stuff, send it to somebody and go and do that. So it's less the, I'm going to take my camera and my phone into the closet while I'm doing the exam and text it to my friends. It's more getting that information ahead of time. That's one, at least in one, but I think in others, they have historically tried to block it during that time. So yes, you couldn't smuggle your phone in, take a picture, send them to your friend or somebody who's going to pay you a lot of money for it or whatever else. But I mean, come on, you're shutting down everything. The blast radius there is really significant for what is arguably a limited - Yeah. I mean, you're blowing up, you're using dynamite or something. You're blowing up everything just to go when you need to really take a scalpel or something. I mean, it's like, wait, so people can't order the equivalent, the Iraqi equivalent of DoorDash or whatever. I mean, all the gig workers who are there are just chilling out because they can't do anything. Right. No medical, no e -commerce, no e-banking. Right. Oh, you want to pull something out? You're on a WhatsApp chat. You're interacting with your customers over WhatsApp or some other system. You can't do calls with them. I mean, not until 9 a.m. or whatever. And you posted a schedule in your blog post that really showed these are many, many things. In our pulse, in our country report for Iraq, we showed there were 121 shutdowns over the past 12 months. Okay. Some of these were like a couple hours here, a couple hours there, whatever else. But I mean, that's just like, hello? I think the multiplier there as well last year was that there was a separate set of shutdowns in Iraq, and then sort of I'll call it Iraq classic. And then in the Kurdistan region, they also implemented a similar set of shutdowns. So it's sort of this, it wasn't just everybody shutting it, it wasn't the whole everybody in the border there shutting it down. It was the two separate areas that were implementing their own independent shutdowns. Well, and you had middle school exams and you had high school exams, different levels of exams at different points. So when you're looking at the list, because I play a minor role in Pulse where I help write up some of these shutdowns and the pieces are there. And Hannah and some of the other folks are very, are much more focused around this larger picture of things. But when I was looking at that list of all the shutdowns- Yeah, it's crazy. I'm like, wait, what? The interesting thing about this whole discussion on Iraq actually though, is that those shutdowns didn't happen in November. So that we had expected them, I think. And as you mentioned, I posted the, or I included the schedule within the blog post. And there was, I think between the 13th and the 21st of November, they had the third round of the 12th grade exams, and they had shutdowns for the second round and the first round in like August and September. But there was no observed shutdown in November, which is really interesting. And so I think the question you asked me last night of, are you sure they even had the tests? And to be honest, I hadn't thought about that as a potential reason. But doing some initial research, it appears that they did have the tests. And earlier in the year though, there was the Ministry of Communications, sorry, the Ministry of Education had said, hey, we want to shut down the Internet for this set of exams. The Ministry of Communications told them to take a hike. Yeah, right. Get all the complaints, right? All the Iraqi businesses and all the gig workers, everybody else, they're all complaining to the Ministry of Communications probably saying, hey, why are you shutting us off? Right. But they wound up shutting it down anyway at that point. But we didn't see it in November. So the question is, I guess, is this a new set of behavior for them? There's another set of exams coming up starting February 8th. So we'll see. That'll be the question. I bounced this off Hannah. And two things. One, I think maybe there is this tension going on between the Ministry of Communications who's saying, wait, we're trying to build a digital economy. Shutting off our Internet for blocks of hours on multiple days is a really bad idea. Maybe they've looked at tools like the Internet Society's net loss calculator, which calculates how much you're losing during a shutdown. Maybe they've looked at some of that and said, hey, economically, it's not really there. Now, Hannah also pointed out there was an Iraqi election in December. So there could have been pressure within that to not go ahead with that. Fair enough. Yeah. So we'll have to see. But I think February, if that's February 8th, let's see what happens. Yep. And the other option there potentially was that they shifted to a new method of distribution. So it could be that maybe they've shifted more to an online exam where there is no trucking anymore. Or something. So it's sort of you fire up the site at whatever o 'clock and that's the first exposure anybody has to any of those questions. So yeah, we'll see. It'll be interesting to watch. Yeah, yeah. We'll have to go in and watch that and see what goes on. Got it in my calendar. Looking at- What do Internet shutdown geeks wait for? What's in your calendar? Oh, it's sad. My calendar has the weirdest stuff in it. Yeah. But I mean, we should point out in fairness, I mean, Iraq was, as you said, it wasn't alone. Algeria had some shutdowns back in June. Syria had some in August. India did some shutdowns earlier this year as well. We could do hours on just the India shutdowns. Well, yeah. But there's this idea. And so I think our message certainly has been to people, hey, we understand that you have security issues around exams. Okay. But fixing it by blowing up your entire Internet, not literally. Okay. We're talking military stuff on the show. So I shouldn't say literally, but shutting it all down. It's a technical solution to a non-technical problem. Right. It's not the right answer because you're leaving all those people who are trying to learn, trying to communicate, trying to collaborate, trying to work, trying to conduct business. All of them are shut offline because you haven't figured out the right security mechanism to manage your exams. So- Exactly. Exactly. Yep. You're losing too much money in your digital economy. Yes. Yep. And to transition to another outage or another issue that we saw in the fourth quarter, once I'm talking about Ukraine, we're actually coming up on the, sadly, the second anniversary of the conflict there. We've been tracking for two years on and off the outages, the disruptions that are going on there. A lot of times it's been due to, as we talked about physical damage, has been a lot of power outages. Actually, in Q4, there was some weather -related outages in Ukraine, but the one I wanted to talk about specifically was at Kyivstar, which is the largest mobile operator there and one of the largest broadband providers in the country. And they said that they've seen, over, I forget the period of time, but they've fended off on the order of 500 attacks. But there's this latest one or this particular one, I guess, is big enough that it caused what they call a technical failure and it impacted connectivity there for almost over three days. So I think that's, again, an area where Internet connectivity is vitally important, but has become sort of, I don't want to call it unreliable, but has become impacted by just everything that's going on in the area. Yeah. I mean, the good news with Ukraine, I mean, this was bad for Kyivstar and for Kyivstar's customers, but the good news for Ukraine is that it actually, there's a good bit of resilience in there. Yes. Exactly. They actually have a very resilient infrastructure. Yeah. And that's, I think, an important part. Pre -conflict. Yeah. But even in pre-conflict and even as the conflict's been happening, they've been able to maintain a strong level of resilience. They've had multiple transports in and out, a thriving retail infrastructure in that area. So that's, I think, the good news is that they've got that, you know, as long as they can keep the infrastructure from not being destroyed. Yeah. And I think they've done a really great job over the last two years of, you know, when stuff is becoming damaged or becoming unavailable, they have been very quick to fix that. Deploying resources, rerouting, or doing whatever they need to do. Yeah. I mean, Amrish Fokir, one of our people involved with Pulse here wrote a post on this back in February about, it was a case study of the Ukraine as a role model for Internet resilience. And it's actually fascinating just to read about the Internet connectivity and the ways that it's there that does provide this kind of connectivity. And, you know, if we only had that in many other places, we wouldn't be having some of these kinds of issues, which stepping back to Iraq, part of challenge is they can enforce these shutdowns in part because they don't have a rich ecosystem of providers in other places. So they don't have a rich, you know, a number of connections going out. So they're kind of stuck. Yeah. Yeah, absolutely. I remember doing stories. Your post covered so many other things. Yes. No, no. Let's cover one more big one. And then when I get to also talking about some of the Pulse resources and some of the radar resources around Internet disruption. So Optus in Australia. So they are a major, obviously a major Internet service provider in Australia. Went down for about, I think it was about, we clocked it at about 10 hours, I think. And I think as it was happening, there was a lot of sort of shades of Rogers. Back in July of 2022, Rogers ostensibly shot themselves in the foot from a routing perspective and took their network offline for, I think it was more than a day. I remember I was in Delaware at a friend's wedding, so I couldn't even pay attention to it. But, you know, I think that when people were concerned, that was what was going on here. But again, you know, major carrier. So not only... Yeah, it's the second largest, you know, in Australia, right? It has, I think, 10 million customers in some form, 400,000 businesses. All those customers, right? And then you have the hospitals and payment systems and transport systems and all these applications and stuff offline. Again, it talks to, I think, the integration of the Internet in everybody's everyday life. And when you have something like that happening, it's sort of like, oh, I can't get an Uber. I can't get a takeout food. I can't, make a deposit or make withdrawal or transfer money or buy a book or whatever. Yeah. And I think it's important that, you know, it looks like it was configuration. It was some different things that happened when they brought systems up and those, and there were, to get, without getting too geeky, but there's elements in there that exceeded certain safety levels. Yes. So things all cascaded. It was a big mess and it took people off for that period of time. One of the things that we would love to see though, I think, in all these cases and several others that you mentioned in there is, mistakes happen, things happen, but we need to understand them. We need to communicate about, this is the same thing with the thing in Egypt. You know, we'd like to understand what happened. You know, there's a thing in the security community called root cause analysis or RCA, which would help us understand and help larger people. And so I think, I think we're still waiting for that in Optus, but. I think so. Yeah. I think, but I think that the underlying message here is we're imploring the telecom providers, please be more transparent. Please be more communicative when things like this happen. I think there are, to some credit, some of them are really good about that. And they'll be quick to say, Hey, look, we've got a problem. It was a cable cut. It was a fire at this pop. It was a fire. And they'll be really good about communicating that. But yeah, others are like, you know, you watch the support accounts on X and, you know, only several hours in, they're like, yeah, we're sorry. You're getting, you know, having trouble getting online, but you know, we're working on something. To be honest, you guys are a great example too. When Cloudflare has it out, did you write these ginormous blog posts that explain everything? Transparency is part of our DNA here. And that's what we need. Optus, we need, you know, Egypt telecom. We need all of those to go and do that because that's how we learn. That's how we grow. That's how we prevent these kinds of disruptions that are out there in some way. Right, right. So we've got a couple of minutes left. I wanted to jump over to just highlighting the Pulse platform and talking about the shutdown work that you guys are doing there. Let me see here. You should see my screen. So if you want to talk about the work that the Internet Society is doing around this page, and then also around the net loss calculator. Sure, sure. And I can, and people who are looking at this can see the other headlines that are up there as well. So this highlights the shutdowns that we're tracking. Obviously that what you see there in the orange is the ones that we currently acknowledge as live. If you scroll on down there, you'll see there's the list of, you know, what are the most recent ones? There was one just recently in Comoros that was there. India, you know, you can go on down from there, but this is our, we're writing about, these are government mandated shutdowns, or sometimes you can't be sure. This is one of the challenges, right? That you you're well aware of. You're never, you can't always be sure, you know, like this one from Comoros. It, it happened during protests, during an election, pieces like that. It sure looked like a government mandated shutdown, reaching out to some of the folks there. It may have been a technical issue at the main provider that just happened to work in that same kind of time, but it's hard to know, but this is what we do. We show some narrative. We show some analysis. You can see there's some iota in there. There's some Cloudflare radar. There's some other different pieces that are there. And we try to explain to people in plain language, you know, what's going on. Why is this important? We include links to other community sites, to news sites, to pieces that are, people are covering this and we make this available and you can go here and look at this. There's, there's yep. Jumping out there to, to what access now had on that. So in Pulse, we do that. You can also search by country. So if you want to see all the shutdowns that were in Egypt, not from, you have to go back to that main screen, but you can see it there. So that's what we're doing in this, in this particular place, or you can click on any of the countries that are in there and you'll see whatever's there. Here we go in Iran and some of the ones that are there. We also, the net loss calculator that you mentioned is a tool that we have, which allows you to calculate what would be the cost of a, of a shutdown on a, on kind of a daily basis in a given country. And so, yeah, if you pick, sure, let's pick Iraq and you want to say, sure, just today, if they were to shut it down, or if it was a couple of days an Internet shutdown, what would the calculation be? So we would, we would say that they would potentially lose 1.4 million us dollars in this kind of thing. So this again is helping characterize it and the methodology for people who really want to dive into it, there's a ton of information on there that you can go and look at. But this is the net loss piece. While you have Pulse up there, I'll mention that there's a resilience tab that has our Internet resilience index and the piece of there. We have country reports for about 170 different countries that are out there. We have a blog that has all of these stories kind of on an ongoing basis around what we're seeing and also technologies, which looks at IPv6, DNSSEC, TLS, and those kinds of things. So that's the Pulse platform. And we, come and visit us, pulse .Internetsite.org. Excellent. And now I just got to figure out how to stop sharing. Oh boy. Oh, zoom. Oh, there we go. So different screen, of course. All right. So we've got about a minute left. How can folks find you online? So, well, I should, I should say that they can find Pulse is, is on, as on X these days as ISOC underscore Pulse. You can also, they're on LinkedIn. There's a monthly newsletter people can subscribe to, and you can find all of that off pulse .Internetsite.org. I'm on, I'm, I'm Dan York on many services. I'm not on Twitter as much anymore. I'm more on Mastodon at mastodon.social. And you can find more info about me at danyork.me or at www .Internetsite.org, which is our main organization website. I should be able to do that by heart and listen to your podcasts. I'm, I'm, I'm David Dolson. You can find me pretty much everywhere at D Dolson, D B E L S O N. And Cloudflare Radar is at radar.Cloudflare.com. So thank you, Dan, for joining me today and for, for doing this. Oh, you're welcome. This is great to be here. And thank you. And thanks for all who are watching.