Radar Bulletin: Q2 2024 Internet Disruptions

Welcome to the second quarter 2024 Internet disruptions overview. My name is David Belson. I am the head of data insight at Cloudflare and I'm joined by Dan York of the Internet Society. Good to see you again, Dan. Great to be back. Thanks, Dave. It's good to be here talking about this super important topic. You're welcome. It's something I think we both follow every day. I think I'd really like to get to the point where we don't have to follow it every day. I was about to say, ideally, this goes away, but I think we're probably always going to have it for disruptions of certain kinds. Although I think we'd like to see one kind of disruption go away. Yes, very much so. And we can we can actually jump right into those. And those would be the government directed Internet disruptions. So in the second quarter, I think there were certainly two big government directed Internet disruptions that we wanted to talk about. The first one, you know, we can kind of get to that point every year where if it's springtime, it must be exam time. And with exams come, unfortunately, some Internet shutdowns in certain countries. Yeah, this is crazy. You know, every year this this happens, particularly in I mean, where we saw it this year was we led with Iraq starting in May 20, May 21st or so. Syria then picked it up around May 26th. And then in early June, Algeria did it as well. You know, and that those three in particular have done it year after year where they go to shut down the Internet to prevent cheating during a century exams at the end of high school. Right. The baccalaureate exams. Yeah. Has there ever been any any study done on whether or not the shutdowns are actually effective in reaching that goal? I don't know. No, it's I I think the good news, I think we saw this year was a growing frustration voiced online, which we've seen in past years, but even more so, you know, people were being able to say, you know, like, wait, it's 2024. What are you doing shutting off the Internet for 40 million people in Syria? You know, what are you doing? How can we grow a digital economy? And that's yeah, I think that's the big issue is. Right. Yeah. Right. You want to have this blast radius that's that's sort of out of out of scale to the scope of the problem. You know, I thought what was interesting as well is digging in. I had an opportunity this year to get a little bit more into how they were being implemented. You know, I think that in Syria, for instance, they, you know, pulled all their routes. Yeah. So there was effectively, you know, what we saw for the most part was that at least UDP traffic, I believe, could get out of the country. We saw that in our DNS logs where we saw spikes of request traffic when the shutdowns were implemented, suggesting that sort of standard DNS or just traffic to our resolver was able to escape the country. But there was no path for it to get back in. Right. Interesting. Yeah. I mean, we just saw from on the Pulse, Pulse .Internet.org is what I work on the site there. We just saw, you know, Syria and Iraq just dropped BGP routes, Border Gateway Protocol for people listening. It means there was no route. There was no way to get into. And BGP is the fabric that makes the Internet work. Whenever you get your domain names, you go and you get your IP addresses. The IP addresses are then routed to the appropriate network and things. And so what was happening here is just, you know, Iraq and Syria just said, OK, done. We're shutting. We're killing it. They closed their door. Basically, nothing could get back in. Yeah. Actually, nothing even got to the door. Stuff couldn't find its way. It's like you put it in the mailbox and the post office went with them and goes, yeah, I have no idea how to get that letter to Dan. Sorry. Right. Yeah. Yeah. Can't get that from here. But this is the thing, right? They did this, those two in particular, and they withdrew from the rest of it. Now, part of this is also because there's a limited number of Internet service providers in those countries and a limited number. So, well, so Syria, certainly. So Syria has the one main provider that contributes as a gateway. Iraq actually has a richer ISP landscape, but still not not thousands, not not thousands and also not a lot of transit routes out of the country, etc. So you wind up with, you know, choke points where the government can basically go and say, OK, you know, if you want to continue operating in this country and I'm sure you do shut it down. So they have that ability to wield that that that that bully pulpit. I don't know what you want to call it. Yeah, no, absolutely. And you've got the thing about. The interesting thing about Iraq, too, is that you've got the country sort of almost split between the Kurdistan region and the rest of the country. So when we see the Iraq shutdowns, we see them in both areas sort of at different times. And you can kind of look at the different ISPs that are being affected that are going offline for those multi -hour periods. And you can kind of you can sort of say, OK, these three or four are based on or are located in the Kurdistan region. You know, these others are located in the other part of the country. And Algeria actually is interesting as well. You know, several years ago, they said, look, like, you know, we we in the past, we've done these these large scale or a shutdown. But we're not going to go that route anymore. And last year, they took a more limited approach. And it appears that they were sort of doing more filtering. And it appears this year that they, I think, pursued a similar a similar approach, at least for some of the data we saw. Where we saw one of the one of the metrics we're going to roll out on radar soon is looks at TCP connection tampering. So looking at TCP connections to our platform and understanding, you know, what are they? What are they sort of what are those connections getting shut down before they would normally get shut down at the completion of a transaction? We saw an increase there that aligned with those those quote unquote shutdown periods in Algeria. The twice daily periods. Yeah. And that was the big difference, because, you know, as we said, that Syria and Iraq just dropped traffic. They shut the doors, boom, you're gone for that period of time, four hours, three hours, whatever it was. And sometimes, you know, it was multiple blocks at different times and just different things. Algeria, though, it was much more content based. And you're right. It was blocking specific sites. It was blocking specific things. The BGP route stayed up. Yes. Technically, there was a route to get into the country. Packets could get in there or get out, but they were blocked. You know, right. It appears that they were doing TCP based filtering. At least that was, you know, what we saw in our in our metrics. Yeah. And we saw that, too. At Pulse, we look at your your data. Obviously, we also look at it from IOTA, from from the Open Observatory of Network Interference, Uni and some others. And they all correlated the same kind of thing. There's there's blockage happening here. The routes are still up, but there's blockage happening on other ways to get in there. And this is costing, you know, these governments, it's costing them millions of dollars, you know, in terms of right. And, you know, in terms of loss to their their their digital economy and everything else. And we and we saw a story in one of the ones I picked up in our post that we specifically put about. Algeria was talking about somebody who had an interview via Zoom with with a foreign company. And suddenly he couldn't he couldn't go. And he could he got an email, you know, but he replied and tried to postpone it, et cetera. But right. Get that out. And it was kind of like, how does that you know, what does it look like for a country when people in the country can't even engage with people outside the world during these blocks of time in these blocks of hours? Right. Right. If I'm looking for somewhere to expand to, that's not a country I'm going to put on the top of my list. Right. And so as these countries go to try to do it, you know, to your point, your question earlier, I don't know if we've seen comprehensive analysis of whether this is actually preventing cheating. Somebody on on social media had said, you know, kids are going to cheat or they're going to try to cheat anyway. Right. It's not, you know, so every year we're hopeful that maybe these governments realize that this is killing their online economy. It's not allowing it's not making them an attractive place to go. We almost saw that with Iraq this year. Right. There was a disagreement among the ministries because. Yes. Yes. The communications and others wanted to not have this happen because they realized they needed to have this for their digital economy. But the Ministry of Education won out. Right. I remember that. I mean, and I mean, to your point there also in terms of, you know, whether or not it's effective. I think one of the things that we can point to certainly is the net loss calculator from the Internet Society to be able to show these these governments, you know, you're shutting down completely. You know, we estimate that that has a cost of, you know, to your economy of X, you know, X number of jobs, X million dollars in GDP. So, yeah. Yeah. You may. Great. You know, these kids. Maybe you're cheating, but you're you're you're really impacting the whole rest of the country in a really tangible way, which is obviously bad. Yeah. I know there's some folks working on research and analysis. I'm looking forward to seeing what they come out of this 2024 season with, because hopefully we can get some more data, some more information that says, you know, this is just all around a bad idea. Right. So we'll have to see. But that's the big and that's going on right now, actually, as we record this show live. Right. Well, actually, it's this morning, this morning, this morning, today, Syria once again shut down their shut down the Internet for several hours again for that. So we're seeing that started. That started last 25th. So late last week. And it's going through. I look at my schedule, August 8th. Yeah. It's got about two week or two week run there. Yeah. It's crazy. The upside is that they publish the schedule. So it's sort of, you know, at least folks like us, we can look at and say, OK, this is what we know we need to expect and when, you know, for better or for worse. But. And yes, I mean, to the degree people in the country could just know that for those, you know, two to four hours or whatever it might be. Well, we're just not going to have the Internet. We're going to have to go do something else or whatever. But right. I should plan to be offline at that point. Yeah. But even then, let's segue into one that was sort of I mean, these are always contentious and there's always sort of disagreements about whether or not that should happen. But one that was. You know, also, I think came with some disagreement, which is the the the outage of the disruption related to the the finance bill 2024 protests in Kenya. I think this one was interesting because we initially the communication, there was some concerns, protests going on, government going to shut down the Internet and the community said, hey, no, you're gone. OK. The next day we saw Internet traffic in Kenya and there were a couple of the major ISPs there, Safari Com and Airtel, who said, oh, there was a submarine cable issue. But then we saw additional impact in other neighboring countries like Burundi, Uganda, Rwanda, Tanzania. So there was there was sort of that's that was unusual. And there was some disagreement about whether it was actually a submarine cable outage or if it was, you know, sort of more government directed. Yeah. And it's really important because in these protests, which were which were around this finance bill, there were these concerns and people and you're absolutely right. They'd gone to to the to the communications authority of Kenya and everything. And it said, you know, please, you know, don't shut this down. And I want to read actually on this, the press release from the communications authority of Kenya. You quoted it in your post. We put it in ours as well. And it said, you know, the communications has received inquiries for the avoidance of doubt. The authority, the authority, the communications authority has no intention whatsoever to shut down Internet traffic or interfere with the quality of activity. Such actions would be a betrayal of the constitution of a whole as a whole, the freedom of expression in particular, and our own ethos. Further, such actions would also sabotage our fast growing digital economy as Internet connectivity supports thousands of livelihoods across the country. That seems like a pretty clear statement of like, hey, we get it. It's important. We're going to leave it alone. So clear that actually on our pulse blog, we wrote a post the next morning congratulating them on keeping the Internet on and everything else. And and that was true through most of that until they published and looked around again and went, oh, no. Yeah. Yeah. Yeah. Later in the day, what happened was sadly, those protests became much more violent. There was a lot of, you know, just chaos happening in Kenya at that time. And the government, you know, whatever they reached for the tools they had and they went and shut down the Internet, you know, in all the different ways they could. So it was it was a very tragic situation happening there. Now, the point, though, that you raise is that the the unanticipated consequences. Right. Because if you go back up that path, if you look at who else was shut down, you know, Burundi, Uganda, Rwanda, Tanzania, you know, they were all also affected by the King's decision to go and shut this down. Right. And we can look at maps and say, well, part of that is because of the fact that, you know, Kenya is the the the provider downstream from Kenya. So they're getting their connectivity coming through there. Right. So we're back to talking about resilience. What happens? That's true. When your connections are all coming through one place. And suddenly the government decides to go and shut this down in that particular place. You're you're right. You're you know, you're you're shut. You're being shut down. You know, I think what it also calls for, and we talk about this a lot as well, is additional transparency to where, you know, the government sort of unequivocally is saying, we're going to not we're not going to do this. We're not going to shut it down. And then we saw something that looked very much like a shutdown the next day. Then you had the local providers saying it was an outage. But but as we were talking about as well, we saw recovery within a matter of hours. That's generally not how submarine cable outages work. You know, you got to go find the cable problem. You've got to bring it up. You've got to get a ship. You've got to get a ship to this. Right. Right. If you have a subsea cable outage, you're you're in days to weeks to months. Right. You know, it's not it doesn't get recovered in just hours. So. Right. Yeah. I think there was pretty round, you know, pretty clear concern that that was not really what happened. But yeah, I mean, so so I think, you know, like we talked about, we need, you know, in addition to resiliency, we need truth and transparency from the involved parties, whether it's the government or the network providers. You know, it helps those of us who, you know, follow this and those of us that are concerned about it, you know, to understand what we're seeing. We know we know what we're seeing. We don't know why we're seeing it. So it really helps to give us that sense of like, OK, this is what happened. And this is how we can associate these sorts of metric shifts with these sorts of events. Right. It's the attribution issue. Right. I mean, yeah. The good or bad news about the exam related cuts is that it's very clear there is 100 percent certainty that it's a government shutdown because they tell you. Right. They say the quiet part out loud. Exactly. They're like, we are going to shut down our country because we don't think the digital economy matters as much as as protecting our exams. OK, well, fine. But they put a schedule out there. But, you know, these other cases, it's super hard sometimes to know. We will never know precisely about the Kenya. I mean, it looks like a government mandated shutdown. It certainly seems like all of that, but it could have been. Yeah, that's the problem is you don't really know. You know, and it could have been people showing up at a cable landing station with a lot of guns and saying, you know, stop this right now for a while. I mean, we don't know. Right. Anyway, and so while we're in Africa and while we're talking about submarine cables, we can segue into some of the cable cuts we saw in the court. So one of them was with the EC and CECOM cables. We both wrote blog posts about that. And those took out Internet connectivity, at least, you know, to some extent for, again, Kenya. But other countries like Madagascar, Malawi, Malawi, Mozambique, Rwanda, Tanzania and Uganda. So, again, those countries that are reliant on connectivity to those cables, you know, again, if they don't have the proper resilience, they're going to see issues. But I think what we're seeing, though, is more countries, I think, experiencing these issues and. It's a building better resilience. We're seeing the impacts of these cable cuts are sort of becoming less drastic. In places where they have this greater resilience. Right. Because if you jump back, you know, a couple of months when we were here talking about the Q1 disruptions. Right. We had the ones in March. Yeah. In March off Cote d'Ivoire and that took out a good part of Western Africa for a period of time. But here on the eastern side of Africa, you know, we saw tremendous drops. You know, I think there was like Tanzania had a I think you'd written about 50 percent, you know, drop. And or we had one of us had written about that, you know, and those kind of things. We saw a big drop, but it didn't totally go away. Right. Right. And that's really the point. There was resilience. There was different kinds of areas. In fact, we saw some like in Kenya. It continued. There's enough other paths into Kenya. Yeah. But Kenya stayed OK in a lot of different parts of what it was, whereas some of the other countries had fewer paths and were much more, you know, cut by that in some way. Right. And I think to underscore the resiliency and actually to go back to the comment we made earlier about, you know, recovery within a matter of hours. And that's not how so many cable outages work. I think if you have sufficient resiliency, like we looked at, you know, looking at the graphs in the blog post, you know, you see that loss of traffic. When the cable cuts happen and then you can see, you know, a more rapid recovery if there's more resilience because you don't have alternate paths to to, you know, to send that traffic through. And I think that's what we're that's what we saw with a number of the countries where the East African countries with these two cable cuts impacted. And then during the month, also, there was an issue with the Africa coast to Europe, the ACE submarine cable. And the ACE cable was problematic. They always used to always be in the news, at least in Q2. It was in the news a bunch where an issue took out ISPs in Gambia, Mauritania and Senegal. And here we saw outages that lasted that were, you know, at least in Gambia, Mauritania. Yeah, I mean, looking at the impact of countries in Senegal, the traffic impacts were significant, where they were almost near outage, sorry, near complete outages, you know, and lasted for, you know, almost looks like almost 12 hours or so before they could get the traffic shifted onto alternate paths. Yeah, and I think that's, that's an important part is, I think part of this, though, is also if you look at what's happening in in Africa, on both the east and west coasts, is a tremendous building spray, more and more subsea cables. So I think the good news coming out of this is that we will see increased, we are seeing increased resilience, we're seeing much more of this, we're seeing Internet exchange points being built up in different locations, where you're getting much more connectivity of networks locally, allowing people to have much better resilience in the face of these kind of cuts, and being able to route to other routes, you know, to find other places, etc. So we're seeing this building of a richer fabric on both sides, and with connectivity going out. And, you know, and just if you look at any of these, you know, the pictures from telegeography, the kind of the canonical folks around submarine cables, they're just building and building and building. Right, right. Those maps are getting busier and busier, which is good. Yeah. Absolutely. We have about eight minutes left. So just to transition to another few causes. So power outages, we talked about this every quarter as well. Generally, they're more localized. You know, we see power outage in a given country or a given area, takes out of your network briefly. But I think one of the ones we saw in Q2 was really interesting, which was an outage that impact ultimately impacted Internet traffic or Internet connectivity in Albania, Bosnia and Montenegro. And this was interesting, because I guess the three countries have an electrical power infrastructure that's linked through what's called the Trans-Balkan Electricity Corridor. So they, I guess, can share their load and, you know, share power or whatnot, which is interesting, but it's one of those things where something goes wrong in one place, it has a much bigger ripple effect across multiple countries. Yeah. You know, it's very interesting that I've been doing a lot of time looking at it in my own personal interest, sort of in climate resilience and adaptation of our existing Internet structure. And so often it comes back to power. It comes back to electricity because, of course, that's what we need. So, you know, I mean, and we saw this just recently, I guess it's Q3, but Hurricane Beryl that came in and hit some of the islands in the Caribbean very hard. We're seeing this with storms, with flooding in parts of the world in different places where we're seeing the electrical infrastructure is taken offline, which therefore means the Internet infrastructure is. And so I think it will be interesting to see as we keep thinking about how do we build that resilience with the Internet infrastructure is what's the corresponding resilience with the power infrastructure. And, you know, and I know the power industry obviously is very strongly looking at that, obviously, too. But hopefully the connection between those two, which is only becoming more and more clear that we need to be thinking about both of those as they go through things. Yeah. So it's one of the things I've been monitoring also. So I have my lists on on the tweet deck of the ISPs. And, you know, so when they post like, hey, we've got an issue here, whatever. But I also follow a number of power companies because those are like the ones where there are countries where the power, you know, they have poor power infrastructure, frequent outages. That's often what we'll see them say, like, hey, we had an issue. Right. Yeah. And we're seeing that like this summer, we were in extreme heat in many different parts of the world. And the demand for air conditioning and cooling, both of residential and businesses, but also data centers, is creating kind of unprecedented strains on the power grids in those areas, which has then caused, you know, outages or brownouts or types of different things that that create challenges for connectivity in general. Right. We have about four minutes left, so I want to just jump to a couple more. One that I thought was particularly interesting was an issue we observed in Nepal during the quarter. So this was two ISPs there, Vianet and Dishome, were apparently after warning cut off by Bharti Airtel, an upstream provider in India, for non-payment of bandwidth bills. So this was a root cause that we don't see really all that often. You know, it wasn't an act of God. It wasn't their power outage. It wasn't a cable cut. This was, hey, pay up. Somebody forgot to pay bills. Yeah. And it was actually, it seemed to be probably, digging into it, it seemed to be fairly interesting where the providers were then, I think, blaming the government because there was some issue with exchanging currency and they didn't get what they needed from the government, the Nepalese government in time. And Bharti had told them, look, your bills are overdue. We reserve the right to cut you off. And they eventually did. And that lasted, I think, for not too long. It was a few hours or maybe a day or so. But it was interesting to see that that was one of the causes we saw. Yeah. No, I think it's, you know, and that's, again, contractual issues, right? And because it's all commercial in this kind of space. And again, we can talk about resilience, right? And multiple paths and things. But yeah, all of those kind of issues. Sorry, go ahead. You were about to say. No, no. And the last one I wanted to touch on was something I know that you follow very closely, which is Starlink. Yep. So Starlink had a really brief disruption in the quarter, during the quarter. It was, I think, about a half hour long. You know, to their credit, they posted about it on X and said, hey, we're in the midst of, what did they say? Starlink is currently in a network outage and we're actively implementing a solution. You know, then shortly thereafter, they said, hey, the network they posted, the network issue has been fully resolved. So the interesting thing, I think, about Starlink outages is, you know, for all the others we've talked about, they've been, you know, either geographically scoped or sort of network scoped, or I guess network scoped from a geographic perspective. But Starlink, when Starlink has an outage, it's got a much broader geographic scope or impact, I should say. Yep. Well, and that's the thing, right? Because if it's truly, well, it partly depends on where, and this is part of one of the challenges. And yes, to your point, I led a project in 2022 around low-Earth orbit satellites. I was looking at that. Yeah, yeah. This is the thing. If you want to read more, Internetsite.org. You can read all you want. If you want me to come on and talk about that, I'll be glad to talk for hours. But the point is, there's a lack of transparency, coming back to your earlier point, in that we just don't know. There's a certain opaqueness to the Starlink system. We know there's a couple of ASNs that are Starlink. We know they have certain connections in different places. But we don't actually really understand the infrastructure, meaning people outside of SpaceX. So when it goes down, yeah, we don't know. But it could have global impacts, depending upon where in their non -transparent infrastructure this happens to go down. So we don't know. Right, right. But it'll be interesting to watch and see. It'll be also interesting to see if any of the other providers, the Kuipers and the OneWebs and others, are any more transparent about their issues when they finally get their constellations up and running and serving traffic. Yeah, yeah. It'll be very interesting as we move into having more and more of these space-based systems. Yeah, yeah. So in the last 45 seconds, where can folks find you or find your work? You can find this work at Pulse, P-U-L-S-E, .Internetsite.org. And I'm Dan York on many social networks and other things. I can almost recite your podcast, Closer, by heart. Yes, you can find me on different things. Thank you for having me on again, David. This is always fun. You're welcome. And we're at Radar.Cloudflare.com. And we are at Cloudflare Radar on most of the popular social media sites. So, again, thanks again, Dan. It was great to see you again. Great to talk with you again about this. And we will connect again next quarter. Sounds great. Looking forward to it.