This Week in Net: One year of war in Ukraine, Fediverse on Cloudflare and Chief Zero Trust Officer

Hello everyone and welcome to This Week in Net. It's the February 24th, 2023 edition and one year ago the full-scale war in Ukraine was beginning and we'll discuss on today's program about some of the Internet trends related to this historical event. I'm João Tomé, based in Lisbon. Actually, I'm in the Lisbon office and so are you, John, our CTO, John Graham-Cumming. Hello, John. It's nice to be very close to you in the Lisbon office. Exactly. We could be talking face-to-face almost, but yes, here we are. Here we are. Of course, this way we can share the screen with everyone and also show some of the blog posts we've been writing. We haven't done this for two weeks now, so a few topics to discuss, but because today is one year that the full -scale war in Ukraine started, why not start there? Because we published yesterday a blog post about some trends and attacks and topics in terms of resilience in the Internet in Ukraine that we saw, right? Yeah, well, you know more about this topic than I do because you're one of the authors on that blog post. But yes, I mean, it looks at everything we saw, I think, from the very beginning of the war to essentially to today. And I mean, perhaps the big story, you kind of point this out, is how amazingly resilient the Internet has been in Ukraine in general. And I think that's linked to a few different reasons. One is that there's a very diverse number of ISPs in Ukraine, so you're not concentrating on a small number of ISPs who can be damaged and therefore take people offline. Ukraine relies on a huge number of cross-border physical connections that are not undersea. So there's actually a lot of connectivity through Poland and through Hungary and through Romania and through Moldova, etc. And so the connectivity is pretty good. And then there's just been a tremendous amount of heroic work by people in Ukraine to restore damaged infrastructure. So, you know, one of the biggest reasons we see Internet outages in Ukraine is because of power outages, right? That was particularly happening after October of last year, right, where the Russians were targeting power infrastructure. So you see that cause problems, but you also see fiber optics getting damaged. And so people going in and repairing it. And we saw an increased use of Starlink in Ukraine as well, giving you, you know, sort of, in some sense, at least ground infrastructure-less Internet access. So I think the resilience stuff has been pretty amazing, right? We've really seen the Internet be used by the Ukrainians continuously, pretty much. Obviously with outages in different areas, worse in the east, where there have been, you know, so much fighting and so much of the land is occupied by the Russians at the moment. And certainly throughout the country when there's power outages. We have this map, it's from the first days of the war, but that shows well, the change Internet volume change from the east, where the most of the fighting was happening in the south, to the west. So the green areas here in the map show us Internet traffic growing. And in the more red or areas, darker areas show a decrease in traffic that went as low as a minor drop in 60% of traffic in that first week of the war. So definitely related to people fleeing, leaving. This was, I think, the interesting thing about this particular map is this really reflects the people moving westwards, away from where the principal fighting was. Of course, there were missile attacks across the entire country. But you can see around the Polish border in particular, you know, where people were massing to cross the Polish border and other borders into Romania as well. You know, you see that too, but you can see it's really striking. A component of this, of course, is outages caused by the actual fighting. But the biggest component is the migration of people westwards. Exactly. We have a bunch of charts related to some of the outages, some from the early stages of the conflict, even Boucher also referenced here. So people left, leaving meant less traffic. But at some point, when there's an occupied territory, sometimes traffic, sometimes it's residual because there's no Internet there, no energy. We also saw in Mariupol in the south, something similar. And then we tried to divide this in areas. One is the first days of the war, first weeks. The other one is Ukraine's counter offensive that started late April, but mostly was in September. We also have some trends related to the nuclear power plant in Zaporizhzhia that we saw some peculiar outages there when inspectors... We saw outages. Yeah. Yeah. We don't really know if they're connected, but they seem to happen around the same time. Yeah. There was quite large Internet outages that happened around that time. I want to stop on the Crimea chart for a minute. This is kind of interesting. You have that little map there, so you just scroll past it. So this is showing a decrease in Internet use in Crimea. And the assumption here is that this was Russians leaving Crimea as the war got closer to them. And you see a significant drop there in Crimea. And what was the date on this one? September. It was mid-September because after the counter offensive from Ukraine that got to Kherson, but also other parts of the East Kharkiv, there was some news that Ukrainians could be going to Crimea next. So there was some news on that. So after that, there was some decrease in traffic in Crimea and some news reports of people leaving, families leaving. And there was of course the bridge that was attacked as well, right? So that was obviously adding to that. Exactly. So I had to get out in October. That's when most of the energy infrastructure outages that also brought Internet traffic down or outages in the Internet had an impact. Yeah. And you see this here, like in this Lviv chart here, which is very far in the West of Ukraine. And you can see outages happening. And these are primarily power outages, in fact, causing Internet disruption. So yeah. Exactly. And after October the 10th, this was regular. Every few days, some city had a real world impact in terms of Internet traffic related to energy infrastructure impact. And we saw this through December in different areas. And also even in January, most recently in January. So a big impact in several different regions, Odessa also. And also ASNs, of course. Right. Specific networks as well. Yeah. There's a graph later on in here, right? Where you show the rerouting of the network into Russia, which was quite interesting because one of the things that's going on is that within the occupied parts of of Ukraine. So in the Donbas. So this is Kursyn. Yeah. Yeah. It's Kursyn, right? So this is interesting, right? Well, you saw this on April the 30th, the connectivity between Kursyn telecoms, a local provider, and the rest of the Internet went through to Ukrainian networks. Remember, the Internet is a network of networks. So stuff hops between networks. And then, essentially, a day later, it had been rerouted through a Russian network. And actually, I think you have a lovely chart lower down where you show, you know, within Kursyn Oblast, which is like all of the networks. And you see this. So if you look at the so the things to look at on this chart is the black is the Internet isn't working. The red is going the Internet is being routed through Russian networks. And then the white is working normally. You see outage, the vertical black line there, about April 30th, I believe. And then you see this incredible thing where all of the local networks were running through Russia, essentially, for some time. And then, you know, in October, well, I think the story is interesting. It's a combination of two things, right? One of it is, it's no longer routing through Russia, partly because it's not going nowhere, the Internet is damaged. And also, it's back on going through Ukrainian networks. And so you can see on the Internet routing, essentially, the sort of the fighting and the occupation at that point. We've seen this in a number of spots. And there's another thing here, which shows overall the story, right, with the maps of the overall story about how Ukraine is connected to the rest of the Internet. And I think one of the things that's interesting is there's a chart down in here, where you can see that mostly, keep going, it's a little bit further down, there's a table. If you go down to the table, there's a table showing the next hop, i.e. when you're going from Ukraine to the next network you're going to, where are you going? And prior to the war starting, the top, like next hop countries, whether it was Germany, Netherlands, UK, Hong Kong, interesting enough, Sweden, Romania, right? So then Russia was fairly far down the list. Now, you see that Russia is actually towards the top of the list. And this is really an indication of the fact that you have occupied parts of Ukraine where traffic has been diverted into Russia. There's a little bit of switching around in other ways. This is connectivity issues or connectivity changes. Poland has increased a little bit from before. But I think the really interesting change here is Russia having grown enormously in terms of the share of next hops. And I think that's totally around the occupied areas. Exactly. And there's something, an interesting element here that is related to the importance in terms of war, of redirecting the Internet in occupied areas through Russia. For those who don't know what this means is that the limitations of Russian Internet with all of the blockage that is done in terms of sites, for example, Facebook, Twitter, and even news organizations in Ukraine or Western media like BBC are all blocked in Russia. So what does this mean is there's a control in the Internet that people have available because the Internet is rerouted to Russia. So it's the Russian Internet. That's why it's being done. Yep. Absolutely. From a filtering perspective. Exactly. Should we talk about something else? Sure. People want to know this enormous amount of detail. I mean, Joao, David, and Kristen did a huge amount of work putting this together, looking at the year and looking at what's happened, recommended to go read about it. But let's talk about some other Cloudflare. Just one last chart. This one shows mitigated attacks towards Ukraine. So a complete increase after the war started. So just want to share that. Yep. Cyber attacks did accompany, definitely accompanied the start of the war and have continued to be part of it. Exactly. Where should we go next? Actually, from last week, we have our Fediverse Cloudflare. Let's talk about Wildebeest. Here it is. So what's Wildebeest? So some people may know there's an alternative, if you put it that way, to Twitter called Mastodon. Mastodon is different from Twitter in the sense that Twitter is very centralized. There's one Twitter. You make an account there. Twitter controls the content you put out. They control whether you can have an account. The thing with Fediverse is a similar concept. They toot rather than tweet to put things out, but you can send out your little statuses, maybe with images and stuff like that. But it is done in a federated way, which means that there are multiple players who have servers and they all communicate with each other to share the toots, depending on who you follow and who you don't follow. It's open source and you can set up the official server on your own machine if you want to. We decided to build our own from scratch implementation of Mastodon, so compatible with the APIs, compatible with the open source API called ActivityPub, but built on the Cloudflare Workers platform. And really, this was partly to give people another way of getting on the Fediverse, but also to demonstrate the complexity of an application you could build on Cloudflare. So we use Cloudflare Workers. We use the D1 database. We use Cloudflare queues for queuing. We use Cloudflare images for image hosting. We use Cloudflare Zero Trust for access control, and it's a complete... So even if you're not interested in the Fediverse and Mastodon, if you wanted to know what it's like to build an application on the Cloudflare platform... Oh, of course, Cloudflare Pages. Actually, the whole site is on Cloudflare Pages. This will give you a really good idea. So this post by Celso and Sven digs deep into how it was built. And obviously, you can go get the open source project from GitHub, study it, implement it, and roll it out and build your own Fediverse server in this Fediverse using this. So yeah, one of those technical posts that really talks about how to do things on Cloudflare. I really think this is interesting in a way that it shows all of the products that one can use from Cloudflare, that one can use to build something from scratch, only using Cloudflare, which is amazing. It also shows the full ecosystem at play. Yes, it really does. And it shows that the workers platform, you can build a complex application on it, and this is how you might do it. And I think by showing the D1 database, by showing the queuing functionality images, Zero Trust, in a way, there was so much that the Cloudflare platform gave them so they could build their specific application on top of it. It was pretty damn cool. So yeah, if you're interested in Mastodon, you can set up a wildebeest implement on Cloudflare and run it there. And if you're just interested in how to build on Cloudflare, well, this is a great sort of sample project because it's not a toy. It's the real full thing. Exactly. It's a very good example, for sure. Where to go next? We also had a a record-breaking DDoS attack mitigated by us. We did 71 million requests per second, which is pretty wild. If you think about what Cloudflare is doing in terms of millions of requests per second for normal traffic, which is I think on average about 40 million requests per second, this was larger than what Cloudflare does total globally, almost twice as large. And yeah, attacking websites hosted by Cloudflare came from someone's got a botnet. They went after a variety of things, cryptocurrency, various other things they wanted to attack. But it was more the scale of it. And we know from talking to other providers that they turned this canon on some other folks and other folks had to scramble to deal with it. But yeah, that was the largest we've seen in terms of requests per second. Not the largest in terms of bits per second, which I think is around 2.1 terabits per second. But whether it's bits per second or requests per second or packets per second, these things can knock you offline. And it shows also the trend that DDoS attacks are not recent, but they're increasing and they're around. So a lot of people are using those to do harm in a sense. It's interesting, actually, because I think at some point there was this escalation in DDoS attack sizes. Everyone was reporting a larger and larger DDoS, and it seemed to plateau for a while. It was like they were still there, but it wasn't in the news because it wasn't the biggest or twice as big as the last one or whatever. We're still getting these massive attacks. And I think within the last year or so, we've seen record -breaking attacks again. And I think it's partly due to a trend away from using people's homes connections. If you remember, originally, a lot of these botnets where your computer at home got hacked, or even your phone was participating in a botnet. And lately, we've started to see hosting providers. So the big cloud providers and other hosting providers get... People are hacking the servers themselves, not the infrastructure, but actually, we know that 70,000 is vulnerable software on a hosting provider. We'll hack into that and we'll use the hosting provider. And those providers are well-connected. And so they tend to be these quite large attacks. On the other hand, they can get shut down quite quickly. Obviously, Cloudflare was blocking this attack, but we also were in contact with the hosting providers who were then able to shut it down on their side, which is in some ways easier to do than with ISPs. But I think we do seem to be in a little bit of a trend right now with larger and larger attacks. We also have a Zero Trust, Chief Zero Trust Officer blog post related. Do you want to go there? Yeah, I want to go there because I want to tell a story about this. So this was an idea that a chap called John Engates, who's CTO of Cloudflare, and Trey Guinn, who's also a field CTO. They both work for me. And they were doing a post or some thinking about trends for 2023. And they came up with a bunch of ideas about what's going to be predictions. And one of them was this Chief Zero Trust Officer. And I said, I hate this idea. This is the stupidest idea ever. No one's going to have a Chief Zero Trust Officer. And they didn't listen to me. That's great. I'm glad they didn't listen to me about this. And this, it turns out, has really resonated with people in large companies. I think the reason it's resonating is that there is a real desire to move to Zero Trust functionality. But there is also a sense that it is such an all-encompassing thing that a company needs to do over time, and that you need someone driving it. And they're like, is that the CIO, or the CTO, or the CSO, or whatever. And even if this role doesn't exist, the concept that there is some Chief Zero Trust Officer who is really seeing this through has become very important. So this was John's write-up about what this role really means and the process of how you get to Zero Trust. So well worth reading, I think, if you're in a C-level or SVP position in a company that's thinking about Zero Trust. Because even if you don't have someone with that title, this is really talking about what adoption looks like and how you do it. And maybe there will be a Chief Zero Trust Officer. And maybe I'll look even more stupid for saying it was a dumb idea. Yeah. But in a sense, it just highlights the need to have that in mind. Organizations have that in mind. Because it's getting more problematic out there in terms of privacy, of safety. And there's also the Zero Trust roadmap. There is the roadmap, which is well worth looking at, because it's like how people can get through the roadmap. So yeah. Yeah. There's a full website about that. In a way, it's just some highlights of those who aren't in Zero Trust, what they should consider, why they should consider Zero Trust in terms of architecture. It has a bunch of features there. It's ZeroTrustroadmap.org. So definitely an interesting thing. Well worth it. Yeah. Today, we also launched a more technical blog post. Want to go there? Go on. What do you think? Let's ruffle and lull about it. That's a good title right there. So this particular blog post, when I saw it, it was very personal to me. Because the module that was rewritten here in Rust is a thing called CFHTML. And CFHTML was at the heart of the CloudBleed security problem we had six years ago. And it's one of those pieces of software that we did work on, but because of the history of CloudBleed and because it was C code, it was an area where we were sensitive about making changes. And we actually were hypersensitive around any errors occurring with this piece of software. And in fact, in general, with any piece of software on our network. Because when CloudBleed happened, one of the things that, looking back, we said, could we have spotted this? It was a little bit of like, well, we knew that this module was segfaulting. And that was sort of a... So one of the things we did was we began a move in lots of places towards memory-safe languages, hence Rust. We also accelerated our move off of Nginx. Not that Nginx was a default in CloudBleed, but there were lots of reasons. There are lots of blog posts on the Cloudflare blog about how we've moved off of Nginx. But we moved this critical module to Rust so that we can eventually, as we get off of Nginx, it will become part of the next thing. And we're so sensitive to crashes that one of the things we do is, or rather I do, is I get emailed any time a process segfaults on our network anywhere in the world. Because one of the things was obviously the teams are looking at it, but I kind of made myself the backstop for this. If a team started ignoring this signal of a problem, I would come in and I do. They know that I'll turn up and bother them about it. So we can really get down so that we know that our software, which is absolutely critical. I mean, Sam makes a point in here. We do an average 40 million requests per second. An issue that affected 0.001% of requests still has a huge impact. So really getting stability, performance, memory safety, all this stuff. So this is a post. It's a post a little bit about CFHTML. It's a post a little bit about Rust. It's a post a little bit about interfacing Rust to Nginx. And ultimately it's a post about how we are moving some of our last Nginx dependent components into new things. There's another post coming about a proxy called Oxy, which is again, written in Rust, which is part of our infrastructure now. And this is all about moving these things away from the Nginx, open, rusty, Lua world that we used to live in. Any of the news regarding Internet in the past two weeks, you want to highlight some interesting thing you're looking more attentively towards? Well, who knows what 2023 will be? If you go back four years and start doing yearly predictions. And one of the reasons why I was skeptical about John Engates and his yearly predictions and the Chief Zero Trust Officer is, if you think about what 2020 brought and then what 2021 brought and then 2022 and 2023. I mean, I'm like, it's a little bit hard for us to predict. I mean, what we know is the Internet is still growing very rapidly. Threats are still increasing for large and small organizations. I think the basic things that Cloudflare do are more and more important to people. The Zero Trust is real and here are people using it. And frankly, if you think about the wildebeest thing, the Cloudflare Workers platform really has the extensive set of you can build significant applications on it. So that's all happening. I also think 2023 and beyond are going to be around individual privacy. If you think about countries around the world are legislating their citizens' privacy. And so a little bit before, I can't remember if we covered this in a previous This Week in Net, but we introduced GeoKey Manager 2, when GeoKey Manager allows us to manage where customer keys are stored so that you can deal with the decryption requirements based on privacy, based on countries. And we also updated our data localization suite, which is again around privacy. So there's a lot of, I think there's a lot of stuff going to happen around privacy in 2023, but I'll try not to predict anything else. We still have time for a cultural suggestion type of part. Do you want to highlight something or should I? Well, you were just telling me about something you wanted to talk about. So the National Geographic, you want to do that? Yeah. So it's a magazine related to the exploration era of the 15th and 16th centuries. And for me, what excited me about that is it's about science, science and technology use at that time to go further in terms of the sea. So exploring other parts of the world by sea using science and technology. That's a topic I really like. So technology and science 500 years ago, 1000 years ago, there's really amazing examples before the Industrial Revolution. And that magazine explores a little bit of that, which I found really interesting. Well, give us an example. What's an example of a technology pre-Industrial Revolution technology? Oh, it's mostly, for example, the way explorers by sea would navigate the waters would be, of course, the stars. But they had real world instruments that we have some instruments here in Portugal, built here in Portugal to explore the seas that gave them an exact way of navigating an open sea. There are several tools. And for me, one of the engaging things is how those tools were not created by Portuguese per se. They were created by European and Arabic and other types of technology from other cultures. So a few cultures got together and a new tool was created. And this was like 600 years ago. So that's really incredible to see technology at work at that time. Absolutely. Yeah, it's very interesting, isn't it? And also everything that was done around clocks for navigation, which was an incredible effort to make accurate clocks so you could navigate based on the longitude of the Earth. You're also right about the influence of the Islamic world, all of the star sightings and all of the calculations that were done by the Islamic scholars and how that influences. Yeah, sounds fascinating. You're going to have to lend me the magazine. I will. It's in Portuguese, though, but your Portuguese is already getting there. Let's speak Portuguese. One of these days, we'll do an episode in Portuguese. Okay, that's our time now. And thank you. That's a wrap. See you next week.