This Week In Net

Well, good afternoon from Lisbon. This is John Graham-Cumming, Cloudflare's CTO, and this is a show we're calling This Week in Net. And the idea is to talk about things that have been happening on the Internet in the last week. As this is the first week, I'm going to extend a little bit beyond just a week, but we'll look back at some of the things that have happened, some of the data, and in particular, I'm hoping that people will write in to the email address with questions they'd like answered. We'll be doing this weekly, so hopefully I can get back to reader questions over the weeks and look into data, trends, things you'd like to know about what's happening in the Internet. So one of the big things that's happened for the net over the last week or so is the launch of some more of SpaceX's Starlink satellites. So on the 4th of June, 60 satellites were launched, and tomorrow another 58 will be launched. They're not launching 60 tomorrow because they're actually taking some passengers along with them, a couple of other satellites that are being launched on the same rocket, but 58 will go up tomorrow. And that brings a total of 480 in orbit. These satellites are providing network access via low Earth orbit satellite. And one of the interesting things about this is that satellite Internet access sounds like something that might be really slow, and in particular, might have really high latency. And in the case of Starlink, that's not going to be the case because the satellites are fairly low, around 500 kilometers above the Earth, which if you think about it is really a short distance if you were to drive it. And so that gives the speed of light, not very far to go, and so you get high speed Internet access. Of course, they'll need to build up a constellation of thousands of satellites to cover the entire globe, but for the moment, they're about just under 500 in orbit, and they do seem to actually be doing something. The other day, after the launch of those satellites, the 60, you often get a chance to see them before they spread out into a constellation. And this is a photo that I took as they flew over Lisbon the other evening, actually two days after launch, and you can kind of see this train of satellites in the sky, just a standard snap with an iPhone, nothing special on a very dark night. And these satellites are spreading out and eventually will talk to each other using lasers to build a mesh of interconnected satellites all over the world. For the moment, they are talking only to ground stations, and there's only a limited amount of traffic on them. But I was curious to see how much traffic was actually going on on Starlink. Now, Starlink itself is assigned an ASN, that's an autonomous system number, a network number in the Internet of 14593, and is actually connected in the Seattle Internet Exchange. So we can see the connectivity between the network and the rest of the Internet. Remember that the Internet itself is a network of networks, and each of those networks is given a number. Starlink says 14593, CloudBlaze is 13335, and so on. And these networks all connect together to form the Internet that we use today. Well, the answer is, how much traffic do we get? Four and a half kilobits per second over the last week, so a lot less than even an early dial-up modem. But that's because Starlink is really in the early stages, and this is undoubtedly people just testing out things, checking that the actual satellites actually really work. But there is some real traffic coming through the Seattle Internet Exchange to Cloudflare from Starlink, so some of it's working. If you dig into the architecture, you'll see that the Starlink network is connected to a couple of other networks, HE.net and ZAO, and then out onto the rest of the Internet. Each of these circles here is one of the networks on the Internet, and this is how they are paired together to connect. So these are the hops through which you would take to get from a satellite to the ground and then to a location on the Internet. Now, the way in which the Internet operates is through a thing called BGP, which is Border Gateway Protocol. Why border? Because we're talking about the border of any network that forms the Internet as a whole. So each little network, let's say Cloudflare, which is fairly large, or maybe in a small company, or Starlink, which is relatively small right now, or Microsoft, they're all autonomous from each other, hence autonomous system. And they must talk to each other and say, hey, I'm over here. And in particular, what they're saying is, hey, I'm responsible for this group of IP addresses. This is how you reach me. And when you go to something like Google on the Internet, you're having to hop from system to system to get to that final IP address where it is really hosted. And that's what BGP does. Now, one of the interesting things about BGP is that it doesn't have a lot of security in it, and you may well have heard about things like BGP hijacks. That's where one network says, hey, I'm responsible for these IP addresses, even when it isn't. And that's actually surprisingly common. This was the most recent BGP hijack that I looked up on BGP stream, which actually gives you updates on what looks like hijacking. And it was what looked like a set of IP addresses, in this case, 23.43.170, the 256 addresses, which normally would be announced by Akamai, were suddenly being announced by an ISP in Hong Kong, which actually is a configuration problem for that ISP. And the problem is, this can happen actually fairly easily, this kind of BGP hijacking. And it happens a lot. If you go back through BGP stream, which you can do on Twitter, and look for the HJ or hijacks, you'll see these happen quite regularly. Some of them are really quite big, and affect large parts of the Internet. And you see this occasionally, if you look up information about a route hijack, or a route leak. But why can that even happen? That seems a bit strange when we rely on the Internet for banking, for private communication, etc. Well, here's the reason. In the original documents describing BGP, this is the entire section on security. It literally says, security considerations, we don't talk about them in this document. The entire system is based on trust between the different networks, and is based on a community of people who know how to hook up the Internet. And unfortunately, the Internet has scaled well beyond the trust being the right way to do things. And so what's happened is, you now have to have something else which can replace trust. And the answer to that, of course, is cryptography. Cryptography is how we ensure that we are getting to the right website when we go to it, where it's really our bank, and we can securely communicate with that bank. And at the lower level, at this network level with BGP, we need the same thing. Are we really contacting Google when we're contacting their network, apparently? Are we really getting to Cloudflare? Are we really getting to Microsoft, for example? And so BGP itself needs a layer of security. And there are very many different ways to do this. There are RFCs that describe different security considerations for BGP, and different ways to improve the security, the most important of which is a thing called RPKI. Now, with 800,000 routes on the Internet, so if you think about it, there are 800,000 people, companies saying, hey, I'm responsible for this block of IPs, or I'm responsible for this block of IPs. Doing anything with a trust-based system is just impossible at this point, hence the need for a protocol. RPKI, the PKI here is public key infrastructure, and R is the routing, applies really a bit what was applied for using HTTPS on the web. So how does HTTPS on the web work? Well, if I claim to be Cloudflare.com, I create a private key and a public key. I publish the public key, that's what we call a certificate, and you say, hey, I'm Cloudflare.com. When your browser visits Cloudflare.com, it can validate that. And it does that actually by sending you a piece of essentially random information saying, prove that you're Cloudflare.com by encrypting this with Cloudflare's private key. And because you've got Cloudflare's public key, because that's the certificate, you can decrypt that and say, okay, whoever I'm talking to actually controls the private key for Cloudflare, therefore, it's Cloudflare.com's website, therefore, I can go there. Same thing for the bank. Now, in order to be able to do that, you need a hierarchy of trust because you need someone to trust so that you can say, hey, well, these guys have said they're Cloudflare, can you prove that they have the right to have that certificate? And that's what the public key infrastructure is all about. And that's done through what are called certificate authorities. But RPKI does something very, very similar for routing on the Internet to make sure that if somebody says they're Starlink, they really are Starlink, and you route the packets to their satellite and not to some random ISP somewhere. The idea here is that when you publish a route, you publish a thing called an ROA, which says, this is my route, I own this block of addresses, and I can prove it. And I prove it in a similar way to certificate authorities and HTTPS through signing the route itself. And then someone who's receiving the route, and this is the key thing in BGP is that network of networks, our networks are all communicating with each other. They all need to keep this big table called the routing table, which says how to get from one place to another. So when a packet arrives, they know how to forward it on. So that routing table needs to be validated, and it can be done using RPKI. It allows you to say, okay, this person over here is now claiming to be let's say, bank.com. And they've published a route. But actually, the real bank.com has signed its routes. And I can't validate this other route over here. And because I can't validate it, we call it invalid in the sense, I'm going to ignore it. These people aren't really bank.com. So that's what RPKI does for the network. It doesn't fundamentally change BGP itself, that still operates in the same way. But there's a way of validating whether someone is who they say they are from a network perspective. Now, one of the problems is RPKI is not widely used on the Internet yet. So a couple of months ago, Cloudflare created a website called isbgpsafeyet.com, which explains this entire situation, and also talks about who is either signing their routes, or invalidating the routes to make sure that both what they're saying is valid, and also that they are only accepting routing information from people who can prove they are who they say they are. You can go to isbgpsafeyet.com and check your own ISP, see if your own ISP does the right thing. Unfortunately, mine here in Portugal doesn't and does not check RPKI, which means that if there are a route link, it is possible that they would accept a route from somebody, and that that would cause them to send traffic to the wrong place. So isbgpsafeyet.com will allow you to check that, and you can also tell your ISP what you think, and hopefully ask them to start using RPKI. Now, many people have objected to RPKI because there wasn't available software for it. To help solve that problem, Cloudflare built open source tools and released them to make RPKI validation very easy, and that is now widely used, and that is available again through this website. There's been some recent news. Since we launched this, many networks have started to take RPKI more seriously, as you see that other networks are doing the same thing. So June the 1st, Mobicon, which is a big transit provider in Mongolia, deployed RPKI. So they are safe against a route leak, and they themselves can't be route leaked by others who handle RPKI. And on June the 5th, Cogent, which is the third biggest transit provider, that is the third biggest network that transports Internet traffic from place to place globally, now filters all RPKI invalids, which means that if there's a route leak affecting an RPKI network that publishes its own signed routes, then Cogent won't accept it. And this is super important because getting the really large transit providers to do this means that the network becomes more and more secure overall, the entire Internet. But ultimately, we want everybody to be involved because then we can completely stamp out the scourge of route leaks, which have caused problems in the past. If you look at the site, you can actually see a list of the major networks and what their current state is. So you'll see that Cogent is now up there, it's completely safe. They sign their routes, they filter invalid routes. You can see similar things, Telia, GTT, NTT, major networks are doing the same thing. Cloudflare obviously isn't there, you see Wikimedia is doing the same thing. And we're hearing that some of the networks here that are marked as partially safe are actively working on getting completely safe. Once they do, the Internet will be a much safer place and route leaks will be a thing of the past. If you look at the global scope of this, about 20% of routes on the Internet are currently signed. So only one in five. So despite the fact that major networks are involved, there's a lot of work to do. If you go back to those 800 ,000 routes, there's a lot of routes that still need signing. But we're super hopeful that is BGP safe yet or not, well, dot com will make a big difference. Other news this week was six years of Project Galileo. Project Galileo is a project run by Cloudflare to give our highest levels of service to endangered organizations that are in the public interest. So you might think of advocacy groups, journalism in countries where they might be in danger, groups that are talking about public issues, such as LGBTQ groups in countries where they might be illegal, anyone who is in danger on the Internet. What we see is that these groups frequently are targets of DDoS attacks and hacking attacks, because there are people who don't want them to be online. So six years ago, Cloudflare created Project Galileo to help give our service away to those groups. And we handle a very large number of groups, most of whose names are not public, because they are endangered organizations. But this being six years, we published a few statistics about the state of things. So Galileo is six years in, you can see that we handle a huge number of attacks, and attacks are up this year, unsurprisingly. If you look at the graph in the bottom right hand corner, it's quite interesting to see that during the period of COVID, there's been a big increase in attacks on journalism and media sites, as those sites are giving up to date information about the pandemic as it happens. But you can also see the types of things we are protecting. So journalism is very important, community and welfare, human rights, health, all these kinds of organizations. And, you know, we've seen something like a 30% increase in attacks this year versus last year. Now, if it's on the web, it gets attacked. And we have blocked about 2.4 billion HTTP requests that were malicious. So that means they were either trying to break into a website, or they were part of a Layer 7 DDoS attack trying to knock the website offline so people couldn't look at it, from just the Project Galileo participants. The graph on the bottom left gives you an idea of what happens, which is sometimes we'll have a small number of attacks, and then a particular target comes into focus of attackers, and they'll get hundreds of millions of attacks per day, quite frequently, trying to keep them, you know, when there's a news event happens, or if someone dislikes what's been said, particularly for things like bloggers doing journalism in countries. Just focusing in on the Project Galileo stuff for journalism and media, we've seen an almost 7x increase in attacks as the pandemic really got going. And those attacks have stayed up there at around 3x, which is, you know, I think quite surprising that people would go after those websites to try and knock them offline. But I'm assuming that some people don't like the messages being put out. Another thing we saw was a huge increase since the murder of George Floyd in the last few weeks. So since May 25, there are various sites that use Project Galileo that are dealing with racism and advocating for anti-racism policies. And most of the time they don't get attacked. If you look going back at the beginning of May, there were essentially no attacks on these websites. And then after May 25, particularly around, you know, the weekend after his murder, a huge number going from essentially zero to hundreds of millions of attacks, trying to knock them offline. In this case, mostly DDoS attacks at the HTTP level, try and stop those websites from working at all so people could get information. I got a question from someone outside of Cloudflare, and I would love to hear from others for next week. And I'll give you the email address in a minute to send those things. Saying, you know, do I know anyone who works at Cloudflare, which is an easy way to get ahold of us because we all read Twitter very avidly. I'm hoping to get in contact with someone there who'd be willing to share with me any stats that might have what percentage of traffic today is encrypted. So they're trying to get a realistic picture of how much of the Internet traffic is actually encrypted and how much is not. Now, as you may know, many years ago, Cloudflare launched a thing called Universal SSL, and overnight doubled the size of the encrypted web. And there's been a big push by companies like Google to really prioritize encrypted traffic. So within the search engine, encrypted traffic got higher placement if the website was encrypted. Within Google Chrome, if you're using it, it's now much clearer. And in fact, they switch from saying this is secure to really warning you if something was not secure. So there's been a big push to say, okay, how much, you know, we need everybody to be using encrypted traffic on the Internet. We all want to use a secure Internet. It's become very important lately with all the communication we've been doing with all of the work from home policies are in place with children at home doing schooling. So here's somebody who asked me, well, what percentage is it? Well, I can give you an answer to that. So this is this year from the beginning of the year, on average, on average day, 82.6, let's call it 83% of bytes on the Internet are encrypted. And so something a little bit under a fifth are still unencrypted. And there are a few reasons for that. Perhaps surprisingly, quite a lot of attack traffic comes unencrypted, because attackers use very, very simple tools quite often. And there's also quite a lot of traffic, which is folks going after the HTTP version of a website and immediately getting redirected to the secure version. So you still see that, but you still see a lot of unencrypted traffic. And it really hasn't varied a lot. There's been a very slight uptick in the amount that's encrypted as the pandemic has gone on, but not a fundamentally big change. And it's been like this for most of the year, somewhere about 82. The low was about 80%. The high was about 84% on a particular day, not a big change. If you want to talk to me about data like this, I'd love to get email. So it's livestudio at Cloudflare .tv. I won't be able to do the calculations today live for you. But next week, I can get back to questions that we think we have answers to. And I'd love to hear from you. One thing I thought was interesting is if you look at that graph, you might've noticed there were a little wave like this. And actually, the use of encrypted traffic actually dropped slightly at the weekend, which is rather interesting. So I'm assuming that most things people do at work for business purposes are currently using encryption. And there must be some things they're doing at the weekend, some types of websites, which has not yet started using HTTPS. So if you are using something at the and tell them how easy and free it is with Cloudflare, with Let's Encrypt, with other services to put encryption on the website, because we definitely see a dip at the weekends. And this has been a pattern for a long time. Talking about Internet patterns, I thought it's worth talking a little bit about what's happened during the pandemic and how things have changed a little bit. So this is some traffic as seen in the Amsterdam Internet Exchange, which is a huge Internet exchange by time of day. And this pattern pre the pandemic was very, very common. You can see nighttime is where the Internet dips down. And you can see people kind of waking up and getting going. And there's a peak in the early evening. And that's very typical around the world. Of course, when early evening is changes, but it's the same kind of pattern. So people wake up, they start doing stuff. They go to work, they go to school, it continues and continues. And then they come home in the evening, and it peaks there. So it must be when they come home, they do all the things they couldn't do at work. They listen to music, they watch movies, they use the Internet. And then as the evening goes on, they go to bed. So that was pre pandemic. Now, things changed a bit with the pandemic. So many people were sent home to work for schooling, that the patterns actually altered. So this is Milan, this is Milan, actually today I took this this today. So I'm taking Milan as an example. But this pattern is fairly common. If you look on the left hand side, you can see the nighttime, you can see how it dipped all the way down. And you can see it start to come up. And then you see a little lump in the morning. And you see a little lump after, after lunch, and another one in the evening. So the evening peak is still there. But two other peaks have appeared. And we've seen this across the world. There's a morning peak, people are working from home, they're trying to figure out what they're going to do, their kids are getting online, doing schooling from home, there's a lot more use of the Internet. And then there's sort of a middle of the day lunchtime. And then again, I think the same thing happens, much more to do kids schooling, the afternoons are longer, much more to do, quietens down, and then the recreational period in the early evening. So this fundamentally changed the shape of Internet traffic around the world. But you also see shapes change for a lot of other reasons. So in the UK, for 10 weeks, there was a habit on a Thursday night at 8pm of going outside and clapping to thank all of the carers in the UK, so the NHS, and everyone caring for people who are ill with the coronavirus. And what you would see at 8pm on those days was these drops, and I've circled them here, you can see how Internet access actually dropped. So people would know their phone isn't in their hand when they're clapping, they're outside, they clap for a minute or two, and then the Internet access goes back. You can also just make out on these graphs that kind of multiple peaks thing during the day. But you can see the clapping was fairly consistent, a little bit of a drop towards the end, maybe as people was less well attended, but nevertheless, it was there. And patterns like this show up a lot in the Internet. Now, this isn't quite this week, because this ended on May 28, when it was decided that 10 weeks was probably enough to keep this tradition going, it should be replaced by something else. Another thing that ended very recently was Ramadan. And Ramadan ended at the end of May, has a huge effect on Internet access as well. So if you take a look at this, the sort of purpley pink line at the top is Internet access in a country where Ramadan is very respected by the population prior to Ramadan, the week prior, and the yellow is the week of. And it was interesting as you compare these two, you see some really big differences. So the huge drop is the breaking of the fast in the evening, the sun is down, people break the fast to eat and pray, and they stop using the Internet. And then it comes back on back up again during the evenings. And you'll notice the days are shifted, the days are shifted later. So people are up later, and is a little tiny bump in the early morning, which is the first prayer of the morning. So you see people's activity on the Internet happen through graphs like these. And we see this around the world, we've seen it change with the pandemic. We see it with religious festivals like Ramadan, we see it with social movements, like clap for carers. And there may be many others that I haven't looked for. And again, I'd love to hear from you live studio at Cloudflare .tv. And we'll look at these kinds of patterns. And that is it for this week in net, I'm going to take a look in the chat room to see if there are any questions from the audience. Pick up my phone. As this is very live. Okay, let's see. Do I have any questions? I'm not sure I do. Okay, so somebody has asked a question and I will read it out and then we'll get back to you. I'd love to know any data you may have on back end software that Cloudflare protects. Do you have any idea what percent of the sites you're protecting run PHP, Python, etc, or what web servers they are running Apache, Nginx, etc. So in general, from the perspective of technologies like PHP or Python, we don't. But would require us to somehow do some kind of scan of our customers websites, which we don't currently know. We will know from the server header that server provides roughly what software it's using. And that's what I can answer, we can take a look at to look at the extent of those. Similarly, we see the same kind of thing for browsers as well, we're able to see the browsers that people are using. And you'll see the rollout of new browser versions, particularly Chrome, where the browser updates are automatic and Firefox, you'll see the sudden change in the browser versions that are being used. One of the interesting questions that is brought up actually during the pandemic was, how did Internet access change in terms of web browsers that were used? If you think about a lot of organizations, a lot of organizations, in fact, I was on a call half an hour before this, with an organization that had difficulty joining Google Chrome, Google Meet, because of the organization's old web browsers. And that's quite common in large IT organizations have a very fixed platform you're using on your laptop is a fixed image with a particular web browser. So one of the interesting questions was, if you send everybody home, what web browser do they use at home versus what they use at work? Now, we can't tell exactly because we don't know who the people are, but we can tell in aggregate. What was fascinating, and maybe this would be a good subject for another trend, one of the trends shows, was that we saw two things happen. A large group of people got a better web browser. So we saw a decrease in some older versions of web browsers, and an increase in some of the much newer versions. But there was another group that went the other way. And we haven't been able to explain this in depth, but it's probably the case that those users were bringing out an old PC at home, an old laptop they hadn't used for a while, because they needed to, maybe because they didn't have equipment they could take home from work, or maybe because a child needed it. And right at the children, educating them, etc. So maybe the children were using a slightly older web browser. But there was a big change. Let's see if there's another question. Nope, that looks like all of the questions for today. I will say goodbye here. This is This Week in Net. Please do send in an email, livestudio at Cloudflare.tv at any time with ideas for things you'd like me to cover, and I'll look at the data, and I'll look at the stories that have happened during the week. Thanks very much. And I shall stay on for three minutes and see if any other questions come in. And as I am British, I'll drink some tea. Cheers. I'd also love to hear from people who go to isbgpsafeyet.com and what answers they got, because we are actively trying to spread that from the 20% of routes that are signed right now up to 100%, or let's say 80%. That would give us a great deal of herd immunity in the Internet to route leaks, because they really are one of the last big problems. And there's no more questions coming in. I think in two minutes, you've got Doug Kramer coming up. That should be good. He's Cloudflare's general counsel. Oh, one other thing I should add, since I have a minute left before they cut me off, is one of the things we looked at during the pandemic, and I have graphs I could show you next week, is did the Internet get slower for people? And we looked at that by looking at the round-trip time. And the answer is, in some countries, it did get a little bit slower, and in others, it didn't change at all. And there seems to be a rough correlation between whether you were using DSL or not, and whether you were using DSL or not. As the primary access method. And some of those DSL networks came under a lot of pressure over time. So we'll look into that next week, as to how the round-trip time changed, what that means in terms of Internet access, given that the Internet became so important during the pandemic. All right. I think I'm at the bottom of the half hour. Thank you very much, everybody. I'll have some more tea. Hope you join me.