This Week in Net: Bug bounties, attacks on health organizations and Groundhog Day

Hello everyone and welcome to This Week in Net, February the 3rd, 2023 edition. I'm João Tomé, based in cold Lisbon, and with me I have as usual our CTO, John Graham -Cumming. Hello, John. It's not that cold, João. It is cold, it is cold. Alright, it's cold for Lisbon, and I've got a jumper on. Yes, yes. And I have a shirt, which is like the most usual thing for me. We had a few blog posts this week. Last week we spoke about Data Privacy Day and all sorts of blog posts. But this week we have different types of blog posts we had in the blog. Why not start with one that does a reference also to our bug bounty program? In this case, it shows how it was really helpful, right? How it made an impact. Yeah, absolutely. I mean, so Cloudflare has a bug bounty program through HackerOne, which a lot of companies use. And if a security problem is found outside the company, it can be reported through there. And we have bounties, i.e. we pay a certain amount of money to individuals when they find bugs. And some of those security bugs are very complicated, and some of them are relatively simple. And depending on the impact, there'll be a larger or smaller bounty available. This particular blog post you're talking about is a write-up of something that was discovered by someone outside the company back in November. And we fixed it very quickly, but we hadn't actually written it up. And we try to write up problems, security problems, outages, all that kind of stuff for the blog. So the team wrote this one up. Before we go into the details of it, if you scroll down in here, there's actually a timeline, which I think is kind of interesting. And so if you look at this, 27th of November, evening time for us here in Portugal, somebody reported this in through HackerOne. And we triaged those reports pretty rapidly. There's a team of people looking at reports coming in internally for things and also externally for things. And about, I think it's about sort of 15, 20 minutes later, our on-call CERT, our Security Incident Response Team, was paged by the folks handling the bug bounty. So about 20 minutes after the person reported this problem, we paged them. The manager of the on-call got together. It was declared as an incident about 40 minutes after the bug came in from the outside. And about an hour, I think an hour and 30 minutes, sorry, two hours afterwards, we had a hotfix ready and we started deploying it. So it was a pretty rapid turnaround. We look at the severity of a problem and we try to get this stuff out very quickly. And by the next day, so by a day later, we had a permanent fix out there and it was fully merged within a day. So if you look at this sort of start to finish, it's like two days to go from somebody reports something problematic to a permanent fix merged and rolled out worldwide, but actually only a couple of hours before a temporary fix, which stopped the problem, was actually rolled out. So when you look at how we handle these things, we tend to move pretty fast. And hopefully, you can see that the processes we have there. Now, if you go back to the actual bug, back at the beginning, it's rather interesting. So if you're not familiar with this, obviously your computer has an IP address and it also has something called a loopback address, which is basically saying, me, I want to talk to my own computer. Perhaps the most well-known is 127.0.0.1. And we don't want any DNS records to be able to point to the loopback address because we don't want any systems that are controlled outside of Cloudflare, by our customers, to be able to talk to internal machines. And so we block all that kind of stuff. Now, IPv6 also has a concept of loopback address, but it also has a way for backwards compatibility to embed an IPv4 address in an IPv6 address. And this is what this weird thing is here. So that loopback thing, 127.0.0.1 in IPv4, becomes this thing, colon, colon, f, f, f, colon, 127.0 .0.1 in IPv6. And an external script researcher managed to realize that we hadn't done a good job of mapping, of blocking, basically loopback when it was using an IPv4 address embedded in an IPv6 address. And that allowed them to cause one of our bits of software to contact the local loopback address, which could be very dangerous because it could be internal services running on there, and we would like to make sure that they're not contactable at all. So that was the bug, and we very, very rapidly fixed it. But this is, I think the big message of this is that we really love our bug bounty program, and we love the things people find, and the folks who report them, many of them are really fascinating bugs that they find, and it helps make us more secure. Yeah, there's a little bit of information about how it happened. So it was done using Cloudflare Workers, and a specially crafted DNS entry. Makes sense. And one of the things, so in this case, actually, when is declared an incident? So when are we sure that it's an incident and we should look more into resolving it? Well, actually, anybody in the company can, using our chatop system, declare an incident. You can just type it into our chatop system that you're raising an incident with a priority and the reason why, and that kicks off an entire process with a chatroom created, a Jira created with information about it, paging to the on-call incident commander, et cetera. So we have a pretty low bar for that. I think the last incident that I raised was actually for the death of Queen Elizabeth, because we were worried about how traffic was going to shift on the Internet once we knew that she had died, and that could have caused us a problem. It didn't actually cause us a problem, but I decided it would raise an incident, and other people can. So in this case, I think it got raised because there was the possibility of someone being able to write a worker that could contact an internal machine at Cloudflare, and you want to make sure that that's not possible. The interesting thing about this is obviously we fixed it, and there you go away, but actually the reason why was a mismatch in how IP addresses were getting parsed, and so what was happening was one bit of software was saying, this is an IPv6 address, but it was emitting internally the IPv4 address, and then the other system was saying, well, this is an IPv6 address, but it was actually an IPv4 address, and therefore it didn't match the deny list, so it didn't actually get blocked. So it's one of those things where two bits of software, like an impedance match between two bits of software, and this is the sort of thing that security researchers are often really good at finding is this kind of unintended consequence, and so that was a blog this week. Exactly, and it helps avoid problems for sure, so those are important. Absolutely, and it's nice to write them up, and people can see the sorts of things that you deal with and how you deal with them, and if you do find something, please report it to us. We're very grateful, and there may well be a bounty involved. Exactly, so you can earn money by helping us. Yep. This week there's a lot of talk on, in this case, healthcare organizations experiencing targeted DDoS attacks, and this was related to some concerns about one group, one hacker group that is well -known of creating problems, and we pushed a blog post out just focusing on that specific topic, right? Yeah, I think we had seen a group that claims to be this pro-Russian group called Killnet, it's a little hard to know if it is actually them, but it claims to be them, targeting healthcare organizations, and what we see is targeting them with HTTP DDoS attacks, and so knocking offline perhaps access to systems within a hospital or information about the hospital, and we've just seen an uptick of that in the last couple of weeks, since the beginning of January actually, and maybe this is related to the conflict in Ukraine, it was a little bit hard to know, but just notable that for some reason hospitals and other healthcare providers had come under attack, and we showed what we were seeing from our perspective, we do obviously protect a number of hospitals around the world, a number of healthcare organizations, quite a large number, and we left some information there about how people could prepare and how they could get a hold of us if they need to protect something that's under attack. Exactly, and in this case, we also explained why DDoS attacks are used in this way by these types of groups, because they are in a sense easy to use, and we also explained how everyone can prepare better using our systems in order to avoid these types of problems, because they are avoidable, but if you run into this type of problem, and in this case it's a hospital, it has a really urgent need to have access to those systems, this can create real harm, so it's important to be aware of that. So we also have this message of those, if someone is under attack, we can help, and we have a specific page for that purpose also. Yeah, and by the way, if you do contact us when you're under attack, that isn't a sales call. We're not going to tell you, oh, by the way, this is how much you need to pay us for us to fix the problem. We will get the site or the service back online, make sure that it's stable, and run like that before ever talking about the cost of it. And so what typically happens is we get people up and running, and then we let it settle down, and then we'll have a conversation, do you want to continue with this service, but not, if you're under attack, we will get you back online. Yeah, that's important, especially in this type of situation, healthcare, those providers, that will make a world of difference for sure, just resolving the problem and don't think about anything else. We had more blog posts this week. Where should we go in terms of blog posts you want to mention? Well, let's go to this, it's very complicated to say CVE 2022 blog post because that was kind of a fun one. And it's all about a bug in this case. Yeah, a bug which turns out to be a denial of service in Linux. And so this is only one if you really, really want to get into Linux internals and get kind of nerdy about something. But basically what happened was we had noticed there was sort of a bug to do with Linux namespaces, in particular user namespaces. And we had actually noticed this, but we thought, oh, yeah, well, that's a bug, and we reported to it, reported it. And then it didn't seem very important at the time. Actually, we went off and we notified the kernel.org mailing list to say, oh, by the way. That was June, right? Was that in June? I think that's about right. It sounds about right. No, I think it was actually, I think it's even longer ago than that. I think it might be like March of 2019. I think it's like ages ago. We're like, oh, by the way. We wrote about it in June, so possibly it was before that. I think it was 2019. We were looking at using what's called queue disciplines in Linux to move around packets and actually in particular to do some stuff to do with token buckets so we could decide how much traffic goes to particular locations. And there's this thing called no queue, which essentially is meant to pass packets on, but they're meant to be just to get dropped, basically, which is helpful for us. And there was this thing in documentation which said, oh, by the way, you can't set these up on physical devices. And we're like, well, I wonder what happens if you try to. So you run a bunch of Linux commands and you get a kernel panic, a null pointer dereference. And this was back in 2019. We're like, oh, well, okay. So, you know, so what? So we should report it. Report it to the mailing list. Didn't seem very, very important. Okay, fast forward almost three years, and we're doing a lot around user namespace hardening. And we were like, wait a minute, remember that bug? Can this, you know, can we do that? Can we use user namespaces to do this? And it turned out we could. And it turned out we detailed how this happens. And so, you know, there's a patch. We published a patch for this. We have some takeaways about how to harden your systems against this kind of bug because it would be possible in the right circumstances to set something up that on receipt of a packet, you get a kernel panic, which you would not like because that's a denial of service. So this was a pretty deep dive, nerdy post about a bug we found that turned out to be a serious bug that turned into a patch to its kernel. Just a question, in terms of how many bugs like these there are. So bugs there are a lot. But they have different scopes, in a sense, right? I mean, I don't know the answer to how many didn't get written up as blog posts. I mean, if you go to the kernel mailing list, you'll see lots of things getting reported, right? And so little things. It's just that this is one we ran across in our work because we're obviously very interested in hardening our use of Linux. And, you know, we reported it, and there it is. And that's a patch. So it's a good one to have. Where should we go next? We also have two blog posts from today. We do. We have a couple from today. These are product features. So one of them is advanced HTTP alerts. And basically, you know, we give people alerts on traffic to their properties behind us. But we've added some pretty detailed alerts where you can configure them. This is something that our enterprise customers can use if they want to get very, you know, very, very specific alerts about, you know, traffic patterns or traffic volume or errors or stuff like that from our system. So it's a nice new feature, and it's good if you're managing something complicated on Cloudflare. Yeah, you can get a better perspective of what's happening, in a sense, too. Another one is about egress IPs in Cloudflare Zero Trust. So, of course, the Zero Trust platform is all about, you know, you authenticating to the applications, to the services that you use, and then we can apply rules around who you are and where you are and what your device is and give you or not give you access to the thing, to the service. But there are some applications still around that want only to be accessed from a fixed IP address or a fixed range of IP addresses, which is a little bit of an older process, but still, you know, it happens. And, you know, they sort of consider this to be like a sort of second factor. You know, it's John's logging in and he's coming from this IP address. So we have dedicated egress IPs for this. And, you know, if you need it, you can now manage that within our system. So this allows you to set up a policy that gives you access from specific IPs for specific applications. And that is sometimes important for the way applications are built. And it's prepared for this hybrid way that we have now. Different types of systems, different types of usage of services, right? Yep. I mean, there's all sorts of ways you want to connect to our systems and IP addresses. Well, that's one of the ways some older systems needed. And there's a Portuguese example here, which... Yes, there is. But Montijo here in Portugal. Yeah, exactly. Exactly. I wonder what they were testing in Montijo. But yes, absolutely. It could be a Portuguese IP you need to access the system from. You never know. Exactly. It could be very specific and also next steps for those who are interested in this situation in specific. In terms of blog posts, I think these were the ones we had this week and the ones we already had mentioned last week. In general, we're already in February. So the year is already coming along. Can we share something about our next innovation weeks or next types of things to share? I don't know about the next innovation weeks. They're coming. We're working on them. I'm going to keep that as a bit of a spice. But I saw you tweeted a graph of Groundhog Day. Can we bring that graph up? That was kind of cool. We can. Let me find it. Here it is. Mostly, at first, I'm a big fan of Groundhog Day because of the movie, of the Bill Murray, Harold Ramis movie. And so that movie, for those like me who live all this life in Europe, I'm not accustomed to Groundhog Day. And that movie gave me the highlight of Groundhog Day but also gave me one of my favorite movies for sure. So it was yesterday. Listen, you and I totally disagree about this because I had never seen Groundhog Day until about two months ago. Oh, no. And I was like, I'm going to watch this movie. It's such a classic American film. I really, really need to see it. And I watched it and I was just like, what is this film? It's so weird, right? It's just weird. I mean, I like Bill Murray. I like Andy McDowell. But I was kind of like, okay. So anyway. It's definitely like a dark humor type of thing. So it's not for everyone. But for me, it's like really important because it allowed me to see how a comedy can be. So less comedy and more dark and more philosophical. So it feels like philosophical because the day is always repeating itself. And this week, there was a study that someone did, don't know why, related to that movie actually, the Bill Murray movie. And the study said that in order to learn all of what the Bill Murray character learns when the day repeats itself, the day had to be repeated like 12 ,000 times. So that would mean 33 years of looping the same day, always the same day, for him to learn like piano, things like what he learned. So in terms of being an expert in different areas. So that's a weird thing that popped this week. You are way more into this than I am. I saw the film one time. I was like, okay, great. I've seen it. It was fine. But I'm not going to go into an analysis of how long it would take him. I think maybe you should be a moderator of a subreddit dedicated just to Groundhog Day. Who knows? You're a big fan. Anyway, what happened? I guess there are websites. Exactly. So these are Groundhog Day websites. There are a lot. And mostly what we saw was an increase, no surprise there, when the Groundhog made his prediction. It was around 7.20 Eastern time. And we saw at that time, around that time, 16 times increase in DNS traffic to those websites. And the peak was a few hours later, like 2 p.m. Eastern time. So that's mostly the story that we were able to tell with our data. You've buried the lead here. So Punxsutawney Phil, did he see his shadow or whatever the tradition is? I'm not even sure I understand the tradition. He sees his shadow or not, and then his spring is going to happen, or I don't know. You explain it to me. I'm not an expert on the tradition itself, but from my understanding of the news, he saw the shadow. And that means there's six more weeks of winter. So that's the theory. And anyone can see if that theory will hold up. So hopefully Phil will be on the spot here. But there are other Groundhogs. So Phil is not the only one. Shall we set a date six weeks from now? You and I come back on this week in net. But did we have six more weeks of winter? Because as you were saying, it is a little chilly in Lisbon. Maybe we do have a bit more winter coming up. Maybe. I'll put on my schedule, because if not, I will forget it for sure. But yeah, just one very specific trend here. Any other type of suggestion you want to give more nerdy for our audience? More nerdy? Wow. What are you reading these days? Okay. So I'm a big fan of a French graphic novel series. This is hyper specific. It's called Blake and Mortimer. It is in French, and it's about two very British heroes of the 1940s and 50s and 60s who are protecting the empire from all sorts of things. It's totally ridiculous. And I love it partly because it's in French, and I read it in French. Also, it disperses lots of English words and is totally outdated at this point. It's sort of so old-fashioned. But they keep producing these two characters, David Blake and Mortimer. And Philip Mortimer, who is a Scottish scientist, also a pretty good boxer as well. Philip Angus Mortimer. That's a good one. A very specific one. It's very specific. He didn't expect this, but yes, they're sort of playing on this archetypal British gentleman. Philip Mortimer is both a gentleman, a scholar, and also quite good at boxing as well. And Francis Blake is that dashing guy in the trench coat there with his uniform on all the time, who happens to also run MI5. So there's also a relation with Tintin, right? Tintin in Portuguese. Yes, because I think that the guy who originally created it got going because he was involved with the comic in which Tintin was published. I've never been a Tintin fan, but there we go. But Blake and Mortimer, I just find it dashingly British. The villain, there is a perpetual villain, the Colonel Ulrich. I'm not quite sure where Colonel Ulrich comes from, but they never managed to quite get rid of him. He's always coming back. So that's what I'm reading. There you go. You asked. You did ask. I did ask. But it's always a good suggestion. For me, I'm not reading nothing right now, to be honest. I'm in between books, but I'm seeing The Last of Us TV show on HBO that comes from the video game. And I'm kind of surprised because I played the video game before a couple of times as a journalist. I was testing things like video games sometimes. I'm not an expert on video games. The video game was fine, but I'm not a very big fan of video games. To be honest, I'm a much bigger fan of the TV show because it's very dense and long. And you don't need to understand or like the video game to like the TV show, which I think is a good value there. And there's only three episodes because HBO does it like I like, like one episode per week. So you can digest each and every one of those episodes. And one of the writers also wrote Chernobyl, the TV show, a good one. And one of the things I like, they do this thing. If you want to explore more about the show, and the show is a little bit philosophical also, in some sense, they have a podcast. So anyone can hear a podcast specifically with the creators of the show, which I found interesting. And you're a big podcast fan. I'm a big podcast fan for sure. I don't remember the last time I listened to a podcast. You must start to hear the This Week in Net podcast, which is available. Is there anybody good on it? I heard sometimes there's a good show there. Let's hope it continues. Yeah, but mostly that was a suggestion I want to give. We had a few time. And next week, more blogs, more things to see on blog.Cloudflare.com. That's right. So more things to see. And I'm going to give you a shout out to another Cloudflare page that you might not have looked at, which is research .Cloudflare.com. And that's the research team, which is on my team. And they work with standards bodies and universities on long term stuff. And that's actually another fascinating place. If you've run out of content on blog.Cloudflare.com, the research team is here. And you can read about a lot of very interesting stuff going on there about helping to build a better Internet. Exactly, publications. And even recently, I did two interviews with visiting researchers that come from university. And you can watch them on Cloudflare TV. So there you go. And another suggestion there. Yeah, the site is research.Cloudflare.com. And anyone can see it. And I think we're out of time. Thank you so much, John. Thanks very much. Stay warm. You too. See you next week. Yeah, bye-bye. And because we still have some time, here's from Rita Kozlov, our Senior Director of Product. In a conversation I recorded for our segment, We Are Cloudflare, where she mentions 2023 and what to expect in our workers platform. Just a few areas that I think we're always focused on. The first is always ensuring that the developer experience is as good as possible. And we like to think that we're best of class today, but that's never good enough. And so I'm looking forward to sharing some of the big developer experience improvements that we have coming. I think observability is going to be a pretty big theme for us. And I think that's a really big part of developer experience, right, is making sure that you can not only write the code, but also that when things go wrong, you're able to understand what's happening and where and dive into it. And quickly, right? And quickly. And quickly. And quickly. Giving developers access to one of the constant areas of growth for us is giving developers access to more tools. We want you to bring as much of the full stack to us as possible, right? And so last year or a couple of years ago now, we announced R2. Last year, we announced D1. And so continuing to kind of build out those building blocks, I think those are all really important directions. And then, yeah, continuing to see the community grow and would definitely encourage anyone to join it, whether on Discord or interacting with us on Twitter. I promise we're very welcoming.