Threat Watch

All right. Well, welcome to Threat Watch. It's Monday, the 20th of July, and I'm coming to you from sunny Lisbon. This is John Graham-Cumming. I'm Cloudflare's CTO, and Threat Watch is a show that I do mostly weekly about interesting threats or problems that have occurred on the Internet in the preceding week or two. And this week, I want to talk pretty much exclusively about web application firewalls, and a particular problem that occurred with one vendor and how it got patched, and how it worked, and what role WAFs play in it. As many of you will know, if you're a Cloudflare customer, Cloudflare offers a firewalling product, and the firewalling product has a number of components to it. There is the web application firewall, which blocks attacks on APIs and web applications by filtering out known attacks or things that look like attacks. So SQL injections or attempting to run commands on machines, known CVEs, known attack types get filtered out. The firewall also allows our customers to configure things they want filtered out, might be by a particular IP address, by a particular signature they want got rid of. And the same firewalling stuff also blocks quite a lot of Layer 7 DDoS attacks. Those are the DDoS attacks that happen at the HTTP level. So everything I'm going to really be talking about today is going to be mostly around the HTTP level, which is requests that get blocked. And just to give you a sense of the scope of this, Cloudflare blocks, well, billions, in fact, tens of billions of bad requests per day. So this is just the last week, the last seven days, and you see well above 60, sometimes well above 75 billion HTTP requests that are blocked per day. So these are HTTP requests being made to our customers that the firewall has been configured to block. And a very large proportion of this, in fact, most of it is blocking by the WAF. So the WAF is going in there and blocking out attack types that might cause a website to get defaced or broken into or data stolen. And you can see the trend. It's fairly static in terms of what happens over a weekly basis, but billions per day. So many per second are being blocked by Cloudflare. And it gives you some sense of the scale of the sorts of data that Cloudflare is dealing with when blocking attacks. So billions of requests per day. And how does that break down? Well, obviously, within the firewall product itself, we actually have a ton of different rules to block things. In fact, there are thousands of rules for blocking different attack types. But you can kind of summarize them into some different areas. And I've talked about this previously on ThreatWatch around the OWASP top 10. But I'm going to show you some sort of sense of this. So this graph is a slightly older graph. It goes back to March and April. But fundamentally, this mixture doesn't change very much. These different colors are different attack types blocked by the Cloudflare WAF. So this excludes anything that's DDoS activity. It excludes anything that a customer has written a rule to block. This is looking at the types of things the WAF blocks. And you can see I reordered these so you can essentially see the largest category at the bottom and then going up. So the largest categories, roughly a quarter of the requests that we block, are attempts to inject commands. And that's typically an attempt to run code on someone else's machine. And this is probably one of the most dangerous types of attacks. Because if you can run code on someone else's machine, then you can start to actually infiltrate the machine, maybe knock it offline, maybe get more information from it. And a very typical thing attackers do with machines is they will install a shell on that machine, in particular a reverse shell, so they can then log in when they feel like it. Because that will allow them to log in even if the hole they were using actually gets patched. So command injection, actually running commands to someone else's machine, tends to be the most popular. I'm going to skip over the Perl 4 and talk about it in a minute. Above that is SQL injection. This is sort of the third most popular thing. SQL injection's been around for what seems like forever. It's attempting to manipulate a web application, be it through the web or through an API, into giving information from a database, from a SQL database. Many, many web applications take input from users, like when you type in your name on a form when you're going to order something online, and they put that into databases. And maybe they even look it up. So for example, you're logging into a website, you type in your username, that username might get looked up in the database. So the username is taken and is then used as part of a query to a database. And in that movement of data that the user controls to the database, it's sometimes possible to manipulate what's in there to actually perform a SQL command of your choice. And usually, exfiltrate data from the database, usually are often masses of data. And those are called SQL injections. And they're quite common. You can see the attempts to do them are very, very common. Above that is file inclusion, which is attempting to actually put files on the machine. Often this is done actually, so those files can later on be executed, because they're actually code files, but it could be for other reasons. You can see a little sliver of blue, which is cross site scripting. So changing how a site operates, so that it will actually execute some JavaScript, perhaps when someone else is logged in, perhaps when all users are logged in, and so on. Just to return to the purple, we've got protocol anomalies. These are people trying to manipulate the HTTP protocol to make the server do things that it wouldn't normally do. So it might be, there's a thing called request smuggling, which is you try and put one request inside another and get a server to do something. There are various things. And this is quite a large category, protocol anomalies. But by far the biggest is command injection. And it's very dangerous, especially if when a command gets run, it has some sort of privilege. But even if it doesn't have privilege, you might be able to get what's called a privilege escalation, which is once you get a foothold, you then turn yourself into the administrator on a machine and can do more. And these are very, very dangerous. We block a lot of them every day. And a large number of the WAF rules are actually around blocking that. And if I just go back to the top, there's a thing called information disclosure. And information disclosure is where some information on the server is actually disclosed to the end user. Quite often, this is around things like source code control systems, because people will put the files on the machine, and they may actually have a .git directory or files from source code control. And if an attacker can get that information, then it gives them a wealth of information about the systems being run there. Sometimes they might even give them the source code. And sometimes that source code might even contain secrets. And so it's a dangerous thing to do. And so we have to be very careful when we have things on the file system that don't get disclosed, and the WAF actually has rules to block those things. One of the key things about WAFs is that it is still absolutely essential that people patch their software. If there's a vulnerability released to the public about a piece of software that you're using, be it a library that you're using, a framework, a server platform, it is absolutely essential to patch. And the reason is the WAF can do a lot to protect web applications, but attackers are looking all the time for different ways to get in. And what the WAF can really do is help you with what I refer to as the patching gap, which is a vulnerability gets released, immediately the bad guys start looking at that vulnerability and writing code to try and exploit it. And it is very hard for organizations to actually patch faster than the attackers can start attacking. And because of exploitation tools that can scan the entire web very quickly, it's often the case that attackers are actually trying out new vulnerabilities that have been released to the public within hours to single digit days of the release. And so we really see this happen a lot. And we have an example over the last few weeks with one particular vendor that had a command injection vulnerability that was very, very serious. Now, this command injection vulnerability was bad because it was on the administration interface for this particular device. And it didn't require authentication, i.e. you could just do essentially a drive by, you could just get a particular URL from the management interface without ever having to log in, without ever having a legitimate account, and you could actually run commands. And so it's really has a very, very high CVSS score. That's the scoring to tell you how bad a vulnerability is, because it was run code and run it without any authentication. That particular vulnerability was CV2025902, which you can Google and you can find it in the MITRE database. And you'll also find the vendor who was involved in this. So the vendor in this case was F5 Networks. F5 is a very big producer of networking hardware. And they have a product line which is called BigIP. And if you've been around for a while, BigIP is very well known as a load balancer. And it's been around for many, many years. And BigIP also has many other functionalities around DDoS mitigation, traffic management, all sorts of things, but fundamentally often uses load balancers. And so they are often exposed directly to the Internet because they are load balancing Internet traffic. And there was a fault with the management interface, which is TMUI, which is a traffic management user interface in F5, which allowed an unauthenticated attacker to run an arbitrary command on the machine. And it's a Linux based system, so they go and run things on the system. And that was messy because anyone could go and do it. And so, you know, this was a problem, especially if or only if the management interface was exposed to the Internet, which often it was so that people could actually manage the device. So the actual description of the CVE says the following. It tells you the particular BigIP versions. It tells you that it's remote code execution vulnerability. And it says undisclosed pages. And we will see the pages that should get used. It's very simple. It's typically the login page because you can typically find that on the Internet if it's exposed. Now, if you have an F5 BigIP product and you have not patched and you are watching this right now, the rest of what I'm going to say is completely unimportant to you and you should turn me off and go and figure out how to patch. You'll quickly find F5's page on it. If you search for the CVE number, let me go back to it so you can search for it. This is the CVE number. Go search for that thing, F5. They have a very detailed page which they've been updating over time that tells you how to patch and also how to put in place a temporary mitigation. So if you've got one of those products, please don't listen to anything else I have to say. Go patch it. Now, if you don't have one or you patch and you want to learn some more, let's talk about that a little bit more. So one of the things that, you know, came out very, very quickly was that there were exploit tools around and those exploit tools and even just simple things on GitHub said, you know, this is how to do it. And so the structure of these attacks looks pretty much like this. These are five, I think it's five, right, example URIs that people were using to test whether a TMUI interface was vulnerable. So it's just an HTTP. You could just do this in a browser or with curl or something like that. And you'll notice that they're all going to login.jsp. So there's a login page for the device. And then they are doing something else. And you'll see there's two instances of .jsp. So there's the actual page that is accessible. And then after the dot dot semicolon, there's another thing, which is another file. And you'll notice that there are really two here that are being used. There's file read.jsp, which actually it turns out will read an arbitrary file on the file system and return it to the end user. So that allows you to pull information. And the last one on there is tmshow command, which actually will run a command in the command line language of the F5 device. So two kind of scary things. In the examples here, they are pulling down information. So they're pulling down etc password, etc host, which allows you to find what host you're connected to, and then information about the license. So these were quite widespread. And the last one at the bottom there is running a command to tell you who the admin user is of the device. So these are things that people could just curl and they could mass scan for. The common element in all these, if you see that dot dot semicolon, and that's actually sort of the core of the vulnerability here, which is that it was possible to request one page and then actually get it to execute another page because of the way in which the Tomcat server and Apache inside the device was set up. Now, because of that common signature, some of the initial mitigations for this were to block anything with dot dot semicolon. And actually in F5's initial release at the beginning of July, they put in place a change to the HTTP to Apache on the machine to actually match anything that had dot dot semicolon in the URI and 404 it and say, no, you can't get to that. And that was very effective. Unfortunately, the folks who actually found the original vulnerability realized that this wasn't enough. There were other ways to exploit this and get arbitrary code execution to happen on the machine. And so they had to go a bit further. So if you've been following along to here and you say, oh, yeah, I put in place that mitigation that F5 told me about on the 6th or 7th of July. Again, please stop watching this. Please go and read that page again and look at the mitigation that's in place. And if you haven't patched, please, please patch. But the mitigation has changed a little bit. So you need to go ahead and do that. All right. So what Cloudflare did is we put in place a WAF rule to block this. So we put in place a rule to block these attempts to break into the TMUI. And we rolled that out to our customers and the anyone who's using the WAF got that protection. And what's interesting is, just as I was saying at the beginning of this, you see exploitation within single digit days of something becoming public. And so if you take a look at this graph, which is log scale, so if you take a look at the log scale on the right, we saw a small number of hits on the 3rd of July. This was made public about the 1st of July. And then after the 4th of July, things really got going. And it really ramped up to over a million blocked requests per day. And it's continued like that. So the window of time in which you could actually patch this was very, very small. And this is where a WAF can buy you some time. And also, we can update the WAF as the mitigation information change. So change from dot dot semicolon to something else. So you're seeing here that this will be here forever. Basically, what happens is these attacks, first of all, are done manually. People try them out. And then they add them to scanning tools. And that means they will be tried out all the time. And in fact, that is what's happened here. This attack has been added to scanning tools. And we're just going to see forever some low level of scanning going on in the background. Now, what we actually see is people trying this. So this is actual data from our network. And this is in the order in which we see people trying stuff. So trying to read arbitrary files on the system with fire read is right at the top there. Then there's a couple other ones, which is get, tab, set, and auth properties, which people are trying to do. And then running a command, which is the bottom one. So these are really, really popular right now. And people are actually trying them. Now, part of the reason they're trying them is they've been built into those scanning tools. So they're not necessarily an actual compromise, but they are scanning for this vulnerability. And if they work, then you'd be able to break into the F5 device. If you just take a look at the actual file. So if you go back and look at that first line there, which is file read dot jsp, it has a query parameter, which is file name, which is the name of the thing to be read from the operating system. And again, this is in order of the things that are being asked by actual attackers or scanners right now. So Etsy password seems extremely popular. People try and get the passwords off the machine. Etsy F5 release, which just returns a small file saying which version of the operating system is on the device, and so on down the line. Those top two are the most popular. And the Etsy F5 release one is really, really popular. And one of the reasons it's popular is it has been built into common scanning tools because it's kind of a safe way of checking to see if a device is actually vulnerable. It doesn't ask for some very sensitive information. It's not asking for the password file, for example. But if you do get back a response with the F5 release information, then you know that the system is vulnerable. And so it has been built into Metasploit. Metasploit is a very, very popular exploitation platform. And you can actually see this is what it does. It actually does a post, it does file read, and then it has this file name F5 release. By the way, GETs and POSTs are vulnerable here, not just GETs. A lot of people are using GETs as part of their test exploitation. So it's been built into Metasploit as it is today. In terms of commands that people are trying to execute, if you remember there's that executing commands. The most popular one is list auth user admin. Once again, the reason for this is that that's been built into some scanning tools as a way to see if the system is vulnerable and actually gives you back the username. This is a little bit more bad, really, in the sense of the one getting the F5 release is quite, you know, it doesn't actually hurt the system using it. I can argue that getting the user admin name is a little bit of an intrusion into the system. But you do see circulating on the web curl commands that use this particular thing. And that's really the only command we're seeing. So people are scanning. Now what F5 did in their knowledge base article, and it's right here, was they updated, and they've been updating, in fact, the last update was three days ago, what mitigation to put in place. So if you're not familiar with this particular language here, this is the Apache command line language, because, sorry, the Apache configuration file language, because they're running Apache, they can make a change. And they're adding in two things, which are location matches. And what those are is they match URLs requested from the server against patterns. In this case, the two patterns are really simple. One is semicolon. Notice that originally, when they put out the first information, it was dot dot semicolon. Now it's just anything with a semicolon in it. That's because it turns out this vulnerability works without the dot dot. And then there's hsqldb. It was also possible to get hold of access to a SQL database on the system. And that was another way of doing it. So they put those two things in. So this is the key thing to if you are patching the system rather than updating, but please, please, please update it. If we now compare those two, the previous graph and this graph, we can see something interesting. So the yellow here is the same as the yellow I had before. So that was matches against the dot dot semicolon style of attack, which was very popular. It's built into Metasploit and all those kind of things. The purple here is where there's just a single semicolon rather than dot dot semicolon. And so you can see that, you know, because the dot dot semicolon has been built into exploit tools, it's remaining very, very popular. But since the updates to say that only just a single semicolon will work, you can actually see those ramp up as well. So people are trying both methods. So again, new variant comes out, people start using the new variant. And curiously, actually, the single semicolon variant was being tried out right at the beginning. And it's possible. This is actually the security researchers who found it who were actually trying it, and we were catching it automatically. So that's my threat watch for this week. WAFs are useful. They're not the solution to everything, but they are useful, especially when a new vulnerability comes out because the WAF vendor can patch the system virtually before you've had a chance to update the firmware or the operating system on it. And if you've got one of those F5 devices, please go and do it. Attacks stay around forever. Tools like Metasploit means that old attacks, things like shell shock from way back, things like image tragic from way back, they're still happening every single day, because people will scan to find a vulnerable system. And so it's fairly easy to do that. And this attack on this particular interface will last forever. It'll be tried out on systems. If the system is identified as an F5 device, the scanning will happen. So a WAF will show that up, and you'll be able to see the extent of it. If you don't have a WAF in front of the system, and you want to see if your F5 device might have been compromised, there is also information on the F5 website on the same knowledge base article about what to look for in logs and what commands to look for that might have been executed on your system. But given the incredible amount of scanning that was going on, and the fact that attack has started very early, if you have an F5 device, and you didn't patch quickly in the beginning, it is very likely that someone tried out those URLs, and it's very likely that someone broke into it. So I would urge you to go look at those systems right now. So that's it for Threat Watch. Thank you for listening. It's an interesting example of a bad vulnerability, a remote code execution, a little bit about how WAFs can help, and hopefully you've all patched your F5 devices, and everyone is fine now. Thanks very much for listening. I'll be back next week with whatever's happened in this week, and I'll tell you about those threats. Thanks very much. Cloudflare Gateway protects offices, homes, and corporate networks from malware and other security threats without sacrificing performance. Gateway provides a secure DNS resolver and filtering service that inspects and logs all DNS queries to apply policies that either block or allow the request. This video will show you how to get started with Cloudflare Gateway by configuring a location, creating a policy, and using that policy to block security threats. To get started, navigate to the Cloudflare Gateway dashboard at dash.teams.Cloudflare .com. If you don't have a Cloudflare account, you can sign up and the browser will redirect you back to the Gateway overview page. Now, let's configure a location. A location is typically a physical location, like your home, office, store, or a data center that you'd like to protect. For this demo, let's call our location aus-1. Gateway should automatically detect your IP address, which allows Gateway to know which requests are coming from your location or network. Now, let's configure the DNS resolvers. To take full advantage of Cloudflare Gateway, you should change your router settings to the Gateway IP addresses. For this demo, I'm only going to use the IP addresses that Gateway assigns. Now, let's configure the DNS resolvers. To do this on a Mac, go to your laptop's system preferences, click network, then advanced, and navigate to the DNS tab. You'll see your existing Internet provider's DNS server IP address here. Add in the IP addresses from the Gateway dashboard by clicking the plus sign. If your network supports IPv6, make sure to add the IPv6 address here as well. Click okay, then apply. Now, my laptop is sending all of its DNS queries to Gateway's DNS resolvers. To complete the location setup, navigate back to the Cloudflare Gateway dashboard and click complete setup. After configuring your first location, you'll see the Gateway overview page. Here, you can view your location's requests and if they were allowed or blocked. After the initial setup, the graph may take a few minutes to show data. While we're waiting on the data to populate, let's confirm that our location was properly configured. It looks like our location is properly configured, but as you can see, there's no policy assigned. Let's create one. Create a policy and apply it to your location to protect your network from Internet security threats like malware and phishing. The policy will control what the user can or cannot access while connected to your location. To create a policy, click policies, then create a policy. For the purposes of this demo, I'm going to create a policy that blocks malware and social media. Let's call this no malware or social media. We'll assign it to our location by clicking here. Here, you can enable a block page, which will show if a user attempts to access a page that's been blocked. Let's enable it, then click preview to see what a block page would look like. Let's disable it for now. You can also enable safe search, which allows Cloudflare to automatically filter content based on the same restrictions that large search engines use to protect users from explicit content. Now, let's identify what security threats we want Cloudflare Gateway to protect against. Gateway allows you to block all security threats listed here with one click, which include malware, phishing, and spam. Let's just block malware for now, then move on to the content categories. Gateway allows you to block certain content categories. Since we want to block social media with this policy, click society and lifestyle, then social networks. If you'd like to allow or block a specific domain, you can do that in the allow block tab. Let's enter chatgoogle.com to ensure that it's blocked and click add domain. Now that the policy has been configured, let's click add policy. The policy will propagate throughout the Cloudflare network in a few seconds. So in the meantime, let's check out the Gateway activity log. The activity log is where you can see all the requests to your configured location. You can also see what content categories the requests were associated with. This request was associated with content servers and information technology content categories. It was an HTTPS request created from the aus-1 location and was allowed as it didn't trigger the policy. Now, let's test our policy to make sure that it works properly. Let's test the social media portion of our policy by attempting to navigate to Twitter. Shortly after hitting enter, you'll see an error page indicating that Twitter cannot be reached. Cloudflare Gateway has successfully intercepted the request and blocked the page accordingly. During this Cloudflare Gateway walkthrough, you saw how to configure a location, create a policy, and use that policy to block Internet security threats. To learn more about Cloudflare Gateway, navigate to teams.Cloudflare.com backslash gateway You run a successful business through your e-commerce platform. Sales are in an all -time high, costs are going down, and all your projection charts are moving up and to the right. One morning, you wake up and log into your science analytics platform to check on current sales and see that nothing has sold recently. You type in your URL only to find that it is unable to load. Unfortunately, your popularity may have made you a target of a DDoS or Distributed Denial of Service attack, a malicious attempt to disrupt the normal functioning of your service. There are people out there with extensive computer knowledge whose intentions are to breach or bypass Internet security. They want nothing more than to disrupt the normal transactions of businesses like yours. They do this by infecting computers and other electronic hardware with malicious software or malware. Each infected device is called a bot. Each one of these infected bots works together with other bots in order to create a disruptive network called a botnet. Botnets are created for a lot of different reasons, but they all have the same objective, taking web resources like your website offline in order to deny your customers access. Luckily, with Cloudflare, DDoS attacks can be mitigated, and your site can stay online no matter the size, duration, and complexity of the attack. When DDoS attacks are aimed at your Internet property, instead of your server becoming deluged with malicious traffic, Cloudflare stands in between you and any attack traffic like a buffer. Instead of allowing the attack to overwhelm your website, we filter and distribute the attack traffic across our global network of data centers using our Anycast network. No matter the size of the attack, Cloudflare Advanced DDoS Protection can guarantee that you stay up and run smoothly. Want to learn about DDoS attacks in more detail? Explore the Cloudflare Learning Center to learn more.