Originally aired on July 7, 2020 @ 11:00 AM - 11:30 AM EDT
Join Cloudflare CTO John Graham-Cumming for a weekly look at the latest trends in online attacks, with insights derived from the billions of cyber threats Cloudflare blocks every day.
Episode: June 23, 2020
English
Security
Cyber Threats
Good Good morning from Lisbon and welcome to this week's edition of Threat Watch slightly delayed from yesterday when we had some technical difficulties so don't be surprised by the date saying it's the 22nd of June. You are correct today is the 23rd of June but the 22nd was when I was going to do this last night so here we go again. The idea of Threat Watch is to look at mostly weekly but perhaps sometimes not weekly interesting new threats that have occurred on the Internet in the preceding week or so. Last week I did the first one of these shows and it was all about some of the older trends that we've seen over time getting this in place but now I'm going to talk about some things that have happened literally in the last week or so and two things of interest. One of them is a DDoS attack that didn't happen yet it seemed to get widely reported and the other one is a DDoS attack that did happen and so I'm going to talk about start with the DDoS attack that wasn't and well how did this happen? How does the DDoS attack happen that wasn't an attack? Well the actual denial of service in a way was a lot of people on Twitter talking about how things were going horribly wrong in the US when in fact there was only one thing that was really going horribly wrong in terms of telecommunications. This all started with anonymous so your non-central which is the main Twitter account for the anonymous collective tweeted twice about outages in the US one of them about the US being under a massive DDoS attack sorry I need to see the voice massive DDoS attack and the other one being about how every cell phone provider every mobile phone provider in the US was having terrible difficulties and massive outages and well the real story of what happened is slightly different if we take a look at what anonymous tweeted so this is 9 58 p.m my time they've tweeted this picture of a rather dramatic war games looking attack of all of the world appears to be attacking the US so even plucky the Maldives there New Zealand for some reason was attacking the US etc etc and if you look at on the left the things pointing up into the sky well actually that's what um Arbor Networks who creates this graph does when they don't know the source for a particular attack but it kind of looks like Starlink might have been attacking the US anyway the US was apparently under major DDoS attack and um well not quite true actually so we looked into this because Cloudflare provides DDoS services one of the first things we did was said are we seeing anything like this is this really a major DDoS attack happening that we haven't noticed because we have systems that would alert on that and having a look at that and looking at various other systems around the world and talking to the network operators we came to the conclusion that this was well a great big pile of something it's completely untrue that there was a massive unusual DDoS attack happening on the US on the 15th in fact if you go to digitalattackmap.com which is where that particular chart comes from and you start scrolling back in time you'll find something rather interesting this is the day before the massive attack on the US okay this seems to be happening on the day before so why didn't we hear about it on the day before anonymous and well the day before that on the 13th the same thing seems to be happening in fact the reality is that the US and actually most major companies are under massive DDoS attack all the time there are DDoS attacks occurring all the time and in fact this graph makes it look incredibly dramatic and is great for marketing because it means that people go out and buy your DDoS mitigation service but it doesn't actually turn into anything serious for consumers and the reality is that this looks exciting because this is a botnet which is being used to DDoS something but the actual data rates are not very high I'm going to come to that in a minute but hey let's keep going back in time June the 12th massive DDoS attack anonymous didn't mention it June the 11th it's the same thing in fact if you go back in time you will see very similar charts some of them even more dramatic than the one that was tweeted by Euronon Central over time so the reality is DDoS is just continuous on the Internet it happens all the time and there was nothing particularly unusual about June the 15th except that anonymous decided that there was one way to check this kind of thing is to go and look at the Internet exchanges around the world so if you pop into these Internet exchanges they mostly have public graphs of the traffic going across them the exchanges are where networks come together to actually form the Internet so this graph is from the Amsterdam Internet exchange is one of the largest Internet exchanges in the world and many many networks connect together here to exchange packets because if you need to get data from one network such as an ISP to another one in the world you need to exchange it somehow one way is to get in the room together and plug all the equipment together so this is showing in Amsterdam there was a peak about seven terabits per second of traffic passing through the exchange and you can see you know the difference in terms of the in and the out and if you look at these two days which is June the 16th and June the 15th you don't see any difference there's no sign of a massive DDoS if there was a really massive DDoS knocking out the US and causing all that trouble you'd expect these graphs to have a little bit of a peak on it the only peaks you see there are the early evening peaks which are very typical around the world people get home from work they get online they start streaming movies listening to music doing banking everything they couldn't do during the workday and Internet use tends to peak up a little bit and you can see it at night it goes down but there's no sign of a DDoS attack here I remember that DDoS attack was coming from everywhere so it should have shown up in Amsterdam it also didn't show up in Hong Kong similar kind of pattern here people go to sleep at night they do things in the early evening on the Internet but there's no dramatic peak of traffic here across the Hong Kong Internet exchange the same thing in Germany very similar pattern everyone goes to sleep at night traffic goes up in the day but no big peaks so the reality is this supposed DDoS attack was nothing unusual at all in fact some of those DDoS packets would have been being passed across these exchanges but they're lost in the rest of all the other traffic nothing unusual nothing massive here now that wasn't enough for anonymous four minutes later they tweeted this picture at 10.02 p.m my time saying all major sorry all the major cell phone providers across the United States are currently suffering from major outages except they're not so this is the more interesting story there was one Internet sorry one mobile provider in the US that was having a problem with voice and SMS service and that was T-Mobile and they had a problem in the southeast with a third party and looks like it was a cable cut that then caused them a problem of congestion on their network and that caused problems for T-Mobile subscribers but hey put these two things together nine four minutes before the sky was falling in the US with a DDoS attack and now all mobile providers are down everything's going scary but not really true again this was a great big pile of something it wasn't true it was based on graphs that were on the Internet that don't really tell you the truth and what you needed to do was a little bit of digging hopefully not in this pile to figure out what was really going on around the network so T -Mobile was in trouble that's absolutely the case in fact their president of technology at 918 said this we're working to resolve the problem we have a problem with voice and data and it's affecting customers around the country they quite quickly finished the data service and in fact once they'd fixed it they told their customers to start using things like whatsapp and other services to make calls and eventually they did fix the problem and if you look at it everything came back to normal so why did anonymous say that the sky was falling and all major operators were in trouble well because of a service i think called down detector a down detector is a fantastic service that looks at things that are happening on social media pauses them looks at what people are writing and figures out if there's a problem with a particular service so they look at twitter and people are saying hey my T-Mobile service isn't working T-Mobile is down T-Mobile is down and once it reaches a certain threshold then they report it as an outage and they do that on their website and sure enough they had this outage for T-Mobile and T -Mobile themselves had admitted it and the fact that was if you're into it there's a big thread on reddit T-Mobile which goes through all of this with all the detail so down detector was telling everybody that there were problems and there are those nice graphs where reports are coming from around the US make them look exciting as heat maps so T-Mobile was in trouble but why did it look like all the other major carriers were in trouble well here's the interesting thing down detector published a blog post about this a few days later the graph here the main peak here is the reports on twitter and other social media sites saying T-Mobile is down and you can see it shot up around the time of the problem and then it trailed off as things got fixed but you can also see that people were saying that AT&T and Verizon were down as well now this wasn't actually the case but here's the point if you're a Verizon subscriber and you try and call someone on T-Mobile and it doesn't work or you're an AT&T subscriber and you send an SMS to your friend who uses T-Mobile it doesn't work you think your service is down not necessarily theirs when you can't get through to them to ask them if their service is not working anyway so then you go on twitter and you say hey my AT&T service isn't working and so you see this little bit at the bottom here saying these other services are not working in fact AT &T tweeted that there was absolutely nothing wrong with their service across the country Verizon too no problems at all but people thought there was an issue with the service and so it looked like all major network providers sorry all major network providers are having a major problem but they weren't and in fact interestingly down detector had seen exactly the same thing in the UK about a week before Vodafone in the UK had a problem and at the same time other networks were reporting problems where people were reporting other problems with networks other than Vodafone on Twitter so three which is a big network in the UK O2 and EE which are both mobile providers in the UK also people were saying I've got a problem with this network and it's almost certainly the reason they couldn't reach their buddy who was on the Vodafone it was nothing to do with him in fact those networks were working working fine so you have to look out for this kind of shadow that's cast across the entire Internet when things go down now because anonymous said the country is under a massive attack and then all the network providers were in trouble this got amplified and you've noticed that the tweets by anonymous were widely shared and then you get people who put things together like this saying this DDoS attack is super serious it's taken down Instagram Facebook T-Mobile Verizon Twitch well you know 2020 is a nightmare if 2020 wasn't bad enough apparently now the cyber world was going to collapse well didn't actually happen so on the left you have that map which shows a pretty much run-of-the-mill day of DDoS attacks on the Internet this time targeting the US and frankly you could have filtered on any other major country and seen something very very similar and on the right you have down detector and if you look at the top you can clearly see T-Mobile having a bad time Metro PCS which is also run by T -Mobile having a bad and then some other services having problems now why are they having problems well unrelated to T-Mobile and also if you can't get access to something on your T-Mobile phone you may then think oh wait a minute Facebook's down or I can't get to messenger etc so none of these things were really happening but it started to look really scary and this is five and a half thousand retweets almost ten thousand likes so people started to run around worried now what happened was after that T-Mobile put up an explanation exactly as I said there was a problem with the provider they're working with they didn't have the right amount of redundancy this caused a massive failure the network got overloaded and they fixed it and so this was all fixed but if you were on Twitter you might have thought everybody panic the user is under attack and all of the mobile networks are collapsing everything's going wrong but actually just T-Mobile had a problem so that was the DDoS that wasn't but there was a DDoS that was and there was two reports by two service providers in the US about large DDoS attacks that had occurred and what's interesting here is if you remember a few years ago there was a very large attack on GitHub 1.3 terabits per second in 2018 knocked off GitHub for five seven minutes something along those lines while they flipped over to their DDoS mitigation service now some DDoS mitigation services operate on a standby mode so if you get knocked offline you flick a switch and you switch over and what happened with GitHub is they got knocked offline they diagnosed the problem they said okay this is a DDoS and they flipped over so they were down for five ten minutes max during the day other DDoS providers such as Cloudflare work on automatic mode where they're always on and they do detection automatically and so we would normally detect your DDoS in a few seconds and you wouldn't have as much downtime but hey 1.3 terabits per second in 2018 pretty big attack and that knocked them offline and the same year there was another attack this again reported by Arba the same company who produces those pew pew pew pew maps 1.7 terabits per second and that was the largest that was publicly acknowledged at that point and has been for a long time and these very large attacks seem to fade away and a couple of reasons for that were that companies like Cloudflare made DDoS mitigation of these volumetric attacks free we called unmetered mitigation from us and so people could easily get DDoS mitigation services and this is very like the world of spam where long ago you used to get a lot of spam in your email if you went back to the beginning of the 2000s this was a really big problem and what happened was service providers gmail for example hotmail they started to really attack the problem the spam still out there the DDoS is still out there but the mitigation services are so good that they deal with these things and if you have a mitigation service in front of your website or your API then you don't have to worry about this stuff so back in 2018 1.7 terabits per second that was in the news and then everything went silent until this year and just recently Akamai big legacy CDN provider said this month they mitigated a 1.44 terabit per second attack so somewhere somewhere between the github and the biggest one ever seen pretty significant I mean if you have a site that gets hit by terabits per second it's going to go down no one's got the capacity on their single website to handle that in fact no one's got the capacity to handle hundreds of gigabits per second let alone terabits per second so Akamai said that and then another service provider in this case amazon said huh 1.4 terabits per second fascinating and the reason they said that was they put out a little report saying that they had mitigated in February 2020 2.3 terabits per second so now this is now the record for the largest publicly acknowledged and I say that because there are many rumors in the industry about larger attacks on some well-known service providers who have mitigated them and never acknowledged the size but 2.3 terabits per second of traffic in February of this year quite something and there's another really interesting difference between the Akamai report and the Amazon report the Akamai report talked about nine different DDoS vectors being used simultaneously so sin attacks and reflection attacks and all of it being piled on the same time to get to that 1.44 terabits per second but Amazon it was a single DDoS vector using a protocol called cldap and they were getting 2.3 terabits per second so someone put together a pretty big tool a pretty big gun to fire that at Amazon and Amazon mitigated it because actually not complicated to mitigate if you've got the capacity and so they said it here now one of the things that's interesting in the Amazon report is they make the point that even though there was this very large attack the reality is that most attacks are in the gigabits per second they said 99 percentile in the first quarter was 43 gigabits per second now if you somebody tells you 2.3 terabits per second you might get excited about that number and think wow that's what I've got to really worry about but pretty much any website or API that is on the Internet will actually get knocked off by 43 gigabits per second if they don't have some sort of mitigation they just don't have the capacity even though that seems really small compared to that 2.3 terabits per second and this is the reality and we've looked at this and Cartha has a report out for Q1 as well this is the reality which is that most cyber most DDoS attacks volumetric ones are in the tens to hundreds of gigabits per second and they happen all the time yesterday I was talking about what I was going to say on ThreatWatch and I asked our DDoS team and they said yeah there's an attack going on right now it's 600 gigabits per second we're mitigating it that's just the reality the daily reality of DDoS attacks and that's why that anonymous chart showing an attack on the US really wasn't dramatic at all now I see LDAP which Amazon was talking about has been around for a long time back in 2017 Marek on the team wrote for the Cloudflare blog about reflection attacks and LDAP being one of them because Akamai funnily enough had mentioned that LDAP attacks were on the rise and in fact we've been seeing them since about November 2016 and filtering them out very easily let's just talk quickly about reflection attack so what's the idea of a reflection attack okay you get some service on the Internet that will respond to you and give you an answer so for example DNS is great right you ask it hey what's the IP address of this particular domain name and it replies to you or NTP which is a time protocol can you tell me what the time is currently yes and it replies to you and see LDAP which is a directory to directory protocols or finding out say okay what access does John have to these particular services hey does John have access to this service yes he does or give me a list of services he has access to yes he does now those things are all done over UDP which means you send a single packet typically with a request just like the DNS request and you get a packet in reply and here's the problem often in those attacks there are two things you can do one there's no authentication that you sent the packet with your IP address in it I can spoof that I can say hey I'm victim over here so I use the IP address of the victim I send that to the DNS server the NTP server the CL lab server I say hey give me a response and that server happily sends the response not back to me but to whoever the victim is that's called a reflection we reflect off of something and the other problem is there's an amplification factor so very often you can send a small request pretending to be the victim and the server will respond with a much larger response which means you don't have to have too much firepower of your own it gets amplified it's something that hits the target so this is very common and it's been around for a very very long time and the amplification factor is really the thing that decides how powerful these things are and what's interesting about the Amazon attack is the amplification factor of the CL that is not that great it's 50 to 70x that means if you send in one byte to the server you might get 50 bytes back and back typically you'd have to send in a few hundred bytes and you might get thousands of bytes going to the victim so 50 to 70x which means they had a fairly large network to hit that 2.3 terabits per second. I also asked the team how often we see CLDAT attacks ourselves because Cloudflare is mitigating attacks all the time on our customers you know we have about 27 million domains Internet properties apis that use us and they get attacked constantly and they're just focusing on CLDAT to see how common this is because we've been seeing it for a while and well the answer is kind of interesting so if you look at this year this the largest size of attack we've seen with CLDAT was actually in February too so somebody in February was attacking Amazon was likely attacking Cloudflare too and perhaps they had their own little service they would like to be using and it was over 300 gigabits per second so clearly a lot less than what Amazon saw but still quite something in terms of the the size of attack that you know a service would easily get knocked over by that and it's varied a little bit in May we had one over 200 and so far in June a little bit over 100 gigabits per second so we definitely see attacks of hundreds of gigabits per second just using CLDAT and this discounts the thing Akamai was talking about which is multiple vectors at the same time so someone's got a someone had a tool released in February where they could really do some damage the other thing is how often do we see a CLDAT attack well here's the interesting thing this is a graph showing the number of attacks we see using CLDAT so somebody launches something it runs continuously against some site and we can narrow that down by either domain name the domain name it's after or the IP address it's after here's the stunning thing in May we saw a new CLDAT attack every 48 seconds there were 50,000 55,000 of them in that month and so to go back to the anonymous chart this is just the sad reality of the Internet DDoS attacks are happening all the time if you have something online it is very likely it will get DDoS maybe because you annoyed somebody maybe because someone just wants to vandalize you maybe you do something political and people want to knock it offline but DDoS attacks are easy to do in the hundreds of gigabits per second and they happen all the time so go get yourself a DDoS mitigation service of some sort because you're likely to need it and there are plenty around including Cloudflare which can do this stuff for you so that was the last week or so anonymous told you that the sky was falling but it wasn't and yet Amazon and Akamai came along and said actually there are some pretty big attacks out there but the key message I think is that despite those big headline numbers that get the press excited and there were lots of stories about the Amazon announcement around the 2 .3 terabits per second the reality is there are many many tens to hundreds of gigabits per second attacks every day on websites around the world on APIs trying to knock things offline and the only way forward is some sort of DDoS mitigation and that's it for Threat Watch this week thank you for watching I will be back next week hopefully with an update on what's happening or if I skip next week maybe it's because there weren't any big threats on the Internet wouldn't that be a wonderful thing thank you so much for watching so Zendesk is one of the world's premier customer service companies, providing its software suite to over 125,000 businesses around the globe. My name is Jason Smale. I'm the vice president of engineering at Zendesk. My name is Andrei Balkanashvili. I'm a technical lead in the Foundation Edge team at Zendesk. Zendesk is a customer support platform that builds beautifully simple software for companies to have a better relationship with their own customers. We have over 125,000 businesses around the world, all using Zendesk. And then within those businesses, there's hundreds of people whose day job is to sit in front of Zendesk and use Zendesk. For Zendesk, security is paramount. And when it came to safeguarding its network, Zendesk turned to Cloudflare. Web security is very important to our business. Our customers trust us with their information and their customers' information. So we need to make sure that their information is safe, secure. The initial need for Cloudflare came back a couple of years ago, when we suddenly started to see a lot of attacks coming towards us. And all of a sudden, we'd get thousands of requests, hundreds of thousands, you know, like millions of requests coming at us from all over the place. So we needed a way to be able to control what came into our infrastructure. And Cloudflare were the only ones that could meet our requirements. It's been really impressive to see how Cloudflare's DDoS mitigation continues to evolve and morph. And it's definitely the best DDoS mitigation we've ever had. I think Cloudflare just gets you that, and so much more. And you don't have to pick and choose and layer on all these different providers, because it's just one. And they're great at all of those things. It's easy. It's a no-brainer. By tapping into Cloudflare's unique integrated security protection and performance acceleration, Zendesk has been able to leverage Cloudflare's global platform to enhance its experience for all of its customers. Cloudflare is providing an incredible service to the world right now, because there's no other competitors who are close. Cloudflare is our outer edge. It makes our application faster, more reliable, and allows us to respond to confidence, to traffic spikes, and make our customers happier. Zendesk is all about building the best customer experiences, and Cloudflare helps us do that. With customers like Zendesk, and over 10 million other domains that trust Cloudflare with their security and performance, we're making the Internet fast, secure, and reliable for everyone. Cloudflare. Helping build a better Internet. Cloudflare Stream makes streaming high -quality video at scale easy and affordable. A simple drag-and-drop interface allows you to easily upload your videos for streaming. Cloudflare Stream will automatically decide on the best video encoding format for your video files to be streamed on any device or browser. When you're ready to share your videos, click the link button and select copy. A unique URL can now be shared or published in any web browser. Your videos are delivered across Cloudflare's expansive global network and streamed to your viewers using the Stream Player. Stream provides embedded code for every video. You can also customize the desired default playback behavior before embedding code to your page. Once you've copied the embed code, simply add it to your page. The Stream Player is now embedded in your page, and your video is ready to be streamed. That's it. Cloudflare Stream makes video streaming easy and affordable. Check out the pricing section to get started. What is a WAF? A WAF is a security system that uses a set of rules to filter and monitor HTTP traffic between web applications and the Internet. Just as a tollbooth allows paying customers to drive across a toll road and prevents non-paying customers from accessing the roadway, network traffic must pass through a firewall before it is allowed to reach the server. WAFs use adaptable policies to defend vulnerabilities in a web application, allowing for easy policy modification and faster responses to new attack vectors. By quickly adjusting their policies to address new threats, WAFs protect against cyber attacks like cross -site forgery, file inclusion, cross-site scripting, and SQL injection. What is caching? In caching, copies of files are saved in a temporary storage location, known as a cache, for quick and easy retrieval. In the context of a Content Delivery Network, or CDN, a website's files are cached onto a distributed set of CDN servers. Imagine a user in Tokyo trying to access a website hosted in Los Angeles. The user's request will have to travel over 5,000 miles to reach the web server, and the response will have to cover the same distance. That can take a long time. A globally distributed CDN can cache the website's files in CDN servers around the world. This way, when a user in Tokyo wants to access a website 5,000 miles away, they can minimize latency by getting the files from a CDN server close to them.