This Week in Net: Making the Cloudflare WAF smarter (and Impact Week intro)

Hello and welcome to This Week in Net, our weekly review of stories we've been writing in our Cloudflare blog and things affecting the Internet. I'm John Toomey and as usual, I have with me our CTO dream coming. We're both in Lisbon. Hello, John. Hello. How are you doing? I'm good. I'm good. New week. And we don't have a lot this week, but we have an upcoming Innovation Week all about impact week. So we also will be discussing about that. And we survived the Great flood of Lisbon. Exactly. Still even this last night, still ongoing. So yeah, if you're not in Lisbon or other parts of Portugal, you won't be aware that there was a very heavy rain and to the point where there was some quite, quite large flooding and in particular in Lisbon, the campus underground station became a new public swimming pool. So crew and cars were flowing around. It was like a real big city flood for sure. It really was. It really was. Well, there we go. Okay. So we didn't have the rain didn't affect Internet too much. And it didn't. It didn't. That was good. So let's start with, in a sense, South Korea, because we have a blog post from Alex Kim that join Cloudflare to be country manager of South Korea. So let's take it from there. Yeah, I mean, so the interesting thing is, I mean, we've had we've had servers in South Korea, in Seoul for actually more than ten years. It was one of the first locations as we were building out the network. And you think about our network being, you know, getting close to 300 locations. The fact that it was the 23rd was actually early days now. And why South Korea? Well, one of the most interconnected, one of the highest uses of Internet in the world, in South Korea. And so we've been there to service our customers. Customers. Right. So the data center is there so that our customers get good performance and good security locally. But we also need local people to help sell, manage, support Cloudflare in country. And so Alex Kim joined us on November 1st as the new country manager for South Korea. And we'll be building out a team in South Korea. So, you know, another great location for Cloudflare. I mean, I think Cloudflare is growth has been in terms of service but also in terms of offices. You know, look at us, Lisbon office is one location and some of our offices are very focused on servicing the local community. So if you think about Munich, where you're handling the dark region or the German speaking countries, I think about Paris handling France and you think about now South Korea. We have Tokyo as well. Singapore. We've been in Singapore for a long time, Sydney and Australia. So Cloudflare is expanding both in terms of server footprint but also office footprint where there are customers who need to work with us closely. Exactly. And South Korea is a very specific market. And I think all of the specific markets help the global super cloud in a sense. And for example, South Korea. And Alex Kim explains here all the particularities of South Korean market in terms of types of companies. They have future oriented industries such as Metaverse. Really big gaming is really big there. So have been prepared for all sorts of markets also is like a relevant thing I think. Yeah and it's just you know important for us to have local people locally to support the many companies that are using Carrefour because Cloudflare's impact is now huge and the number of companies using us is huge. So welcome, Alex Kim to Cloudflare and I am very excited to see Seoul and I'm actually very excited because I think I might get to go visit. I'm hoping that at some point I'm going to do a little bit of a tour of our Asian offices, meet customers, meet our staff, and get a better sense for it myself. Let's move on today. Friday, when we were recording this, there's two new blog posts, mostly in general, there are related to WAF, in this case security analytics. But in the other case, why not show it already stop attacks before they are known making the Cloudflare smarter? Which one do you want to start? Let's start with this one. So first of all, you think about how UAVs work. There are there are really two things that UAVs do in trying to block attacks on webs, web applications and also APIs. And they use rules that are typically written by humans. And those rules really fall into two forms. One is a very specific rule written to stop a specific attack. There might be a specific CVA associated with something, and we did that, for example, for log four J Right. We were putting in rules to block how people were scanning and how people were trying to exploit the log four J problem. So those are very specific ones and the other ones are ones that are more general and they try to stop a class of attacks. So SQL injections or cross-site scripting or remote code execution. And this is how apps work typically, and there's some very great projects out there, more security and, and the OWASP manager toolset, which has all sorts of things for stopping attacks. But there are, I think, two problems with whats that mean that they can be made better by an application of machine learning. One is that when you write a rule for an attack and you're writing it, given the known attacks that are out there and what happens is the people trying to do the attacks themselves are adversarial and they try to get around your graph. So they look for a bypasses, as they call them, so they look for ways of modifying their attack so that it still works but doesn't get caught, by the way. And the other problem is new attacks that come along. So, for example, in log four J, you know, we know that the at least the researchers who found it had actually sort of tested it to see if it was real a long time before it became public. And so sometimes you have that situation and sometimes you have an attacker who finds something. And so how do you find something? How do you stop something that you haven't seen before or is a variant of something before? And the answer is machine learning. And so we wrote a blog post a little while ago and you can see it link there and proving the accuracy of our machine learning laugh. And what it is doing is going in and looking at the sorts of ways in which people try to evade UAVs and build a machine learning model that can spot that evasion. And that is a big a lot of work here done by the team so that we can train a model on what is a good request and what is an attack request. And we actually do this for classes of attacks. So SQL injection, cross-site scripting and remote code execution and certain actually specific attacks. And so we improve the WAF by using machine learning to figure out how we can block things that are variance essentially of the rules as we've seen before. But then there's also the issue of zero days or unknown attacks which come along. And so again, the machine learning graph can spot things that are similar to or new variants of or the types of attacks that we don't necessarily have a specific rule for. And one example of this is actually in October there was a C.V. on Apache Commons text and it showed up on machine learning where picked it up and said these requests are malicious. And so we are now exposing this ability in a thing called the WAF attack score and actually does a graph. If you scroll back up, you can kind of see what has happened. You get this attack score where when a request comes in, it'll either be a low value which. Is this likely to be an attack or a high value, which is likely to be not an attack? And within our rules engine, you can then write a rule which is if the attack score is less than ten, then let's block the attack or let's let's do a challenge against the against the attack. And if it's greater than 90, well, we don't even do any further scrutiny, which or this is good. These sorts of techniques have been used for a long time. I used to do this for spam filtering back in the day when we were doing spam filtering stuff. You have this score and you have some thresholds and this is now incorporated into our system and if you scroll down, it's incorporated and shows up in our security analytics. So in security analytics you can look at the sorts of sorts of things we're blocking, how we're blocking it, and then you can write rules and you can see it here. The bot score is incorporated down there in the in the in the corner, and then you have the attack score as well. So this is new. It's newly available to everybody. They can use the machine learning. Obviously we keep retraining the model. We learn about the sorts of attacks that are out there and the best way to block them. But this really, I think, adds a huge feature to the web because it enables us to spot the things that are attacks before they're really known about by the general world. And they fit into this new security analytics model, which is the other blog post which talks about security analytics, where really, you know, if you used Cloudflare a long time ago, our analytics for a website or an Internet application, we're kind of poor, to be honest. There was like a few basic graphs and people were always going off somewhere else. And actually we've made an enormous effort to introduce real time information into the dashboard to allow you to cut and slice the data in the way you want. You can still export it, of course, if you want to use an external tool. But the attack score is getting incorporated into the security analytics. And the other blog that's coming out really shows you how those analytics were. They can give you a really comprehensive view of what's happening. And it's particularly important if you're onboarding a new application where you're not quite sure what is perhaps normal traffic or what the profile looks like. And so you can actually throw it into here and then pull up the analytics and we'll say, okay, well we think this stuff is bad behavior, we think this is good behavior. And you can then start going in there and deciding how to tune it for your particular application. I think you can test it, right? You can move the attack levels if you think it's too much or too less. Absolutely. Absolutely. You can see the impact in the charts that are in the available through here. So this is we've had this style of operation in bot management for a while, which is where you look at your traffic and we identify the bots for you, we identify what we believe are problems and then let you tune it. And now it's available in attack analysis too. So I think you know this in addition to the fact we're doing machine learning to actually spot attacks before they're known about it really makes a, a huge, huge uplift on the WAF. And, you know, it's no surprise that so many people rely on it. So that's out today in an hour or so. By the time you see this, this will be out. Exactly. And you can read all about that and how to use it in the sense machine learning is there, but there's manual. So you can leverage machine learning to your own personalized needs, which I think is great. Absolutely. And, you know, we've done that actually a little bit throughout throughout Cloudflare. In fact, there's a there's about there's approximately 50 different machine learning models that are used throughout Cloudflare, which are doing analysis for stopping attacks and predicting performance and all these kinds of things. And we're exposing those things to our customers so they can they can use them in their, in their particular configuration. So yeah, how about it? This is, you know, I think this is a big improvement on the WAF and you're going to see you're going to see more and more machine learning things get exposed. We probably start saying I now because it sounds cooler, but machine learning stuff get exposed, which I think is making making the product even better. Exactly. Exactly. Before we go to Impact Week, that's coming next week, you have a lot of experience in terms of building WAF from the start, from the get go, like more than ten years ago. What do you think in terms of evolution of this area, what that evolution has been since building something from scratch like ten years ago to now? What is the main evolution? One of these one of these things we mentioned is one of those, for sure. But like the overall evolution of that area. Well, I mean, I think one area is that this goes beyond web because APIs are such an important part of what we do. Right. And if you. About over the last ten, 11 years, then the rise of Web applications that are actually a sort of a frontend on an API has really, really happened enormously. And therefore APIs need protecting because they're actually in some sense the application. So why is applying to APIs is important And in fact you then have to handle slightly different sorts of traffic. So actually right at the beginning in Cloudflare as well, there was code to do analysis of XML because XML was being used somewhat for APIs. Now of course JSON is enormously important, right? And in fact one of the reasons to do the machine learning thing is the, the JSON structures they're getting passed around might be very complicated. And actually using a machine learning approach helps us to detect well within this structure. There's actually a SQL injection here. So, you know, these the world gets more and more complicated and I think machine learning is a very important technique for us being able to catch things in that in that very complicated environment. And, you know, yes, manage rules are great and we will never stop updating them because they're super important. But machine learning is really is really important. So it's building on what was done before and like creating a new level of protection there for sure. That's funny. I'll tell you a funny thing. The other day somebody asked me internally if they had any photos of when I joined Cloudflare because it was it was 11 years ago now and. That's why I was actually telling you happy birthday this week. 1112 So and one of the pictures is of my spare bedroom in London where which was the original London office in a sense of Cloudflare. And in that on the table is a printout of a paper about anomaly detection in web traffic using machine learning. Because 11 years ago I wanted to do this kind of stuff and well, other things became higher priority and weren't so important. In fact, it has always been the case that Matthew Prince, who hired me, has made fun of me about this, because one of the reasons why he wanted me to join Cloudflare was I had done machine learning for spam filtering and he's like, We should do the same thing for the WAF. Well, there were lots of other things to do at Cloudflare, and luckily we have a big team who works on the WAF and they've done the machine learning. So I'm going to say a big thank you to the WAF team who have saved me from Matthew's wrath. 11 years on. He was already in on the plans. Right? And now it's it's it's really there. Yes, it's here. Last thing about WAF, you build the WAF in the beginning and a fun fact in building that in the beginning, like in the first days. I mean, it was. Difficult. You were doing like working in London, in London and using video conferencing at the time, 11 years ago. I mean, I think the thing I think the thing that was interesting about the RAF, you know, 11 years ago was that I, I looked around at what the state of the art was, and the reality was Cloudflare was using mod security, it was using Apache and Mod security in this really weird configuration to get the WAAF and you know, there was. It just didn't perform what we needed to do at our scale, and it was very hard to get it going and we couldn't do everything. So I what I actually did was I wrote something that was completely compatible with the mod security language. It's actually for a very long time, the way in which the Cloudflare would have worked was you could write, you wrote a rule in the Apache configuration language as if it was an Apache configuration file and my thing read it. And luckily, some very brave folks took the part that did the parsing. I rather grandiose the called it a compiler. And what that thing did was it converted Apache configuration files into Lua code and it was an entirely undocumented Perl script. And I you know, congratulations to the the team that went in and dealt with that script. And now things are a lot better. But I did do a talk about this actually some years ago. You can find it online and talk about how the Cloudflare work worked. And I think the interesting thing was that it was it was outputting Lua code which ran in Lua jet and we tuned Lua jet to really, really, really work extremely, extremely fast. And it's only recently as we've really got up to scale that we've had to say, Look, we need to go even faster, we need to process even more requests per second. And now everything's written in rust, so. Different things, but I can see creativity and dealing with unexpected things there, which I always found interesting. Let's move on to Impact Week. We have a new Innovation Week next week or with different announcements, a lot of blog posts next week. What are like the main topics we can disclose at this time without like being very specific to them? Yes. So we're going to talk about impact. So what is impact? It's social impact. It's environmental impact, It's governance. So we're going to talk about things like sustainable office design, recycling of our equipment, and how we're dealing with our emissions, how we're buying clean electricity, how we're dealing with our historic emissions, how we think about sustainability overall. We're going to talk about impact. We've seen you often look at this in terms of the Internet use. So we're going to look at what's been happening in Ukraine and give an update on the war in Ukraine from a from a network perspective in Iran where there's these ongoing protests. Right. Against the government. What's happening in Iran in terms of traffic, in terms of Internet outages, blocking that kind of thing. We're going to look at how we participate in Internet exchanges to make the Internet faster. Everywhere we have a thing called Project Pangea, which is a way of giving community networks free access to the Internet so that if you're setting up a community network, you can get an on ramp onto the larger Internet. We're going to talk about our ERGs, which are our employee resource groups, which are a variety of groups internally for people from different backgrounds, from different gender expressions and things like that. So lots of different stuff. One of the things so I mean, there's so much of it's great and I don't want to try and pick my favorite child out of all of these blog posts, but you've been working on with some other folks with particularly with David in the US on a year in review. So there's going to be the sort of Spotify wrapped for the Internet, right? Which is we're going to sort of talk about some trends on the Internet in terms of traffic, in terms of blocking again and in terms of also ranking of websites as well. So yeah, and that's a big interactive website. So you'll be able to zoom in and say, I live in Portugal, what's happening in Portugal? So. Exactly. Yeah, it's a big project. I've been working with the Belgian and all the radar team for a while now and for me it's like amazing to see first how global our super cloud and network is, how global our DNS resolver is. That is what is giving us some information for us to do, like a proxy related to websites or create like a ranking of popularity domains or platforms. We're calling platforms this year because it's like the A name and not a specific domain. We're aggregating different domains and we had a lot of attention last year because we published a blog post was not on the micro. The DNS ranking popularity websites thing is not on the micro side. It was on a blog post. We got a lot of attention last year because ticktalk in our metrics was last year. Number one. Our yeah, our metrics last year were pretty much related, more closed. It's never really the same thing, but it's more close to page views or interaction with a website. It's someone interacts a lot with a website and this is of course natural when you're doing like swapping videos all the time. In one minute you do a bunch of swapping. Let's create more interactions and this creates more DNS queries. So websites that have that have more prevalence in our DNS queries type of information that we have. So that's why last year, for example, TICKTALK was number one, and I don't want to say what it will be this year. That's for next week. But this year there's a new method in town. And the other thing is, right, we've we've spent there's a blog post, we spend a lot of time on the ranking in order this big another big machine learning model right is DNS data and predict what that means in terms of the use of a particular or a particular application or website. And so now ranking. So it's going to be a little bit difficult to compare last year to this because we've actually changed the methodology. And so we're I think the updated methodology is a little bit more stable in some ways than the previous methodology. It should give us good results over a long time. But yes, there is there is there is a new sheriff in town. True. From machine learning. To your point, it's more than one model of machine learning. There's a bunch of them now. Yeah. Yeah. And now it's more close to Nick visits. So that bias, hopefully of more close to page views is now more close to visits. It's neither one It's like. In the middle all the way. But it's a new perspective for this year. But for my perspective, I was writing a blog post, really long blog post with all sorts of trends related to war in Ukraine. News organizations. Things like. Mastodon. Mastodon. Mastodon. Right. Twitter. Twitter. Alternatives. Mastodon is clearly there. I'm hoping also to have open AI related trends there. Fingers crossed. And of course, there's going. To be a graph that goes like this, right? It just goes vertical, right? It goes. It goes really right. And of course, just the state here. And why not? We're focused on a ranking. So sometimes it's not related to traffic. If a website has more traffic, is the position in our rankings. It's a position, Yeah, yeah, yeah. Other things, it's moving around in the ranking. Yeah, that's right. It is. But a lot to uncover there. And in terms of internet traffic, we have all sorts of countries there represented. You could see internet traffic growth over the year, some later part of the year increases of traffic, which I think are interesting. And this goes to the point, for example, in the northern hemisphere, where it is more there is more people for sure. So more Internet usage. It's the holidays. This Christmas, it's like cold at this time of the year because it's winter. And because of that, you could see in all of those northern hemisphere countries, Internet rising because Black Friday week, because people are more at home, they're used more the Internet for all sorts of reasons. So I think that's a really interesting with that a tax also rise. So a tax is also a good thing to see. But we also have really interesting data on, for example, categories, for example, travel, for Internet traffic. In terms of when was travel higher this year, when was e-commerce higher this year, We have categories like that and a proper microsite for people to see. So interesting things there. Before we go to I want to share I'm going to share a quick image because I was interested in this. This is this is from Cloudflare Radar, which you work on. And this was Internet traffic in Peru on Wednesday of this week. And you can see a comparison of the day and the previous 24 hours. And so it's running along, looks pretty normal. And then there's this weird spike. And if you look at that weird spike, you might say, wait a minute, there's something wrong with our data or what's going on or is that an attack? And no, in fact, there was a sort of, well, in quotes, attempted coup in Peru, where in fact the president has had to has had to quit. And this spike corresponds to, I think, when the news of this coup broke. So it looks like later in the day, everyone in Peru went online trying to figure out what was happening with the government. So it's just one of those great little examples of how you see a little thing like that. And I saw it and I was like, wait a minute, is there something wrong with our data or is that something broken? And it's like, nope, that's actually the population. And if you look at it, what's interesting about it is it's the highest peak in the 24 hours. So even if you look at previously the peak was relatively goes up in the daytime and goes down at night and suddenly, you know, it was swimming along there to kind of a normal level. It peaked all the way up to the max of the day. So there was an enormous amount of interest in what was happening and people using their phones, presumably to figure out what was happening. What is happening. Yeah, and it's really interesting to see data, our data showing like real world examples like that. We've been seeing all sorts of different events, which makes sense even for the World Cup. We have been seeing like when the national team of one country is playing, you could see in some cases Internet traffic going up or going up in the half time. So in this case, it's a really important event in the country. And sometimes this is exactly at that time where that peak is. So it's always it was. Interesting because sometimes it's it's smoothed over time. And I think what happened was the news probably broke and everyone was like, well, we're going to find out right now what's happening. And then things carried on a bit. So that's true. That's true. But actually, why not show we still have a couple of minutes. Why not show here a problem that happened in this case related to the west? Yes, North Carolina. So there was a clear drop because there was gunshots targeting critical power infrastructure. Yes. So shot at substations. Right. Is this what happened? And they wanted to power out in a particular location. So. So you could see, like in this case in this area, the US Internet traffic going down because of that. So here it was a trend of the week in a sense. Yep. Yep. Now you see. Is that in that area completely lost. Yeah. Yeah, exactly. So a lot to uncover and still next week just to do like a last summer. Also we have Galileo's some trends about our project Galileo next week related to how we are helping Ukraine customers and even in this case, organizations being online and avoiding attacks. So what is the in 50 seconds the sum up for next week we want to do. And read the blog. Exactly. A lot of there'll be a lot of blogs every day. So I can I can give my input here. I think I'm still amazed reading some of the blog posts that are being written of the impact, the good impact. May that be in Internet and the Internet community in a sense, but also in all sorts of areas that Cloudflare is able to to have in the world, which is always interesting, is a good way to end the year. I think to give thing show a little bit of different side to Cloudflare into general that we want to put out there. But yeah, you get a you get a sense for all that we do. Right. Because there's a lot every day so. Absolutely. Exactly. So be close to blog dot com next week. Exactly how come it's coming Monday. Many, many blog posts, so... Exactly. And that's a wrap. Thank you, John. All right. Well, thanks. Very good talking to you.