Originally aired on January 23 @ 7:00 PM - 7:30 PM EDT
Join Cloudflare Head of Data Insight David Belson, Director, Engineering Celso Martinho, Group Product Manager João Sousa Botto and Product Manager Stanley Chiang to learn all about Radar!
Read the blog post:
Visit the Birthday Week Hub for every announcement and CFTV episode — check back all week for more!
All right. Good morning. Good afternoon. Good evening, everybody. Thank you for joining us today on Cloudflare TV. My name is David Belson. I'm the Head of Data Insights at Cloudflare and I am joined by my colleagues João Sousa Botto, Celso Martinho and Stanley Chiang to talk about today's launch of Radar 2.0. Can you gentlemen introduce yourselves? Yeah, absolutely. I'm João, Group Product Manager here at Cloudflare for Emerging Technologies and Incubation. I'm the Product Manager for Radar and email and web analytics and web performance and a few other fun things. Celso? Yeah. Hello. I'm Celso Martinho, Engineering Director based in Lisbon and I also run a couple of projects including Cloudflare Radar, D1, email routing and some others. It's nice to be here. Thank you. Stanley? Yeah. I'm Stanley Chiang. I work in the ETI org and I'm a Product Manager working on things like the public resolver, you know, Radar and other privacy projects. Cool. Thank you. It's great to see that we're all involved in so many different projects. One of the cool things about Cloudflare is that they all ultimately come together and sort of we, you know, they're interleaved and we use all of our own products in all of our other products. But we're here today to talk about the launch of Radar 2 .0. And I know that this has been a long time in coming. I, for one, am very excited that we've got it out there. So let's start with João and I'll share the screen so we can kind of go through it as well. But can you give us an overview of what Radar 2.0 is and what maybe some of the key differences are with it between this and the prior version that we had out there? Yeah, absolutely. So first thing, Radar is a tool for people to look at the, you know, workings of the Internet, like whatever is going on behind the scenes, how is traffic changing, the attacks that are happening, what protocols are being used. Like people can look behind the scenes. We also typically add some fun stuff. Like we do a year-end review at the end of each year where we show the trends on the Internet and we do blog posts as well about trends that we see here. We've done things about back to school and about like Super Bowl, like the impact of advertising on the Internet. We do a bunch of fun stuff. So this is the second birthday of Radar. Radar was launched exactly two years ago at Cloudflare Birthday Week. And people have been using it for all sorts of things. I gave some examples on a blog post. So things, for example, we see civil society partners that have their teams on the field that notice an Internet disruption on a country, in a country. And so they go to Radar, they rely on data on Radar to see if it's really an outage or if it's just something that they perceive. You're stealing Stanley's thunder. Stanley will go more into that. But lots of people are using things. I also told you that we made some fun stuff in the past. Like we've done, we've put some sites up based on Radar data for, say, the Winter Olympics. And people come here to see, well, are there, is there an increase in cyber attacks during the Winter Olympics? Is there an increase in searches or in traffic to the sponsors of the Winter Olympics? Like things of that kind. We noticed a lot of differences on Radar. One of the challenges with Radar, with the original Radar, is that people thought that the information was showing at multiple technical depths. And so, for instance, we were showing Internet traffic and we were going straight into protocols and things that most people didn't understand. Geeks like us, obviously, we like to go deep into those things. But the general public didn't really understand that. We know that this is a tool that storytellers rely on. People like journalists and analysts and bloggers. And so this needs to be more accessible. And so the very first thing that you notice when making this more accessible is the way that we've completely reorganized things on Radar. What I mean is, here we're looking at the new homepage. And the new homepage only shows high-level data. You'll notice that for each subject, you'll see the little square with an arrow in it. And that means that you can go deeper into any of those topics. So yeah, David, if you click on security and attacks, you'll see deeper information about security and attacks. So these components, what you see at the very top, is kind of the same type of high-level information that you saw on the homepage. We call these quick bites. So essentially, it's more high-level data that you can get a quick glance and grasp some information about how things are working. So in terms of security and attacks, for instance, you're seeing a graph that shows you network layer and application layer attacks. You see UDP versus TCP prevalence in those attacks and the most popular attack types. And in this case, it's obviously a denial of service and attacks that are defended by our application firewall. Then if you drill down, you see on the left, taking most of the screen, you see the graphs that go deeper into this data. So for instance, in terms of application layer attacks, you can compare this week against the previous week. And you see the maximum and minimum and the fluctuation between those. We never show absolute values here. These things on radar, it's all aggregated data coming mostly from our public resolver. And we never show absolute values. So here you see application layer attack volume, you see network layer attack volume, you see the traffic sources that are mitigated here. And so for instance, here you see that the vast majority is denial of service, but then you'll see a big portion of application firewall. And then you see IPR, AR, BM, APSIS. You see all the different types of mitigations that are applied to the attacks that we see and defend companies against. And below you see the attack methods as well. So you see what are the most prevalent attack methods. And you see that the majority is usually UDP, but you see a lot of TCP attacks coming into play as well. One other thing that I would like to call out, well, below you'll see the application layer attack activity, which shows you the top 10 attacks by targets or by source. Like you have a toggle on top of the Sankey graph. And so it shows you, for instance, if the United States is the country that is seeing the most attacks, then you see which countries those attacks are coming from. Do notes that nowadays most attacks are botnets. And so it's distributed. It's not really like the United States is attacking the United States or that Germany is attacking the United States. It's really a visualization about how traffic is flowing, where it's flowing from and where it's going to. And something else worth noting is that on the right, like here on this page and on virtually any page on radar, you also see additional quick bites. And so the quick bites that you saw on top, they refer to this section. So just security and attacks, the quick bites that you see on the right, those are adjacent sections. So those are other types of information that fit together with this information. And this allows you to quickly shift and get more information on those adjacent categories. And users can then go and drill down on those. They can go into the adoption usage area and drill down on those, get more details, right? That's right. That's right. The way that pages are structured is typically common. There's also the ability to filter by country at the top. So where you see select the location, you can always filter any page whatsoever, even the homepage. You can filter it on any specific country. So David, you want to show us what it looks like for say Portugal? Yep, I can do that. So here's a list of locations and boom, as David entered Portugal, what you're seeing here is basically the homepage, the same one that you had seen. There's some stuff that only applies to a worldwide view, but there's also a new module here at the top that is a module for autonomous systems. So basically it shows you the ASs that are most popular in this country. They're in no specific order, but you can click down on one of them and you can also filter by that AS. And so here we're filtering that same homepage by autonomous system. So you're looking at the specific autonomous system on the Internet and you see the same graphs that were part of the homepage at any moment in time. But also if you scroll down, you see some new stuff that is specific for autonomous systems, like the BGP announcements that is at the bottom. So yeah. Cool. Yeah, go ahead. So I think I certainly encourage folks that are out there to go and poke around with the new Radar 2.0 and drill down into country level data, into autonomous system level data, check out the various sections on the side. I think in the interest of time, I'm going to flip over to Celso to talk about domain rankings, which is another new feature that we launched today with Radar 2.0. So Celso, what is domain rankings and why did we develop it? Yeah, sure. So one of the most popular features we've had with Radar 1.0 were the top domains and the domain trends. That's always been a popular content since the beginning of the project. And first of all, all of the work that we do in terms of trying to calculate the top domains, either for Radar 1.0 or now for Radar ranking is based off our DNS data, the data from the 1.1.1.1 resolver DNS. And it's really important to say that all the work we're doing around it, it's based on the premise that we are working with aggregated data, anonymized data. We don't track any IPs or anything like that. Having said this, obviously there's a lot of signal on the data that we aggregate from our DNS resolver. So Radar 1.0, initially we thought doing domain tops and domain trends would be easy. We would just count the number of queries from the DNS resolver. We would obviously do some filtering, some aggregations, but it would be an easy task. And what we've learned from working on it for over two years is that it's really not an easy task. It's actually quite complex to get quality top lists from any data source. But that's a whole Cloudflare TV segment to itself. So the big issue with DNS data is that DNS is used by humans, but it's also used in a lot of other applications from IoT devices, infrastructure, platforms. Anything that's Internet connected ultimately. Everything connected to the Internet uses DNS. And so figuring out what relates to human behavior and what doesn't is really, really difficult. Fortunately, we have a pretty good data science team. And so what we've been doing over the last six months is building a machine learning model that takes some ground truth data that we have from other data sources, some of them external to Cloudflare, and teach a model that can then be applied to DNS data, to DNS resolver data, and extrapolate the results to the whole Internet population that we have access to. And so basically what you see with the domain rankings on Radar 2.0 is the result of that. Our data science team has been working on making sure that we have a very good machine learning model that can be applied to our DNS resolver data and come up with what we think is a good definition of popularity, which is the number of people that are trying to access a particular domain, and come up with these lists. So domain rankings is basically a, first of all, is a top 100 domains ordered by popularity, which you can see for all countries. So there's a global top 100, and then you can just pick any country and you'll get the top 100 for that particular country as well. Thank you for choosing Portugal. And then we also have what we call the buckets. So if you go to the global lists again, you can see that the second part of the page is a number of files that we call buckets. And those are actually using a slightly different machine learning model, much more oriented to what we call the long tail of domains, in which we basically aggregate what we think are the most popular top 200, 500, up to 1 million domains in the largest in the largest bucket. The domains in the buckets are not ordered. And this is by design, we've chosen not to order the domains there. But since they're layered, and you can choose different sizes of buckets, you can have a pretty good idea of how popular and important a certain website or domain in this case is just by looking at that. So this is the work that we're presenting today. The timing is good because one of the most popular services that used to provide top domain listings to the whole industry was the Alexa, Alexa rankings, deprecated like one month ago or so. And so we're basically trying to replace that. And we'll keep improving the machine learning model, obviously. And we think we can do a pretty good job at filling the hole that Alexa left. Great. And how often is this data updated? So, so in the case of the top 100 domains, we're updating that those lists on a daily basis, over a seven week period. And in the case, in the case of the buckets, we're updating the buckets once a week. You can download the CSV files. So if you want to, the data is, is open, right? And it's available through our API. And you can also download the CSV. And you can use the data with any application you have that needs some kind of ranking for at least the domains. So it's all there. Fantastic. Well, thank you. All right. And to, to close things out, I want to have Stanley talk about a new feature launch that is very near and dear to my heart, the Cloudflare Router Outage Center, or, or CROC, and possibly the only, maybe the only Cloudflare feature with its own mascot. Yeah. We'll have to get it onto the site right now. It's only in the blog post, but we'll have to get onto the site as well. Yeah. Well, we'll have to stick them somewhere so that you have to search for it and it'll be a little treasure hunt. Yeah, it's a little Easter egg. Yeah, totally. So, so Stanley, tell us what the, the CROC is and why, why we have one now. Yeah. So true to its name Cloudflare Router Outage Center, this is a centralized place for us to look at Internet outages from around the world. And as you mentioned, it's a brand new page. So, you know, I'm super excited about this launch. And so as far as why we developed it, I think it'd be good to zoom out a little bit and, you know, Cloudflare is committed to building, you know, better Internet for everyone. And as part of that mission, you know, we're generally you know, as Cloudflare already looking at outages and trying to make sure that the Internet is as reliable for everyone as much as possible. But it's not just people at Cloudflare that are interested in, you know, knowing where the outages are and what's going on. People outside are just as interested. You know, we've found people from civil society organizations to journalists to just average people impacted by these outages who care a lot about, you know, what the state of the Internet is. And in order to get reliable information, you know, they're typically looking to report on them or maybe just corroborate information that they're getting on the ground. But with all this interest, you know, in the past, we've done blog posts of large scale outages. We've ramped up efforts, you know, specifically with you, David, on our Twitter account at Cloudflare Radar. And so, you know, that's all really great work. We'll, I'm sure, continue to do all of that. And this is another layer on top to give all those people that I mentioned who are interested in outages kind of a home, a best first place to go to when they're thinking about where the Internet outage is right now and historically. Great. And yeah, just to, you know, within the outage center itself, I'll maybe step on your toes a little bit here. We've got the map view, which shows based on the time period selected, where there have been active outages that have been observed. So Canada is more specifically with Nova Scotia, with Hurricane Fiona, a lot of outages in Iran recently. There was one that was observed in Cuba overnight. So you can see those on the map. And then scrolling down, you can also see more specific information on the table. Each country here links to the country page on radar. And then we also link out to more information. So generally that can be a tweet or a blog post or an article, what have you. Or a chart, yeah. A chart, yep. Ultimately, when we have the customer charts and radar, we'll be able to link out to a chart with a very specific time slice. Well, also, you know, there is an API involved where you can pull this information, but, you know, because there's a lot of users who aren't so, you know, technically savvy to try to automate stuff, you know, there's a download CSV right there. So similar to other pages, you know, you can just download what you see as a CSV file. Right. Yeah, we definitely, I think that's one of the big focuses of Cloudflare Radar 2.0 is to make all of our data more widely available. So whether it's through the API or through the available CSV downloads, and then ultimately through social sharing and embedding, which I know is coming soon as well. The goal really is to enable Cloudflare Radar to be a real source of truth for information about what's happening on the Internet now. So yeah, definitely, definitely, definitely looking forward to that. So let me. Yeah, you mentioned, you mentioned a couple of really important points. One of them is that we didn't have an API originally. So the API launched today. And as of today, all the information that you can see in the UI is also available through a public API. Right. We've essentially built Radar on top of our own API, right? We're just, we're just another consumer of it functionally. That's right. This is not just a facelift. What we've actually done is we've completely revamped the backend, the front end, the entire thing. Also with the interest of building faster, because we want, we have constants, constant requirements from internal teams saying, Hey, I have this really cool insights that my team hasn't covered. And I want to get this onto Radar. And so far we have been slower than we wanted at getting new things on Radar and Radar 2.0 was built from the ground up with the interest, with the idea of being able to really quickly bring new insights here. So as soon as we find new cool things in our data, we can bring it to Cloudflare Radar real quick. Fantastic. Good. You know that I've got a laundry list about a mile long. While we have a few minutes, I know there was a thread in the internal chats about this. Can we talk a little bit about how we're eating our own dog food with this? This is all built on a Cloudflare stack, right? Or we're using a bunch of our services. Can we talk to what's behind the scenes here or what's behind you from wherever it lives? I don't think we'll have time to cover everything, but we're actually thinking about doing a dedicated blog post about how we built the Radar 2.0 using nothing but the Cloudflare stack of technologies. So Cloudflare 2.0 uses obviously the workers as our edge computing platform. It uses pages to host all the assets that Radar 2 .0 needs. It uses R2 for storage, and it uses a bunch of services that we provide to our customers and we use it ourselves. And one of the pretty interesting things we're doing now is we're server side rendering the website completely using Remix and the workers. And so we're obviously still fine tuning everything, but you can already see a performance win when you go to radar.Cloudflare.com. And the goal is to be a top performer in terms of front end. This also helps us with things like search engine optimization and things like that. Performance gets rewarded. So it's really the future and we're really proud to be able to build Radar 2.0 on top of these new technologies. There's a few more details, obviously we don't have time to cover all of them, but we're definitely doing the blog post. John Graham-Cumming was especially excited about server-side rendering. So can you give me the one minute or two minute overview? My suspicion is that server-side rendering is sort of like what I used to do with the web in 1994, which was basically just pushing down pure HTML to the client, not making the client do all the work. Is that effectively what server -side rendering is? The Internet is made of cycles. And in the 90s, when you and me started working computers connected to the Internet, I guess you could say that everything was server-side rendered. But then as browsers became more packed in terms of features and more powerful, there was this trend to put everything rendering and running on the client browser, which is fine. When you looked at the page source, it would be like a link to one JavaScript file and nothing. There was no HTML there. So the problem when you have things like good browsers and good Internet connections is that you start abusing that. And now you've got these huge applications running on your browsers. And sometimes that's not really efficient. And the applications are too big. And then there's side effects to that as well, like the search engine optimization thing I was talking about. So the trend now is to do a bit of both. Obviously, there's a couple of things you can run on the clients and you should run on the client, but there's a lot of compute-intense tasks that really shouldn't be running on the client and can be run server-side. And so things like rendering the charts or processing data, it's really something that we can do much faster if we do those things server-side. So technologies like the Cloudflare Workers and Remix specifically allow you to do this. And we actually have a template to do a simple Remix app doing server-side rendering using Cloudflare Workers on our GitHub. And so we took inspirations from that and we built Radar 2.0 completely on top of those concepts. Excellent. Cool. We've got about two and a half minutes. So as we wrap things up, I want to just round Robin with you. What's one thing that users can expect to see in the near future on Radar 2.0 overall, on the rankings page, and on Croc? I'll start with Stanley because you're in the upper left of my screen. Sure. Yeah. So with Croc, the long-term vision is to be the status page of the Internet, so to speak. And so to achieve that, we're looking at increasing automation of outage detection. So within Cloudflare, we have a lot of internal telemetry. And so we're looking to be able to provide information about, is AWS down? Is this down? Is that down? And so, yeah, long-term vision. Fantastic. So with rankings? Rankings, we're going to keep evolving the algorithms and the machine learning model. And we want to make sure that we do a pretty good job at filling the hole that Alexa left behind. Again, we think we're in a privileged position to do so, and we're going to make sure that that happens. Fantastic. And Shravan? Well, I was trying to pick one, but I'll go with three. So I think that the three that I'll call out is, one, upgrades and user experience. Things like being able to search, having a global search box that takes you to a country, an autonomous system, a domain that allows you to drill down into any specific topic. The second one is, you mentioned sharing. We believe that I'm actually really proud of the UI, the new charts that we have on Radar2.0. And so I'm looking forward to having the buttons that allow us to share on social media. You can put this on Twitter or straight from the homepage, or you can share these things on LinkedIn or whatever your preferred social media. And you can also embed it. So if you have a publication, your blog post, anything, you can put a live graph there that always shows you the most up-to-date information. Last but not least, bringing in new data. And I'm biased because, well, I also work with email. And so I'm really looking forward to having email data here. What percentage of the email worldwide is found, or phishing, or dangerous, or what protections are being used? I'm really looking forward to that. I'm going to cut you off there because we have 10 seconds left. And in that, I want to tell everybody, you can find all of this on radar.Cloudflare.com. Follow us at CloudflareRadar on Twitter and email us at radar at Cloudflare.com if you have questions.
