Internet Insights with Cloudflare Radar: New TLD Metrics, Certificate Transparency, and Data Explorer

Hello André, welcome to This Week in NET. Hey, thanks for having me. First, you're at Cloudflare for over a year now and you actually joined Cloudflare as an intern. So, can you explain to us a bit of your path so far? Yep, it's almost two years now since I joined Cloudflare. As you mentioned, I joined as a software engineer intern in the Radar and Data Insights team. And now I'm a systems engineer, more focused on full stack development for the Radar product. So, I mainly work on the Radar website, Radar API, and other Radar -related projects like the URL scanner, certificate transparency, monitoring, and so on. Because, actually, Cloudflare announced over 1,100 interns for next year, so a big number of interns last year. Can you give us just how was it for you to be an intern and then actually becoming a complete part of the team? Yep. And you're based in Lisbon, for those who are wondering, you're based in Lisbon as me. We are both based in Lisbon. Yes, it was an amazing experience. So, I really felt that I was part of the team. So, I was shipping stuff to production. And basically, I was an engineer. I was an intern, so I had always the intern card, like we used to say. So, there's not so much pressure as if you are a full-time employee. But it was a wonderful experience. I learned a lot, and I still learn today with the senior engineers. And so, I recommend the internship for everyone that is finishing college or still in college and wants to have an experience in the field. Absolutely. And Cloudflare Radar, we spoke many times about Cloudflare Radar in the show, even during a very relevant outage in Spain and Portugal a few months ago with David Belson, our product manager of Radar. Can you give us your two-sentence run-through of what is Radar before we go into a demo about something that we launched this week? Yes. So, basically, Radar, it's an app, a platform that showcases trends and insights of the Internet. So, basically, traffic trends, adoption of protocols, security insights, and much more. So, it's an app where we collect data from very different perspectives of how the Internet works, and we showcase those trends for free for everyone to explore and learn from them. And there's tools there. You spoke already about URL scanner, but we have also data explorer. There's a lot there, even as a tool area for people to explore, for sure. Yep. And this week, it was the Internet resilience, making the Internet better blog takeover week, in a sense, from the research team, but also the Radar team and other friendly teams, in a sense. And you wrote a blog post about something that we launched on Radar called from.com to .anything, introducing top-level domain insights on Cloudflare Radar, TLDs. So, can you give us actually a demo and explain a bit what we launched specifically this week there? Yep. So, let's do it. Let's share my screen. We are in Radar, and in the past, we already had some insights about top-level domains, and these insights were scattered in different sections of Radar. So, for example, if you go to the email security page, you have a ranking of the most abused TLDs. So, based on the email classification by spam or malicious emails. More recently this year, we also launched the certificate transparency page, where we also have some insights about TLD distribution in certificates. So, we also have this here. And so, we decided to create a TLD -specific page with different rankings. So, one of the most visited pages in Radar is the domains ranking page, where you can see the domains that are trending today or this week, the ranking popularity, and also some Internet services trends. So, we decided to create a new page with a new ranking called the top-level domain section under the DNS section. And basically, when the page welcomes the user with this TLD ranking by a new metric called the DNS magnitude. And basically, what we do is, instead of relying on the query volume of queries for the domain, we use the number of unique networks that query that domain. So, it's better to use the number of unique networks because if there's a large client that issues a lot of queries for a TLD, using the query volume, the TLD would be ranked higher, but that would not be like a valid metric for popularity. So, we use this new DNS magnitude metric. So, we have this ranking here. We have the type of TLDs, the manager. So, this is the organization that managed this TLD, and then the magnitude. This value ranges from 0 to 10, and the higher values means that the TLD is more popular. Of course, we have .com here at the top, which is the most known TLD. Also, .net, .org, and also some country code TLDs like .io, .tv, and probably .ai. Yes, it's also here at the top. That's our country code TLDs. However, they are also used for commercial purposes. So, it's also interesting to see those TLDs here. And then, we have also a TLD information page. So, if you want to check specific information about a TLD, you can click on TLD. And similarly, like we do in domains, we have this TLD information with the information about the TLD, the ranking. If it has support for DNSSEC, the extension of DNS for security. If supports are not. And this card is particularly interesting because it shows the date where the TLD was created. Really interesting. I love to explore that. Yeah, it's really nice. But we also have a card that shows that the TLD is available on the Cloudflare registrar, if you are interested to purchase the domains with this TLD. Some DNS traffic trends for our quad-one DNS resolver. So, this is the query volume for domains with this TLD. And also breakdown by record type, DNS support, response codes, the geographical distribution. So, the countries that are issuing queries for domains within this TLD. That's a really interesting part as well, to see the distribution there. And it's worth mentioning, I'm not sure if you mentioned already, that all of this data comes from Cloudflare's resolver quad-one, 1.1.1. So, it's specific to those trends, DNS queries for those trends, right? Exactly. All this data comes from our resolver. Yes, we mentioned this in the blog post and in the page. And also we have the certificate issues volume for domains within this TLD. And this data comes from the certificate transparency logs that Cloudflare monitors. A question there. So, that's part of the certificate area that you shared before, and it was launched during birthday week, so a few weeks ago. And it was already as a Cloudflare page, ct .Cloudflare.com, and now it's on Radar. But it had this cool name as well, Merkle Town. And we actually this week had a very popular blog post about Merkle Trees as well, about certificates there as well. Can you explain to us the importance of this certificate area? Who is the people that will actually benefit from this part, really? Yes, so basically everyone. So, certificate transparency is an ecosystem created to have this transparency of which certificates are issued by whom and to the different domains and entities. So, before we used to trust certificate authorities, and we still trust them. However, if a certificate authority was hacked, for example, it could issue certificates that would be trusted by the other entities, like the browsers and so on. So, it was decided to create this certificate transparency ecosystem where every certificate must be logged into what we call CT logs. So, there are three types of entities in the ecosystem. So, we have CIs, so the issuers of certificates, certificate transparency logs. So, the CIs need to log these certificates into these logs, and these logs are publicly available, so everyone can check them. And certificate transparency monitors, like Merkle Town and now this page, which are responsible for querying these logs and checking if everything is working fine. And basically for a certificate to be trusted, for example, in Chrome, I think it needs to be logged in at least three trusted logs. So, there's much more security. So, we all benefit from it. Of course, makes sense. Regarding the TLDs part, I think you already showed any area. Specifically, there's also a blog post this week from David Belson about the quarterly outage or disruption, Internet disruptions. We can go there to the outage page, so the outage center. Under connectivity, we have our outage center, where we showcase the latest Internet outages and traffic anomalies that we detect. For example, we are seeing here the disruptions in traffic in Tanzania due to the elections, I think. Yeah, it was the post-elections government-directed shutdowns yesterday and today. And Jamaica, the hurricane also brought a big outage that is continuing. So, since Tuesday, there's a big outage in Jamaica related to the hurricane there. Yes, and we also see it in our traffic. So, probably if we just add traffic disruption here in HTTP traffic, traffic volume for Jamaica. So, the outage center is quite amazing to explore and to see for sure. The blog post that also came out this week from David Belson about the disruptions of Q3 are also represented, of course, in the outage center for people to explore, even the latest ones. And this was a busy week there in terms of outages in specific parts of the world. Last but not least, you have a favorite section part of Radar. You've been working in different new areas. We've been launching different new things on Radar. Even recently, the regional sections where people can explore states. But do you have a favorite one specifically? It's hard to say because I worked in the Radar as a whole. So, I worked in many sections. So, probably Certificate Transparency since it's the latest section where I've worked in. I also really like the Internet Quality section, which shows this one is for Tanzania. So, the Internet Quality trends for latency, DNS response time, and other metrics like speed. It's also an interesting section. And probably URL Scanner. URL Scanner, as you mentioned earlier, it's a really interesting tool for security. And everyone can use it. So, if you find a URL that maybe you don't trust or you want to learn more about it, you can just copy the URL, paste it here, and check the report. So, it's also a really interesting section. But feel free to explore Radar. A section for every person. So, if you are more interested in network, we have our routing section with BGP stuff. We also have a lot of sections for security and attacks. Bots. Bots, yeah. It's a topic really hot these days. So, we have our Bots directory. So, if you want to check information about specific bots, like Google Bots, you can search for them, see traffic trends. So, we have a bit of everything. Yeah. And the AI Insights area, also a cool one that we've been adding things recently. Yep. There may that be the crawlers, how much crawling has been going on. The purpose with training. And also how many clicks AI crawlers are actually sending to content creators, to sites, really. And also a cool one. And best practices there is also a cool one as well. Yeah. Popularity. Also very interesting. Exactly. That's close enough to give that I contributed a lot to that. Actually, before we go, one of the things that we're seeing on the news in France, actually, this week, was someone, a news outlet, going, explaining how they used the Cloudflare Radar to show that Linux is growing in France. And that was quite interesting. And they used for that our Data Explorer. So, they just went, looked at Data Explorer. They looked at HTTP requests. And also, yeah, you can go there. And also the operating system. Oh, do you want to filter by Linux? No, no, no. No need. You can do the breakdown on the operating system on top. And just put like 12 months or so. Last 12 months. Even worldwide, you can see this works really well in terms of showing you trends. You can just select Linux, for example. And you can see like the evolution worldwide. But then you can select like a specific country. France, again, because it was the one that was mentioned in the news. And you double-click on Linux, and you can see the evolution of the past year in terms of Linux. A clear evolution there that could be tracked here specifically, which is kind of interesting. Forgot to mention, put human only. Because this also includes bots. And bots are different in terms of behavior than humans. So that's different. Actually, you can see a bigger increase there. From one point something to... It's quite different there. Oh, another thing, actually, because this also includes mobile. Put the device type only desktop. Like, okay, what are engineers doing with Linux? Then the percentage is even higher. It goes from almost 3% to almost 4% or 3 .5%. So, oh, well. It's those little things that sometimes we see others using Radar and actually look at the trends. And we found interesting nuggets of trends, for sure. And they're interesting. So Data Explorer is also an interesting tool that we have. It's a really powerful tool. So if you don't find your specific trends that you're looking for in the Radar sections, that doesn't mean that we don't have it. So you can check Data Explorer. We have a lot of breakdowns, datasets, filters. So it's a powerful tool that exposes our API, basically. So feel free to explore it. Okay. That was a great demo of Radar. So many other things to see, for sure, because it's really complete by now with so many sections. But there were definitely some great nuggets there for people to explore. Thank you for doing this, Andrei. Thank you so much. And see you in the office. See you. And that's a wrap.