🔒 What Launched Today - Friday, March 8

Hello, good morning, good afternoon, everybody. My name is Daniele Molteni. I'm a product manager here at Cloudflare in Application Security.

And today with me, we have two amazing hosts, amazing guests, sorry.

One is Alex Moraru, which is the PM for Threat Intelligence.

And with her, we have also Sofia, which is a system engineer for the radar team.

And we have two great blogs we want to talk about today as part of Security Week.

We launched one amazing feature, which is the URL scanner, which is now part of the Security Center.

And then we have also another feature, which is the request for information and priority intelligence requirements as part, again, of Security Center.

So we're going to go over both of them in a bit, in a little bit.

But first, I wanted to ask Alex about URL scanner. Tell us a little bit more about this feature, about this launch.

Hi, and thanks, Daniele.

Hi, everyone. I'm really excited, actually, about this launch, because Sofia and I have been working a lot on this in the past few months.

And why I'm so excited to announce these enhancements in Security Week is because URL scanner actually was launched exactly a year ago in 2023 Security Week of Cloudflare.

So it's really nice to see things coming around. So we've gone full circle somehow.

So who knows what next year will bring about URL scanner, right? But bringing it back, URL scanner, this is a functionality that came in as a response to some of the needs of our customers and of the general public, actually, to better understand the safety of the URLs that they were accessing, right?

So the service was initially launched, as I mentioned, a year ago, and it has already almost a million requests for scans this year, which, considering that the users were accessing this via one of our platforms, so Cloudflare Radar, I think it's quite an accomplishment.

And what we're adding now is a lot of new functionality. So, you know, the ability to refine the visibility of your searches, I think one important enhancement is the fact that you can do unlisted scans now, and there are various use cases for which you might want your scan to be unlisted and not visible to the general public.

We're showing new security headers, we're, you know, allowing users to access their previous scans in a very friendly manner in the dashboard.

We're also introducing all of this functionality of URL scanner in the Cloudflare Security Center.

So while we did initially have a URL scanner in the Investigate portal, I think adding the existing URL scanner that was present in Radar allows us to have much more and much more interesting functionality for our logged in users.

And Alex, can you, can I stop you there?

Can you just tell us a couple of words about Security Center?

What is it and why, and who is the target user for that? Yeah, yeah.

And thanks, thanks for asking about this. Security Center is a place in the Cloudflare dashboard where our users can go in and check what is their security posture.

It's, initially, we are looking at providing the users with much more visibility around what are their Cloudflare products configured, what can they do to improve that security posture, and we're providing them with tools, especially threat intelligence related tools, in order for them to be able to investigate more indicators of compromise, for example.

So users who are familiar with Security Center are using our Investigate portal to go ahead and check, you know, IPs, ASNs, domains, and try to understand more information and details about them.

And that's where we also added the functionality for URL scanner for a user to go ahead, check a URL, scan it, and learn much more information around its network, its security, and, you know, the categories associated with it, which leads to a malicious or safe verdict, for example, and much more information.

So it's a great way, basically, to target a specific URL, specific page, and check the security and the health of that page.

And can a customer that has access to our scanner scan any page on the Internet, or there are some limitations?

That's an interesting question.

We can start the scan for any page, basically, on the Internet.

But again, we can't guarantee that the scan will work. And actually, Sofia, maybe you can provide a little bit more information about what scans will maybe be rejected for the time being.

Yeah, so for example, if you try to scan a domain that doesn't exist, the scan will naturally fail.

And that's one of the most common reasons.

Another is our URL scanner is a bot. So any website that denies our bot can also reject our scan.

We try to be mindful and respectful of everyone's requests regarding scraping.

So that's what we're trying to be mindful here. Yeah.

Sounds great. Thanks. So yeah, let's go back to what you were describing about the URL scanner in the future.

Yeah, so I think one thing that I would like to outline about URL scanner and the changes that we're introducing now officially, is the fact that we actually launched the API.

So the API has been available for a while now.

And we do have some power users and some early access users who have been testing the URL scanner API.

But I think it's really important to outline now that this is not officially launched.

And we're providing this access to Cloudflare users. Because while one can understand, you know, the use case of going to a dashboard and inputting a URL for scanning and getting that information, you can think of situations in which your volume might be much higher, or you might want to manage things differently than having to go to a dashboard, copying it in, and seeing the responses, right?

And Sophia here is who worked on the API. And she's the mastermind behind this.

So I'm very glad that we have her here. And maybe we can actually talk more about the API functionality and what it relies on in a few minutes.

Yeah, that would be great. But why don't we first show a little bit how it works, maybe in the dashboard?

Is that something we can do? Yeah, actually, why don't I do a little demo?

Let me see how I can share my screen. I'm going to share my screen here, and make sure that we do a little demo from scratch.

Yeah. Can you confirm that you're seeing my Cloudflare account?

Okay, perfect. So what you see here is an account, a test account that we have.

I'm going to go to Security Center, which we've been talking about, the Investigate portal.

And that's where users get to see the Investigate portal and the ability, you know, to understand what can they investigate.

I mentioned really quickly, IP addresses, domains, URLs, ASNs, and of course, URLs.

I really like the fact that users can check what are their latest URLs that have been scanned from this account.

But they can also see all global scans.

I think an important value of the URL scanner is the fact that users can see URLs scanned by other users.

Of course, if these users decided to do the scans publicly, right?

And it's important because then maybe you can see something that you might have not thought about.

You can try to understand why do I need to do another scan when I can just access something that's available.

So let me try to search, to scan another URL. So what you will see here is the fact that this URL hasn't been scanned in the last 30 days.

If it had been scanned in the last 30 days, you would see here a report that you would be able to click, or you would be able to perform a new scan for this specific path, right?

I can choose to make it unlisted or public on the Cloudflare dashboard.

This is by default unlisted. And what that means is that only users of your account will be able to see that this has been scanned and what's the result and access it.

However, if you choose to make it public, then other users will see it.

This will also be displayed as part of the recent scans in the Cloudflare radar as well.

So why don't I, yeah, I'm going to just make it public for now.

And I'm going to click on scan. You'll see that there are different stages of the scanning process.

We're queuing it, we're processing it, then we're doing a lot of magic post-processing that Sophia will be able to talk about a little bit more.

You can see that the scan will take a few seconds, mainly because there's a lot of information to go through, right?

And once the scan is done, you will see the report that is loaded here directly, confirming that this is what we have scanned at the submitted URL.

This is a report as well. Of course, if there had been a redirection, maybe you would see the redirection and the fact that the scan report is for something slightly different than what you input.

But that's what it is.

We have the ability to understand what is the rank of this domain. So clicking on this link will lead you to the Cloudflare radar page talking about domain ranking, timestamp for when the domain was, when the URL was scanned, the category itself, the verdict itself.

So this is, this is marked as not safe, mainly because the category of anonymizer is a security threat.

So again, users can still go and access this if their network is not blocking access to such URLs, but we advise a lot of caution.

It's important to note that users can also see what is a screenshot for this domain.

So you can see it here. This is literally the screenshot of the first page.

You can see this in various different formats. So if you're accessing from a mobile, that's what it would look like from a different size of screen.

And then beyond, I think this is really interesting. We have a summary that talks a little bit about, you know, links, cookies, variables, messages, the technologies that we've identified while scanning this URL, again, with a short description.

And then if the user wishes to, they can go to the different tabs. So you can see the security tab that goes over what are the risks that were found.

So you can see here that this is a security threat due to it being labeled as an anonymizer.

No content security policies, because this wasn't set. No certificates found.

And of course, if you scan a URL that has the information detected, we will be able to see it here loaded.

I mean, I don't want to go through absolutely all of the tabs, but in the network tab, it's something really interesting.

I mean, you can see all the HTTP transactions, the DNS records, which are quite interesting to understand.

And you can just load them as they come. Same for the behaviors.

If you think of, you know, the cookies, the JavaScript variables, the console load messages.

And you can also access, you know, the raw body.

Sorry. Maybe I found the bug. I'm surprised. There you go. So I'll actually...

Great. And I just wanted to mention something. I know the CSP, what you mentioned, the content security policy.

As an application security PM, it makes me very happy to see that in here.

Because, for example, we have other products that leverage that technology.

And for who doesn't know what a CSP is, it's basically a policy that can be enforced on the browser side.

Like an application can basically send headers and tell the browser, you can only accept JavaScript from those domains, for example.

And so you can restrict and kind of lock down the browser environment.

And for example, a product we have, which is called PageShield, allows to set those CSP.

So because we are a proxy, Cloudflare is in the middle.

Anytime there is a request, when we send a response, we can add the header of the CSP and enforce that behavior on the browser.

So it's great that here we can double check whether those CSP are actually properly set and can create a security layer on top of everything else.

Yeah. And on that, it's important to note that actually what we can see here is the CSP in this particular example.

But we've added other security headers that we are detecting. So users will be able to explore that as well if they want.

Maybe one last thing that I can ask.

Again, this is what we've just scanned. If I click on it, you can see the verdict.

If you want to search for it again, you will just see the previous report over here.

So you don't need to scan it now. Again, you can just access it after you search for it.

Yeah, this is an amazing piece of technology, and especially that it's part of Security Center.

So it gives additional tools to anyone who wants to verify the security of pages.

It gives an additional tool to do that.

It will be great to dive a little bit deeper into what technologies we use to build this.

And also, you mentioned the API is actually a big part of this launch.

So I'd like to ask Sofia to walk us through how we built this.

Also, how should we think about the API? What's new? And give us a little bit more context there.

Yeah. So I'm Sofia. I'm a system engineer. And yeah, it's been super fun to develop this API.

We built this all on top of Cloudflare products.

So it's Cloudflare Workers, it's Cloudflare Queues, R2. And let's say the powerhouse of this all is the Cloudflare Browser Rendering API.

This basically a fleet of Chrome browsers that are just sitting there waiting to launch a browsing session.

And this is a REST API. So it's super easy to use it to build products like this.

It's available to all developers as well. We just use it as well internally to build this product.

So it was super quick to do it. Initially, the initial plan a year ago was to, let's set up a cluster in Kubernetes of Chrome browsers, and let's manage all this.

And then, hey, no, we have this internal product that we could do.

It wasn't public at the time, it became public while we developed this product.

We could use this and make this a product. So at the time, it was really fun.

And it's really spread up to development at the time. So yes, what do we use?

We use Cloudflare Workers. So those are the pieces responsible for the API, since as Alex said, we're now API first.

So that's definitely a big, big plus.

And then we use Durable Objects. So Durable Objects are basically a special kind of worker that stays alive beyond a single request.

And they're responsible for the orchestration.

So as Alex showed, a scan has several processes. First, we queue the scan, which basically means we do a call to the worker and the worker itself issues a call to Durable Object, which is responsible for orchestrating the entire scan.

And it says, hey, queue this scan. And then the Durable Object says, okay, I'm going to set an alarm and I'm going to wake up in three seconds, but I'm going to answer now to the user.

So it immediately says, cool, I can scan this. So a 200, okay.

And then about a few seconds later, the Durable Object wakes up, okay, I have a scan queued.

So now I'm going to go through the entire process. It's at this time that we actually call out to the browser rendering API.

We say, hey, I want a browser.

And now I'm going to use, once I have a browser, I'm going to use Puppeteer, which is a very well known library for controlling a Chrome instance.

And then we puppeteer the browser and say, hey, go to this URL, run this code, and get the console log messages and other stuff that we show in the report.

And then we actually produce a HAR, which is like a serializable format, which represents the entire scan and the entire browsing session with all the HTTP transactions, all the cookies, all the requests, etc.

And that is actually what Alex was showing.

When we're looking at the list of HTTP transactions, that is basically the HAR log.

And then after we do that, we now go into the post -processing phase, where we use a lot of internal and public as well, Pulseware APIs.

So one is the RadarRank, as Alex has shown. And another crucial one to this product is the domain category.

So we want to see, hey, is this domain safe to visit or not?

And we're going to add, as time goes by, even more APIs. So we're going to enrich this report even more as time goes by.

And we also get all the technologies that we've identified that the website uses.

So that is all done in the post -processing phase by the DurableObjects.

Once that's done, yes. So quick question. So having the DurableObjects and the worker allows you basically to run the scan async.

So you still perform, right. So that's the trick and the way you maintain a good user experience, because then you don't have to basically wait or hang the screen or the interaction with the user and wait for the scan to happen, right?

Is that correct? Yes, exactly. So we immediately say to the user, yes, we've accepted your request, and it's now queued.

So that's when Alex was showing the several phases.

That's when it goes from the request to the queued phase. And yes, as you said, that's what allows us to respond immediately to the user, but actually do the scan asynchronously in a separate thread.

Let's say, let's put it this way.

And do we have a single thread for all scans that are initiated by Cloudflare customers, or is like one thread per account?

Well, I was mentioning thread in an abstract way, not in the sense of a thread in a server or in a computer.

So yeah, not thread in that way. So it's all based on Cloudflare Workers. So what happens is that each scan gets its own DurableObject, which is basically a process in Chrome.

And then the workers just basically register the request, and they invoke that scan's DurableObject.

So I've got this request, it's the ID X1000, I'm going to launch a DurableObject with ID X1000.

And then that DurableObject orchestrates the entire process from crawling to post -processing, and then to actually saving the scan.

So what we do, we also use R2, which is basically where we store our scans.

So R2, it's the storage solution. And then at the end, in order to empower search, we're actually using Postgres.

So instead of sending every single, each time we get a scan, we would insert it into Postgres, that wouldn't be optimal.

So what we do is we send it to Cloudflare Queues.

And when we have enough scans, we send it all in a batch to Postgres.

So that's why Cloudflare Queues was very useful, because it allows us to optimize this performance a bit.

And just everything that was batched, we could basically batch insert the scans for posterior search, basically.

Yeah, so that's the entire process. It's Cloudflare all the way down, turtles and Cloudflare all the way down.

That's great. And let's, let's finish off with one question.

One more question for Alex. Can you tell us a little bit on about who gets this feature?

And what are the future plans? What are we doing next?

Yeah, yeah, good, good question. So I think it's really important to note that the URL scanner remains free to the general public, because this is this is a tool that will stay in our, in our Cloudflare radar.

But some of the features that we've just launched, are only available to logged in users.

So to on or via the Cloudflare dashboard or via the API.

So that's why it's important, you know, we encourage customers to access URL scanner, either on radar or via the API or in the Cloudflare dash, but it depends a lot on what is their what is their use case.

Also, some of the some of the plans that we have is are related to launching features like scanning from a particular geography or search, searching by hashes or by other IOCs.

And those will be those will be first deployed in our Cloudflare dashboard.

So for our, for our subscribers, right. Of course, we're also we're continuously working on, you know, improving the rate limits on performance enhancements and all of these things, because we need to we need to keep up also with all of the all of the new functionality that we're that we're adding to the URL scanner, because ultimately, it's important to make sure that the user experience remains really positive about this.

Yeah, that's great. Thank you very much.

And that's all for URL scanner. So if anyone is interested in learning more, you can visit our blog.

We have released the the the blog today. So you can you can read it fresh from the press.

There is another blog, though, that Alex is still the owner.

So congratulations, Alex, you have launched a lot of new features this this week, which is very busy for you.

You are introducing request for information, RFI and priority intelligence requirements, or PIR in the threat intelligence, as well as part of threat intelligence.

So can you walk us through a little bit through this feature as well?

Yeah, and maybe it would be interesting to to note, to talk a little bit more about our threat Intel offering, right, because this is That would be great.

much more narrow audience here, right. And while these features sit in Security Center, so it's really been a big week for Security Center, considering all of these announcements, but our threat intelligence customers are also consuming that threat intelligence via the Security Center.

So Cloud Force One is what we refer to, to our threat operations and research team.

So they, this team is focused on tracking and disrupting any threat actors that are targeting both Cloudflare and the systems of the customers that we serve, right?

So in the way that they work, first of all, our Cloudflare security team basically is our customer zero, right?

We really need to make sure that Cloudflare is safe, it's secure, we're aware of the threats around us, and we continuously conduct investigation around that.

And the team have been able to publish a lot of interesting research recently about what's been happening in the world in general.

But one other responsibility of this team is to actually engage with our customers when they have very specific questions about the threats and the threat actors that are posing a danger to them and to their systems, to their network, to their use, to their employees, right?

Can you give me an example of that situation or scenario where...

Yeah, exactly. So maybe it would be interesting to give you an example of a request for information, because that's what requests for information and PIRs come in quickly.

A request for information, or as I refer to them as RFIs, basically, this is a service that's allowing customers to query our intelligence and our threat intel analysts, right?

Because they want to request some detailed information about some threat actors, some IOCs or indicators of compromise, about some campaigns that they've been seeing or any malicious activity observed, really, right?

So that's why they will request from us a report or a list of indicators of compromise or analyzing a particular traffic pattern, right?

So maybe an example is, imagine you're a company in the energy sector, let's say, right?

And then you see some strange, some suspicious network activity on your logs, and that activity is suggesting some sort of potential breach, right?

And then you can do a lot of preliminary analysis internally, and you can check some indicators that could be associated with some ABTs, also with some advanced persistent threats, right?

And a group related to that.

If this group is known for targeting companies in your industry, so in the energy sector, as we mentioned, maybe they are known for disruption, for espionage.

And then the customer, your internal security team might not have the right expertise or the right visibility into what other activities these threat actors are doing, you know, in your industry, in the world in general, right?

So that's when they would reach out and ask the Cloud Force One Threat Intelligence team, hey, can you provide me with some detailed intelligence about this ABT group?

Can you tell me about what are their tools, their techniques, their known objectives, who have been their targets in the past, right?

What behaviors should I look for in order to make sure that I can attribute the campaign towards them, right?

So what our team would do is they would provide all the insights and the TTPs, so the tools, techniques, and procedures that are associated with this particular group, and especially focus on any novel techniques that they might be using, right?

Because they always evolve, right? We would surface any malware that they are known to be using in campaigns or distributing.

And then, which is probably the most important part, is provide recommendations to our customer.

How can you detect?

How can you mitigate? So what strategies do you need to employ in order to mitigate against this threat so that you can best protect yourself, right, and improve your defenses against such threats, right?

That's, hopefully, this is a clear enough example of how we would engage with the customer in that sense.

And what we launched today is just making that experience easier, right?

We literally have a platform where the customer can provide all of that information, ask the questions, request the type of output they would like from us.

Because sometimes it's just a list of known indicators of compromise. Sometimes it's information about a specific threat actor, or maybe it's a binary analysis of some sorts.

There's various ways of engaging with our team. But this new dashboard and this new way of communicating with our team is making things easier both for the customer and for our analyst team, honestly, to be able to have one place in which we can capture all of these requirements.

And that's the gist for requests for information.

Yeah, I guess it's great. This is an additional, we're basically showing an additional way our customers can interact with our security analysts and our security teams, which are busy, of course, looking for signs of compromise or, for example, to create new protection, new rules.

We have also in the WAF, we have an analyst team creating new rules.

So we're really trying to build here, I think, a bridge between the customer and who is working at Cloudflare to improve security for all our customers.

So this is exciting. Yeah, maybe I can do like a really, really quick overview of what are the PIRs or the priority intelligence requirements, because I've been talking a lot about RFIs, which are very, very dynamic.

But the PIRs are the same, the priority intelligence requirements.

This is also a platform that we're offering our customers for them to be able to decide what is the most information that they need in order to make intelligence-based decisions, right?

So these PIRs, they help focus our resources as well and our efforts in order to gather the relevant analysis for our customers, right?

So customers can literally decide, hey, I'm interested in these three topics, and that's the priority order for them.

This is the type of communication that is allowed for these topics.

And that can also inform our team in understanding what is most relevant for our customers.

So putting all of those together can give us a clear picture of what are our users concerned about in the market so that we can create better reports and better intelligence that we can disseminate together with them.

Yeah, that's a great way to collect feedback, right?

And to be able to focus and try to provide as much value as we can. So I believe this is going to be very welcome by the Security Center and Cloud Force One users.

So thanks for that. I think we reached the end of this segment. So a lot of new things for the Security Center product and a lot of new features.

Thanks a lot, Sofia and Alex, for all the information.

That was really, really interesting, and I learned a lot.

So thanks for that. And for everyone else listening to this segment, feel free to go to the Cloudflare blog and read the announcements in more details and test the products.

With that, thank you very much and goodbye to everybody.

Thank you. Thank you. Thank you.