🔒 Log Explorer eliminates the need for third party storage tools

Hey, welcome to Cloudflare TV. My name is Ankur Aggarwal. I'm a product manager here at Cloudflare based out of San Francisco, and we're really excited to talk about Security Week and highlight some features from AI, SASE, which is Zero Trust, and our magic products, application security, and posts on how we're helping to make the Internet better.

For today's segment, I'm joined by Jen Sells to do a deep dive into Log Explorer, and over to Jen to intro herself.

Hi everyone. I'm super excited to be talking to you today about Log Explorer.

I am a product leader in our application security organization and I'm based out of Portland, Oregon.

Awesome. And since we're doing a deep dive into Log Explorer, let's just get a quick synopsis on, you know, what is it?

So Log Explorer is a new dashboard experience and an API that provides the ability for our users to query their logs stored on Cloudflare.

It can be accessed in two different ways.

We've embedded some of the functionality into our security analytics page, giving you the ability to view raw, unsampled logs instead of what is there today.

Additionally, you can also open a core Log Explorer dashboard to query data based on any field.

Awesome. And then since we have a lot of products here at Cloudflare, which ones is it available for today?

Log Explorer supports any product that uses the HTTP request and firewall events datasets.

So essentially that means any of our CDN related products or products in our application security suite, such as WAF, bot management, or our API security products.

Awesome. I know that's going to be super helpful for those customers. So what led us to focusing in on Log Explorer from all the kind of many features we could build?

Yeah. So security analytics was released last year and it's been very well received.

Customers have been using it to identify anomalies and potential attacks.

But however, it does have a gap in that very often when users are doing these types of explorations, the next step they need to do in their investigation is to do a deep dive and conduct a forensic investigation on actual events.

And because security analytics uses adaptive sampling in the background, these forensic searches were not possible for most of our customers.

So until now, customers had to use our log push functionality to send their logs to third party tools for investigation.

These tools can be very expensive to set up and maintain. And we've received feedback from many customers that they would really just prefer to keep their logs within Cloudflare.

And you've mentioned it a few times, forensics, but can we just get a quick definition on like why we're focusing on the forensic side?

Yeah, for sure. Forensic practices are a very important part of security investigations because there are often very strict and specific requirements around reporting and evidence gathering related to the incident.

So investigators use logs to determine the full scope of the attack. What are all the resources that an attacker access?

And what if any data was exfiltrated out of the environment?

Then the logs are often saved as proof of what did and didn't happen.

That's a very, very important part of a company's compliance requirements.

For sure. This is definitely something I've definitely worked with our compliance and security teams to even deploy Cloudflare's here at Trust.

We really believe in securing Cloudflare with Cloudflare.

So I've seen a lot of these practices firsthand.

So as you built Log Explorer, how does this all work under the hood?

Great question. So as we were doing this, it was really important to us that we kept storage separate from compute.

This helps us control costs, both for us and for customers.

And it gives us finer grained controls around scaling and reliability.

Log Explorer in the backend has three main components.

It has ingesters, it has a query engine, and it's based on R2.

The ingesters are responsible for accepting the data, converting that to a Delta Lake format, and then writing that to the storage layer.

The data is stored on Cloudflare's object storage platform, R2. And then there's a query API that's based on SQL that accepts and processes all those queries.

All of these pieces sit behind a new dashboard UI. I want to touch on one piece of what you just said.

So you said we use SQL for that query interface. Why did we choose SQL in particular?

Other similar tools have built proprietary query languages that you need to learn in order to really make use of the tool.

So we discussed this on our team, and we really wanted to provide something that didn't have a steep learning curve for our users.

Standard SQL query language has been around forever, and it's very simple to learn.

And we felt that makes Log Explorer more accessible to more users.

That's awesome. And then now that we've been kind of chatting about Log Explorer, is it available today?

How can I sign up and use it?

Starting today, we are opening Log Explorer for beta testing. We're welcoming any business or enterprise plan customers to join.

There's a sign-up link from the blog that you can go to and enter in your information and ask about joining the beta.

That's awesome. And then what's next? Because I feel like this leaves us a lot of opportunity to build.

Yeah, this is something super exciting, but we are really just getting started.

We're looking forward to getting lots of feedback from our beta testers on what will make this even more powerful and useful for them.

But that said, some features that we do plan to tackle coming up next is the first on the list is adding more Cloudflare logs to Log Explorer, specifically our Zero Trust and Cloudflare One applications, so that all of security investigations can happen in one place for Cloudflare users.

We want to add custom retention periods, because as I mentioned, compliance is a very important aspect of this, and different companies have different requirements for how long they need to retain logs.

And then we also want to do some more integrated and custom alerting on what's happening in those logs.

That's awesome, because especially about the Zero Trust piece, I'm the product manager for Gateway, and honestly, I can't wait until our Gateway logs are within Log Explorer.

So with that, I want to thank everyone for joining us to chat about Log Explorer.

Jen, thank you for walking us through it. And if you haven't already, navigate over to our blog to see our full list of announcements for Security League.

Bye.