Understanding Cloudflare Bot Management

Music Music Music Music Music Music Music Music Music Music Music Music All right.

I think we are live. Hello, everybody. My name is Calvin Scherle. I am a solutions engineer here at Cloudflare.

And today I am going to talk about Cloudflare's bot management.

So let's go ahead and get started through these slides here.

So when I talk about bot management, the first thing I want to really talk about is kind of how we arrived here where we are with our bot management service and why we thought it prudent to develop a bot management product all on its own.

So traditionally, when you think about bots, maybe the concept of DDoS attacks comes up or sort of more targeted attacks on a particular website designed to steal some data or something along those lines.

So traditionally, services like Cloudflare's DDoS protection or web application firewall or even something like rate limiting were sufficient to stop the bot attacks that were sophisticated enough to kind of be triggered by those or be mitigated by those solutions at the time.

As Cloudflare sort of expanded its capabilities and offered new opportunities to customers to mitigate bot attacks and put in place measures to prevent against bot attacks, we developed new solutions like Cloudflare Workers, our Edge JavaScript platform, as well as Cloudflare Access, the ability to handle authentication at the edge as well.

And those allow customers to mitigate bot attacks on a larger scale.

Those solutions, while they definitely help and there's a lot of usefulness there in terms of being able to protect against bot attacks, not on a reactive sense, but more to be proactive about implementing good security measures in advance, there were still methods that could be used to bypass those.

And really, with the sophistication of bot attacks as they're growing now, it's really important that Cloudflare wants to be on the forefront and develop a next generation bot management solution that can leverage the smart data that we have, the vast threat intelligence that we can gain from all the web properties behind Cloudflare, but also that can continue to be even more proactive in a way that doesn't require our customers to do lots of custom development, to have to protect themselves against specific types of bot threats, or to be playing whack-a -mole, manually going in and blocking sets of IPs.

We really want to use this valuable resource that we have in terms of all of this data and these web properties that are on Cloudflare to develop a next generation solution that can mitigate those attacks automatically.

So, when we talk about bots, there's not really three categories, it's more of a sliding scale in terms of how sophisticated these attackers can be.

But when we think about bot attacks today, we're really thinking about some sort of automated process or automated agent that is attempting to masquerade as human.

And there are scripts and automated services that you might call a bot simply because they're automated, but typically where our focus is, and when you talk about bot management, you're talking about allowing legitimate traffic through while preventing illegitimate traffic that claims to be human.

So actually, to come to the next slide here, this is where we get into sort of good bots and bad bots.

And again, when we're talking about bots, it's usually something that is either just automated or is automated pretending to be a human.

And so where we make this distinction is really in the intention of that automated process.

And so when we talk about good bots, we're usually looking at things like search engine crawlers.

So Google and Bing and even Yandex, all those search crawlers, as well as monitoring tools like Pingdom.

And on top of that, additional sort of legitimate scrapers or site assessment tools, anything automated that helps you actually improve the performance or the impression of your site by providing you with valuable data, typically.

When we talk about bad bots, these are the things that you typically would employ a bot management solution for.

Things like malicious scrapers or spam bots or quick fraud. So actually, to go to the next slide here, this is where these are the sort of primary use cases, I guess, or scenarios in which you may be thinking about a bot management solution.

And so to go through these quickly, I'll kind of go through one by one.

Credential stuffing is probably one of the most common use cases that we see.

And this is the ability for some automated tool or service to gain access to a real user's account.

So going through a list of usernames and passwords, looking for ones that work to gain access to that account.

When we talk about inventory hoarding, this is where you might add products to a cart.

So for example, concert tickets or shoes or clothes or something are released and bots will go through and add all the products to their cart or immediately even purchase, make legitimate purchases on these products so that there are none left for any actual users.

So if you've ever seen concert tickets sold out literally the second that they're going live, that's typically due to something like inventory hoarding.

When we talk about content scraping, this is where you have bots which are viewing data, which should only be viewed by actual humans.

So this can be something like product pages or even user submitted content like artwork or something.

And typically for scraping, it's either to gather intelligence about a competitor or something along those lines, or even to take that data if that content is your business.

Scraping that content and posting that to a competitor's site to stand up your own competitor based on data you scraped or aggregated from other locations, that can be really damaging to a business where that content is their actual revenue stream.

Credit card stuffing is probably one of the more directly damaging, I guess, in that these can be actually fraudulent transactions made with similar to credential stuffing with usernames and passwords.

Credit card stuffing is using stolen or just fraudulent credit card information to make those purchases and it's going to have direct impacts on your business.

And then to content spam, where you're submitting some fraudulent data to a form, a lead generation form or a comment form or some forum or something along those lines.

And then finally, application DDoS is really often a side effect of some of these other cases.

What we do have, we do see bots that are attempting to actually bring down a website or something along those lines.

Typically, those attacks tend to be both larger in volume and lower in complexity if the intent is to actually overwhelm a web resource or a website.

So the DDoS tends to be, if you have thousands of bots that are cycling through usernames and passwords, it may just overload your service to the point that you can't keep up with that number of requests.

And so with the exception of application DDoS, there's a couple of common themes here that are really important to understanding where a bot management solution, particularly like ours, would fit.

One is that aside from content scraping and potentially the DDoS, the rest of these are cases where you have some form data being submitted, the usernames and passwords, objects to be added to a cart, credit card information or whatever is on your form that's getting spam.

Those are all cases where you have data being submitted to some location, either by humans or by an automated source.

And again, covering all of these, the other important point here is these are all cases where the actions on their own are not malicious.

Submitting a username and password or making a credit card purchase or submitting a comment to a forum post.

These are not malicious actions, but they become malicious when you perform them at scale.

And that's where Cloudflare's bot management really focuses on is detecting whether a request or some data being submitted is coming from a human or coming from some automated source, whether it's a machine to machine traffic.

And that's really where the focus for Cloudflare's bot management is.

So these are some, I guess, common solutions or common techniques that one might use to attempt to protect against bot attacks.

So when we talk about homegrown solutions, this is usually something like, as I mentioned, I'm manually going through and looking for suspicious activity and blocking IP addresses as they make requests to my website, for example.

There are hosting providers that may have built in bot management solutions.

Things like rate limiting or web application firewall for smaller scale or more targeted attacks may be helpful as well.

And multifactor authentication or other authentication methods may be useful in detecting bots as well.

There are also some solutions which rely purely on some JavaScript that can be injected on the web page response.

But this only really works out of the box as a comprehensive solution for agents which are going to accept and run JavaScript.

So when you talk about things like mobile apps, native mobile applications, which can't run JavaScript on response to API requests, for example, that method where you're purely relying on JavaScript tends to fall down.

Also, you can have bots which simply don't run JavaScript and prevent it from running in the first place, which could be a signal of malicious activity and might be a red flag to throw up.

But if that's your sole detection mechanism, it prevents that from being fully effective.

So Cloudflare's bot management, for one, sits at the network level.

So like many of Cloudflare's services where we act as a reverse proxy between your users on the left hand side, you have browsers, mobile apps, APIs, which could be your customers, your legitimate visitors to your website, and potentially good bots as well.

Things like, again, Google or Bing that are going to crawl your site or other automated tools that you have set up.

As well as bad bots.

Any of these types of agents, if you will, that are going to make requests to your web server, rather than requesting to your web server directly, you have Cloudflare operating as a reverse proxy front.

And this allows us to apply more traditional security services like DDoS protection for distributed service and rate limiting, as well as our web application firewall with its own set of managed rules based on global traffic that we see from all those properties that are behind Cloudflare.

But also Cloudflare's bot management can be inserted as another layer of security when you're proxying requests through Cloudflare.

And so Cloudflare's bot management has three major components by which it performs its detection.

And all of that is in service of generating a bot score, effectively, from say 1 to 99 of how likely a particular request is to be coming from a human source, as opposed to an automated source.

And so these three detection mechanisms then generate their score in concert with each other, which is exposed in three places.

Which are our firewall rules, listed here as WAF rules, Cloudflare Workers, that JavaScript engine at the edge, which I talked a little bit about earlier, and Cloudflare logs, so you can actually see the bot score for every request.

And so the three in this blue box here are our detection mechanisms for actual bot requests.

These are the mechanisms by which we determine if a request is coming from an automated source or not.

So those are, just to be brief about them without going into crazy amount of detail, starting from the right hand side, actually fingerprinting, where we're looking for specific fingerprints based on some properties of the request, machine learning, as well as behavioral analysis.

We're looking at the behavior of the request.

So how we actually deploy bot management is in, again, three separate places.

So one, in our firewall rules.

And this allows you to create customized firewall rules based on that bot score.

Next would be in Cloudflare Workers, where you can again create customized logic using that bot score.

And then finally in Cloudflare logs, where you can visualize the bot score in actual log entries based on data that you either exported to a third party platform or which are available within Cloudflare's kind of log API as well.

So the advantages of Cloudflare solution are really its simple deployment.

As there is no JavaScript required for the solution to run and no mobile SDK required, there's no code changes you have to make to your application.

Simply by having Cloudflare as a reverse proxy in front of your web service, you can take advantage of those accurate detection mechanisms.

So what I mentioned before, behavioral analysis, machine learning and fingerprinting, as well as an optional JavaScript detection module, which enhances detection for things that claim to be or are masquerading as legitimate browsers.

It's integrated with the rest of our security suite.

So again, there's same bot score available in multiple places throughout Cloudflare's platform, as well as rich analytics that you get through firewall analytics in the dashboard, as well as those SIEM integrations.

So things like our analytics partners, such as Sumo Logic or Datadog, any of those other SIEMs that you might be exporting data to.

You'll have access to the bot score in those. And many of those partnerships, we've actually developed customized Cloudflare dashboards that work out of the box to help visualize your Cloudflare data in addition to specifically bot management data that's coming from Cloudflare.

So actually, this is the point where I would want to show quickly in the last few minutes here, how to actually deploy bot management live on a Cloudflare zone.

So I'm going to come out of here and go over to my Cloudflare dashboard.

So here is the typical Cloudflare dashboard and within my dashboard, once I've logged in, I have several domains where I have Cloudflare sitting in front of those.

And so I'm going to come into just calvinshirley.com and here I have a basic overview, as well as the various tabs across the top for performance and security features that Cloudflare offers.

And this is where we can start to actually integrate the bot score that we generate with these other services that I'm already using.

So first, I'm going to come to the firewall specifically.

And I'm going to come to firewall rules. And under firewall rules, this is where I can create customized expressions to be able to take action based on some properties of the request.

So here you can see I have this bot management site wide rule.

I may have other firewall rules set up in here, but if I click this wrench icon, I can actually see this rule that I have here deployed.

Very straightforward. We're looking for a bot score less than 30 on that 1 to 99 scale.

And I'm looking for where it's not a verified bot. And this verified bot flag is our known list of good bots.

So things like Google, Bing, monitoring tools like Pingdom, etc.

These are verified bots that we know are legitimate and we can verify are coming from legitimate sources and the appropriate sources for those specific those specific types of good bots.

And so I'm going to cancel out of this and just show how easy it is to actually build this rule.

So if I create a firewall rule here. So this is going to be something like challenge all bots.

So here under this field drop down. In addition to, I can come in and edit this expression and start typing whatever I want in here.

But I can also use this expression builder to build this customized expression by which I want to take some action.

So as you can see, there's lots of variables available here. But what I'm going to do is look at this bot score and I'm going to say where this is in less than 30.

And 30 tends to be a good starting point. Actually, most of the bot scores that hit our platform tend to be, or for requests that hit our platform, tend to be very heavily weighted to one end of the spectrum or the other.

So this is a pretty safe threshold where I'm expecting only browser traffic.

So I'm going to say if the bot score is less than 30, meaning it's more likely to be automated, where lower is more likely to be from an automated source and higher more likely to be human.

I'm also going to add another condition for that verified bots that I mentioned earlier.

So I'm going to turn this verified bots off. And this is built that same expression I had before.

So where the bot score is less than 30 and it's not a verified bot, this is where I'm going to take some action on those requests.

In this case, I have several actions available to me and these are available with or without bot management.

I want to really be clear that the bot score is an incredibly valuable tool that's added to the firewall rules toolbox, right, but it's not the entire essence of firewall rules.

If I wanted to have this match something other than the bot score, for example, countries, I want to block everything outside of a specific set of countries or IP ranges or paths or host names.

I have that ability as well. In addition, all of these actions are also available completely separate from whether I have bot management or not, right.

But in this case, I want to make some determination and take some action based on the bot score.

So my options here are, again, simply block the requests. That might be a little bit heavy handed, but if I have very sensitive data, I really want to take no chances, I can go ahead and do that.

There's a JavaScript challenge, which is a little more lightweight.

As well as a Captcha challenge, which allows you to actually present a Captcha, which if you're not familiar with Captcha, it's those prove that you're a human tests, right, where it's click on all the pictures of cats or whatever.

And that will prove to the system that you are indeed a human and not a bot that can't identify something like cats in pictures.

This can be definitely useful.

I think one of the more heavy handed approaches has often been, I want to just deploy a Captcha on every single request that hits my site or my application.

There's several reasons why that's maybe not ideal, but the simplest of which is that you don't want to issue Captchas, in this case, unless we think that it is a bot.

So it can actually be quite detrimental in some cases to your user experience if the user is having to solve these problems before they're allowed to proceed.

So this allows you to not issue Captchas or disrupt the user experience for legitimate users, but if there is a chance that a legitimate user was flagged as a bot because of some weird pattern in their traffic or some funky browser extension that they're running or something along those lines, the Captcha still allows them to bypass the challenge if they are indeed human and allow them to access that resource.

So it tends to be a pretty good end game in most cases when I issue Captchas to stuff that seems to be a bot.

As well you have the ability to create expressions which allow, which just means not to trigger subsequent firewall rules and allow it through the firewall rules kind of module, as well as to bypass on subsequent features.

So I'm looking for things like I want to bypass rate limiting on requests that match a certain pattern.

And actually, one of the most powerful actions here can be just a log.

And this allows you to just create log events based on this expression, which appear both in the firewall analytics here, as well as in Cloudflare's raw logs that you can either pull down the API or export to whatever your mom and ring tool or sim of choice.

So if I come back to this overview page, I'm going to lose my changes.

Come back to the overview tab here.

This is our firewall events or firewall analytics that are available within the dashboard for enterprise customers.

And this allows you to see not just firewall events, but all security related events that took place on requests coming into your zone.

And so for everything that's in CalvinShirley.com, I can see all the security events that have occurred here within the last 24 hours, although I can go back for 72 hours or even set a custom time range and go back much farther.

But this allows me to see ongoing events, as well as the action that was taken and the service that performed that action.

So here I can see I have all of these log events, but I can actually click filter and just filter down on those.

And I can see that of those log events, all of them, 100% here, 28 events in the last 24 hours were triggered by firewall rules.

And that's why I have this rule just to log bots.

And so this can actually be useful. And again, if you don't want to take specific action automatically or programmatically, but you still want to just see what type of bot traffic is coming into your zone or what type of requests or common IPs or countries maybe that are sources of bot traffic, you can utilize just that log action within firewall rules.

And so here I can continue to filter down if I wanted.

And you'll say, okay, of those 28 requests which were logged, five came from Russia.

So I'm going to filter down on those. And this is just logging all bots, right?

So it's going to get some legitimate bots like the Yandex search crawler and things like that.

But if I clear those filters, I can also see other security solutions which trigger security events here.

But I also have this bot management site wide rule.

And so this is that rule I showed earlier where it's set up to just anything with a bot score of over 30 that's not a verified bot, I'm going to issue a capture challenge to it.

And here you can see there's definitely some events, a lot of things pretending to be legitimate web browsers or some other kind of suspicious bots that I'm definitely not employing on purpose on this zone.

But the point is here that the firewall analytics really let you dig down into those individual requests and see, again, what type of events were occurring based on those events.

So I know there's only about three minutes or maybe less than three minutes left.

So I did want to touch on just a couple other quick points here.

As we spent most of the time on firewall rules, this is just a slide showing how you can utilize those variables within firewall rules and those fields in the dropdown, which I showed live.

But there's also the ability, as I mentioned, to use Cloudflare Workers.

And so Cloudflare Workers is our custom JavaScript engine which runs at the edge.

So you can run any JavaScript that you want on any given request on a sort of event triggered basis.

So the request comes in, I can look at properties of that request and take some action, again, based on that.

So the simplest option might be to simply take that bot score that we're generating and attach it as a request header sent to your origin.

So that actually allows you to get all of that bot score and that bot intelligence directly at your origin server as well that we're processing all the requests to.

On top of that, you can use that bot score to serve alternative content or redirect to other pages, for example, or maybe even require additional steps of authentication or authorization.

So you can say, okay, your bot score looks kind of low. I'm going to trigger my two-factor authentication in those cases.

Or I'm going to let all users log in.

But if their bot score is low, I'm going to send an email to that user saying, hey, your login looks a little suspicious.

So you can really customize the actions you want to take based on that bot score using Cloudflare Workers.

And then my final point here is simply just that bot score and the scoring source being the module that scored that particular session or that particular request.

And the score itself are both available within Cloudflare logs. So as you pull log events down or push them into your external analytics platform, you'll have the ability to view them there.

Alright, that's all I have for you today. Thank you very much for watching.

You can learn more at Cloudflare.com slash products slash bot dash management if you want to learn more about our bot management solution.

Thank you for watching and enjoy the rest of Cloudflare TV.

Take care, everybody. Cloudflare TV