Understanding Cloudflare Bot Management

Hello everybody, my name is Calvin Scherle and I'm a Solutions Engineer here at Cloudflare and today I'm going to be talking about Cloudflare Bot Management.

This is an add-on for enterprise customers that allows you to manage bots efficiently and at scale.

So before I talk about the actual bot management product, I just want to give a quick intro for our sort of history of Cloudflare's bot management.

So Cloudflare has actually been doing bot management in its own form since its inception.

So with things like our web application firewall and rate limiting, Cloudflare has enabled our customers to mitigate bots that were attempting to either access information on their domains or gain access to stuff or even perform DDoS attacks.

As bots became more sophisticated and customers needed a solution to deploy bot mitigation and other security features at a larger scale, Cloudflare looked into developing Cloudflare Workers, which is our serverless platform which allows you to run customized JavaScript at Cloudflare's edge.

And Cloudflare Access, which allows you to put an authentication layer at Cloudflare's network that sits in front of your origin servers and acts as a broker with an existing identity provider.

And so those allowed customers to add more security features and further customize how they want to deploy those security features at a larger scale.

Now as the arms race continues between bots on the web and those looking to protect against them, Cloudflare really needed to move into a next generation of bot management.

And to do this, we've launched a next-gen bot management solution which uses machine learning and behavioral analysis and other techniques that sort of stands on the shoulders of our smart data that we gather from the 20 million plus Internet properties that are behind Cloudflare.

And so that's what we're talking about today, is that next generation bot management solution.

And it's also important when we talk about bot management to talk about what a bot actually is and why it needs to be managed or mitigated.

And so when we talk about bots, this is usually something that is attempting to mimic human behavior, to pretend to be a human in order to gain access to sensitive systems or to kind of gather data or view data at a scale which humans are not able to do.

And so on the left here where you have basic bots, this is things that are just very simple scrapers, simple scripts that have a pretty repetitive pattern and may just be collecting data.

And these are usually pretty easy to mitigate, often coming from a single IP.

So something like rate limiting or web application firewall or DDoS protection is going to probably be sufficient for those.

Where you move to the right and you get bots which are more sophisticated, which are designed to do something a little more targeted and maybe steal some sensitive data or actually commit fraud on their own without human intervention.

Also scripts that are a little more advanced or even get into puppeteer scenarios where you're using a legitimate web browser but attempting to, from multiple IPs at once, sort of mimic human behavior or adjust their behavior so as to appear human or to not have so obvious of a pattern.

And for the more sophisticated bots, this is where you need really advanced intelligence and really advanced techniques to both detect and to mitigate against those.

And where the bots are tailoring their solution to specific customers, to specific kind of site structures and endpoints and things like that, is where we really need an advanced solution that can also work out of the box for many different scenarios but be custom tailored or customizable for each customer's specific needs.

One other distinction we need to make when we talk about bots is that not all bots are bad necessarily, right?

You do have bots which are attempting to mimic human behavior or are triggered, are sort of automated processes triggered by human interaction.

But these are not necessarily malicious, right?

So things like search engine crawlers like Google, Bing, Yahoo, Baidu, these will attempt to view pages as if they're real humans.

Same with site monitoring tools to see how those sites respond as if they were humans.

And generally you want those, you want your SEO rankings to improve and things like that.

You may also have kind of news feed bots or legitimate scrapers that will generate pages or actually improve the performance of your site by pre-fetching stuff and things like that.

And also maybe you work with partners who send automated requests to you like payment partners and stuff which are sending requests that are triggered from an automated service but you want to make sure those requests get through.

When we talk about bad bots, these are things like well there's just scrapers, spam bots, or bots which might be attempting to mimic not only humans but mimic some of these good bots like search engine crawlers, monitoring tools, pretending to be those so they can gain access to those sensitive data.

So these are some of the most common that Cloudflare sees, most common use cases for why customers would need a bot management solution.

You can think of these as the common problems that our customers are experiencing that led us to develop this next generation solution.

So I'm just going to run through these quickly and you may notice a couple common themes which I'll definitely point out at the end.

So the most common at least that I've experienced is credential stuffing where you have bots that are submitting usernames and passwords that are either completely stolen or just it's running through some generated list of passwords and stuff to try and gain access to a user's account and past that sort of login wall gain access to sensitive data there.

Inventory hoarding is where you also a very common use case where if you ever try to buy tickets or sneakers something like concert tickets or sports tickets and they're sold out instantly.

This is usually due to inventory hoarding bots which will add all the items to their cart maybe not actually purchase them but maybe actually purchase them.

Either way it leaves none of these items available for legitimate buyers.

Content scraping is typically seen sometimes it's behind a login wall but typically this is where you have publicly available information but again you only want humans looking at it but bots will come and scrape through iterate through content and either take that content and download it and post it to competitor sites so that can be damaging to your business but also just simply collecting information on say prices and using that to form a competitive price strategy.

So definitely want to avoid that.

Credit card stuffing similar to credential stuffing although instead of usernames and passwords they're dealing with credit card information which can be used to commit credit card fraud or even test credit cards to see which ones have a kind of one dollar transaction or something that goes through and when it does they know that card is good and use it to actually make more fraudulent transactions.

Content spam is again somewhat similar but this is typically where you have a web form like a registration form or some forum where users can post comments or something like that and bots are going to submit information to that to either kind of misdirect or trick users or also just to flood your registration form with a bunch of fake submissions or something like that.

And then application DDoS. So DDoS being a distributed denial of service attack whereby your sort of origin server is overwhelmed and can't keep up with the amount of requests.

This is often when we talk about bot management this is often a side effect of one of these other cases where there's so many requests coming in from automated sources that are far beyond what your real human sort of demand for that those resources are that it will either slow your web server down or completely cause it to crash.

So I mentioned there's a couple common themes here.

One is that with the exception of spam and DDoS the top four are really cases where or sorry with the exception of scraping and DDoS the other four are cases where you have some data being submitted to a form.

And again leaving DDoS to the side the other commonality here is that all of these are actions which are not malicious.

So submitting a username and password or making a credit card purchase adding items to a cart or simply viewing images or product information.

These are not malicious actions but they become malicious when you do them at scale.

And these are all things that good bots like Googlebot that's searching your pages is not going to be performing.

So these are actions where only humans should be doing them at a human level of scale.

And so that's where Cloudflare's bot management focus really is in determining if a request is coming from an automated source.

And for cases like these where you can equate automation to malicious intent that's where Cloudflare's bot management is most successful.

So something like a login page where only humans should be submitting data there and you can say filter out anything that's automated.

So these are some other common solutions before I get into Cloudflare's bot management.

These are solutions that you may be using today or may have explored using and I think none of these are bad ideas right.

It's just sometimes even a combination of these is not necessarily enough and this is where you need that sort of next generation solution.

So commonly hosting providers will have some built-in bot management solution.

When I talk about homegrown solutions here this is often something like I have a person who is or maybe myself who is looking at traffic looking for common patterns blocking IPs on an individual basis based on some data that's been collected right.

For rate limiting and web application firewall these are definitely useful tools that should not be discounted.

But rate limiting has some limitations where you're dealing with sophisticated bots where they'll fly just under that rate limit.

And the WAF is more designed for targeted attacks on an individual basis so a request that's trying to do some code injection or something along those lines that is very obviously malicious.

Again where bot management comes in is really for actions that are not malicious on an individual basis.

And then multi-factor authentication again fully support using this definitely a valuable security tool but again where you have bots that are just doing scraping just viewing public information where there's no authentication or just submitting form data or something like that it's not necessarily going to protect you against those.

And then JavaScript-based bot detection.

While there are some sort of optional JavaScript techniques that we use for the most part Cloudflare's bot management sits at the network level and so requires no alteration of your code no need to take special considerations of is it going to play well with my application and no actual development effort required.

So how does Cloudflare's bot management actually work?

So here's a sort of schematic view of where you have clients connecting to your web servers and passing through Cloudflare.

So where your clients on the left this is things like web browsers, mobile apps, legitimate customers, and good bots as well as bad bots are all going to pass through Cloudflare's network this orange boxes here in the center.

And then on the right hand side you have your actual web servers so those are sitting behind Cloudflare and all requests being filtered through Cloudflare's network.

This is where we can apply DDoS, rate limiting, web application firewall, etc.

but also Cloudflare bot management. And so in this bot management box in the center there's three items in this blue box here which are what Cloudflare uses to actually detect the bots.

And so there's just to go this quickly behavioral analysis where we're looking at for a particular IP or a particular session what is the behavior of that.

So we will take a baseline of traffic coming into your domain and say is this a single IP with a single user session behaving in some very anomalous way and if so we would flag that.

Machine learning as well so applying on an individual basis for each request generating a signature which is then applied to historical data across all of Cloudflare's 20 million plus Internet properties that are currently deployed on Cloudflare.

That's where we can feed those into our machine learning model and use that as well.

And then finally fingerprinting where we are looking at specific technologies being used that have unique fingerprints that we can easily identify regardless of the content of the request we can identify information about the requester and say this fingerprint matches curl or python script or something very obviously automated.

So by running each request through these three engines if you will this is all in service of generating a score.

So we generate this bot score that is between one and 99 where 99 the higher end of the spectrum is where 99 percent sure it's a human and towards the other end of the spectrum where a score of one would be where 99 percent sure that this is some automated tool there's machine to machine traffic.

And so once we have that score we deploy that score in three places that's in our firewall rules engine where you can build customized expressions and take some action on them.

Within Cloudflare Workers our ability to run that custom javascript at the edge to take more advanced actions and also in Cloudflare logs and this allows you to just view that bot score as well as some other information about every single request and that can be really valuable just seeing just feeding that into your existing monitoring or analysis platforms what requests look like they're coming from from bots or automated agents.

So just to recap there once we have this bot score that we exposed we expose it in firewall rules where you can block or capture it or just log those requests Cloudflare Workers where you can for example send the bot score as a header to your origin server or take some more advanced actions which I'll talk about at the end.

And then Cloudflare logs where again you can see this the score and scoring source for every single request and also export those logs into a cloud storage or sim or other analytics or monitoring tool.

So the advantages of doing it this way as I mentioned some will use mandatory javascript the advantages of Cloudflare solution in particular are again that ease of deployment that there's no javascript required for the solution to run no mobile sdk to worry about and there's no code changes or actual development effort needed on your side to get it working simply by passing those requests through Cloudflare's network you're taking advantage of all of that really valuable varied and large data sets that we have already at our disposal which we also use for the accurate detection.

So by using those three techniques which have some overlap but using the behavioral analysis machine learning and fingerprinting which can all be enhanced by this optional javascript detection to make the score even more accurate we can generate a very accurate score in terms of how likely something is to be automated.

It's also integrated with the rest of Cloudflare's security stack so within firewall rules workers and logs you may be performing other actions as well and you can integrate that bot score and that bot information with other stuff you're already doing.

So creating firewall rules which use the bot score in combination with different ip ranges or countries or things like that and I'll show a bit about that when I show how those rules are created.

And finally you get the rich analytics around it so being able to see within the Cloudflare dashboard firewall events associated with bot activity based on the rules you define as well as being some integrations with other sim providers or analytics providers many of which are Cloudflare partners so Sumo Logic, Datadog and others and that have pre-built Cloudflare dashboards some of which are specifically designed to visualize Cloudflare bot management data.

So this is the part where I want to show in the Cloudflare dashboard a bit about how these bot management rules are created within the firewall rules and what it actually looks like to utilize Cloudflare bot management.

So here I'm going to come over to my Cloudflare dashboard. I've logged in here and have access to multiple domains under my account and I'm going to click on calvinshirley.com that's my personal domain and here I have some if you're not familiar with the Cloudflare dashboard I have some overview here and some tiles across the top for all the different features or options available to me.

So I'm going to click on firewall and the first thing you'll see here is this firewall events which is actually showing all security events trigger anywhere across the domain.

So where I have some requests being blocked, some being issued a capture challenge, some being just logged where they're logged here but I don't actually take action on the traffic as well I can see which service triggered those events.

So in this case 97% of events in the last 24 hours were triggered by firewall rules although I could expand that time range or even define the custom range if I want to look at any time period in the last three days or sorry in the last 30 days.

So scrolling down you can see there's more information available about all the events that are occurring and if I come to this bot management firewall rule that I've defined I can filter on that and this is showing me and I'll show you what this rule looks like but this is showing me all requests hitting my domain which are being flagged as bots basically so anything that looks like a bot I'm flagging here in this particular rule and so I can continue to filter down as well.

I can say okay I have actually five of these requests hitting my WordPress login page so I'm going to filter on those and I can see the countries they're coming from the IP addresses the user names that they're using you'll notice these are mostly claiming to be legitimate web browsers although because they've been flagged here they seem to be bots and the fact that they're coming from hosting providers I think confirms that hypothesis and for each of these requests I can dig down even a little deeper and see properties of those requests or export them directly from here as well.

So what does that bot management rule actually look like?

If I come to firewall rules here's where I can again define those customized expressions and take some action based on that expression.

So if I edit this bot management side by rule you'll see here a pretty simple syntax bot management score less than 30 and that's again on that 1 to 100 scale and not a verified bot and this verified bot flag is for our known good bots so again things like Google or Bing search crawlers of that nature and some monitoring tools such as Pingdom are listed on this verified bots list and so anything matching that expression where the score is less than 30 and it's not a verified bot I'm going to issue it a capture challenge and so that's what you can see in that analytics and actually if I click on this little spark line here in addition to hovering over this percentage and seeing the total issued versus solved captures in the last 24 hours I can also click on that spark line and it will filter just for that firewall rule once again and I can see everything that's triggering that particular rule.

So let me show you quickly how easy it is to build that rule.

If I create a new firewall rule here I can give it a name so I'll call this bot management and here's where I can use this drop down to define the properties that I want this expression to be built upon.

So in this case I'm going to take this bot score I'm going to say where it's less than 30 and I could use a lower value if I wanted.

There's actually most requests fall either in the below 20 range or above 70 or 80.

It's actually not much data in the middle where generally when a request comes in we can be pretty certain which side of the spectrum it's on.

So I can say where the bot score is less than 30 and I can select verified bots and turn this off and so that's just built this expression here for me but the bot score is less than 30 and it's not a verified bot and then I maybe want to in this case I'm just going to log these requests or maybe I can allow these to bypass the WAF for example or bypass rate limiting.

There's other actions available here as well such as allowing from firewall rules just straight up blocking the requests or issuing a capture challenge or JavaScript challenge.

For this case I'm just going to say let's log those and you can add other conditions as well.

So this goes back to the integrated nature of our bot management service is that you can use that bot score in concert with other firewall rules properties.

So I can say maybe I have an API and for those the bot score has to be lower.

I want to be a little less strict so I'm actually going to say where the bot score is less than five.

I'm going to just log those requests and now I'm going to deploy and you'll see this little blue dot which means this is currently updating currently deploying and now that's already deployed to all of Cloudflare's data centers globally.

So within about five seconds you'll see any changes to your firewall rules or anything else you do in the dashboard for the most part will show up will probably very quickly globally and now as those requests match that rule they will show up here in the firewall events and in Cloudflare logs.

So the last things I want to talk about are just a little bit about how to again deploy firewall rules in a more advanced manner.

So again you can create these bot management rules straight from the firewall interface using those variables so the bot score itself that verified bots flag and also there's a static resource option which is specifically for if you're serving lots of images for example and sometimes in some cases you can't do things like issue a captcha as a response to an image so you can use those to exclude from the bot management rule or maybe your images are very sensitive and anything that looks like a bot you just want to outright block it if it's trying to do an image so that static resources matches lots of common static file extensions.

Another point I want to bring up before closing out the session is some of the interesting stuff that you can do with the bot score through Cloudflare workers.

So as I mentioned a couple times workers if you're not familiar is basically our platform that allows you to deploy javascript to Cloudflare's edge and when a request comes in it will run that javascript and there's a lot of data about the request available to that javascript engine including the bot score and information like geographical information and stuff on where the request is coming from and so I think the possibilities are vast and actually going into some of these examples is a bit out of scope for this session but there are quite a few things you can do when you have that bot score data within Cloudflare Workers and so there's some examples here some that I might like to call out are basically just capturing that bot score with a request and maybe sending it again as a header to your origin.

This allows you to basically accept all incoming data but you can filter out the bot requests on the back end without the bots knowing that they've been blocked or mitigated so they don't try and kind of up their game.

This is where that concept of asynchronous warfare comes in in that the bots have no idea that they're being mitigated.

One thing we see commonly is either redirecting to other pages or serving alternative content based on that bot score if the bot score is low.

If something looks like a bot you may just prevent it from serving an ad but again the bot doesn't necessarily know it's been outright blocked or anything it just sees something different than a real user would.

Another common one by sending the bot score to your origin maybe you have a login page and you allow all users to log in but you send that bot score along as well and if the bot score looks low you may notify the user hey we let you log in but the login maybe looks suspicious or that can be a trigger to turn on that multi-factor authentication for that user.

So a lot of really advanced possibilities here using the bot score and bot information within Cloudflare Workers.

Then finally again the bot score available in the logs so in addition to all these other log fields you have you also have the bot score and the scoring source.

The scoring source being was a machine learning or a cookie or behavioral analysis etc that actually generated that score.

So thank you very much for watching that is all the time I have for today.

I really appreciate you tuning in to Cloudflare TV.

You can learn more at this link below Cloudflare.com slash products slash botnash management.

Thank you so much and have a good rest of your day everybody.

What is a bot?

A bot is a software application that operates on a network. Bots are programmed to automatically perform certain tasks.

Bots can be good or bad. Good bots conduct useful tasks like indexing content for search engines, detecting copyright infringement, and providing customer service.

Bad bots conduct malicious tasks like generating fraudulent clicks, scraping content, spreading spam, and carrying out cyber attacks.

Whether they're helpful or harmful most bots are automated to imitate and perform simple human behavior on the web at a much faster rate than an actual human user.

For example search engines use bots to constantly crawl web pages and index content for search, a process that would take an astronomical amount of time for any human user to execute.