Customers + Success: Special Episode: Bot Management

All right, here we are. So we're live. Welcome to Episode 4 of Customers and Success.

My name is Kate Fleming and I'm thrilled to be on Cloudflare TV today with three of my colleagues and potentially even a fourth who's coming a little bit later.

So let me introduce Naveen who's sitting in our Sydney office and he's in our solution engineering team.

Good morning, Naveen. Morning Kate. Hey folks. And also then Ju Song who's sitting in Singapore and he's on the Customer Success team and takes care of our Korean accounts.

Thank you, Kate. And good morning, everyone.

And good afternoon. And Rohit. So Rohit's based in Singapore and takes care of our India territory.

Good morning, Rohit. Morning, welcome. Thank you, Kate.

So this episode is a little bit different to how we've run previous episodes.

The purpose of this series really is to delve into all aspects of customer success and how organisations that are subscription-based, software as a service businesses in particular, can drive customer retention and engagement and really what it means to make a customer successful.

As we move more and more of our businesses and our services to subscription-based models, this concept becomes important because it's not just a one-off opportunity.

It's about proving continual value to a customer, which is a different motion and I think a very rewarding one.

So today we're looking at it from a slightly different angle. We're looking at it from customer stories and we're focusing in particular on customer stories around bot management.

There was a blog post that came out on Flare blog earlier in the week that talked about, we called them Property Portal, one of our large customers here in Singapore and how they had used Cloudflare to reduce issues that they were having with scraper bots.

And off the back of that, we thought it would be a good idea to come and maybe share some more customer stories about bot management and how Cloudflare in particular is helping our customers with this particular challenge.

So before we go into the stories, I thought it could be good to level set everyone who's in this room and on this call and just hand over to Naveen and say, Naveen, can you take us through, I mean, bots, all of a sudden this nomenclature came onto the scene and if I'm a business leader, I might necessarily understand how dangerous these things can be for me or that they're good or bad ones.

How can you help us understand what bots are and what they do? Sure.

Thanks for this opportunity. Let me start with ground zero, right? Let me try to define what is a bot.

If you look at the Internet traffic, it's a very homogeneous traffic.

By homogeneous, sorry, it's a heterogeneous traffic. By heterogeneous, what I mean is if I want to categorize the whole traffic into two big buckets, one will be the traffic that is generated by human interactions, right?

Your web browsers, your mobile applications, your desktop applications, which essentially require someone to sit behind those applications and start interacting with those applications, which in turn will generate this traffic.

The second big bucket is what we term as a non -human generated traffic or I would call it as an automated traffic.

These are typically like machine-to-machine communications.

These are one application trying to communicate with another application, which does not necessarily need to have a human interaction.

These already have intelligence in them and they start this communication channel between each other.

When we talk of bots, we are essentially relating to the non-human or the automated traffic.

We have a non -automated traffic, which is generally human-generated one, and then we have got an automated traffic.

Now, why is automated traffic important?

Should organizations be concerned about it? In one of the recent studies, in fact, we had another blog post that came out yesterday by our bot management teams that tells us that more than 40% of the Internet traffic is a bot traffic, which means the source of the traffic is coming from automated source.

Does that mean that all the automated traffic are bad or are they good?

It's a combination. For example, an example of a good automated traffic could be your scanning tools.

Organizations use scanning tools that can scan their network and show them any vulnerabilities that may be a part of their network.

That helps their teams fix those vulnerabilities.

Similarly, they can have automated tools that can scan your websites, your public -facing assets, highlight any vulnerabilities or gaps, and then your team can take a look at those gaps and fix those gaps.

But again, as I said, these tools are also like, the bots are also like double-edged swords.

A genuine user, a good intentional user, can use it for the advantage of a company.

A malicious user can use it to create havoc within an organization.

For example, if you take a look at a denial-of-service attack or a distributed denial-of-service attack, they are a very classic case of a malicious user who is creating an army of bots, and then these bots, based on his command, will attack a particular victim.

If I want to broadly classify bots into different categories, there are lots of malicious bots.

Just to name a few, there is something called a zombie bot.

The zombie bot is what is used in your distributed denial-of-service attack.

A compromised machine which is waiting from their command center to execute a set of attacks on a victim.

Then there is credential stuffing bots.

These bots are typically used to go to login pages of various websites and keep trying to crack the login username and password.

They try to use the compromised credentials and try to see if any of these credentials is successfully able to use against the login pages.

And then there is very popular scraper bots, especially big organizations, e-commerce organizations, retail organizations.

They deal a lot with scraper bots. Scraper bots are something where the malicious user sends these bots to crawl through the whole website, start pulling the relevant data, start pulling product information, start pulling the pricing information, and then they use this information to actually create a competitive advantage over that particular bot.

So bot definitely, not all bots are bad, but then we are seeing that malicious bots are indeed creating a lot of havoc for organizations.

And it's high time organizations start looking seriously at how to mitigate these bots.

Thanks.

That's a really nice way to put it. At the simplest terms, it's the human versus the automated traffic.

And I think one of the things that we can get confused on is that there are good bots.

My daughter is just getting ready for breakfast.

My apologies. There are good bots out there and that's great. We don't want to stop those, but there are bad ones as well.

And speaking of bad ones, you referenced scraper bots as being quite popular.

I'd like to welcome Jean, who's from our solution engineering team based in Singapore, and she handles our partner solution engineering.

Jean, I know you've recently worked on a customer where scraper bots was the key challenge you're facing.

Would you like to tell us a little bit about that customer?

Yes, Kate. Thanks. Jean here. I have helped a customer.

Their name is Property Guru. Muted myself.

And for people that might not be familiar with Property Guru, I know we have some people from North America joining us today.

Are you able to tell us a little bit about their company and what they do?

Yes, sure thing. Property Guru runs the most reputable property website in Singapore.

They also run the property website for the other countries in ASEAN.

When I moved here in Singapore, I also used their website to search property listings.

I am sure many of our colleagues did as well.

Yeah, I think anyone in Singapore has used Property Guru. So for people from other markets, it's like it's a property website.

So it's very familiar with domain.com or realestate.com if you're from different parts of the world.

And so, Jean, can you take us through a little bit about, you know, were they even aware that they had a bot problem and what problem were they seeing?

Right, so Property Guru is number one in the market and their property listings equals their intellectual property.

So it is natural for them to have a lot of scrapers and crawlers trying to collect those information from their property website.

And the abusers may be trying to reuse this information for their own benefit.

So Property Guru wanted to be able to have control on those bot traffic while not hurting any user experience or the search engine performance.

Got it, got it. And this is what Naveen talked about earlier, which is this notion that scraper bots come and they can effectively rip off your IP.

And if you're a market leader, I imagine there's a lot of people looking to rip off your IP.

And so they had this problem.

They were number one in their market. They had scraper bots coming over their site and stealing their content.

Can you talk me through what solution they were using and then why they ended up coming onto Cloudflare?

Yes, so Cloudflare's product bot management was one of their options they explored.

I would say one of the benefits we had against their other options, so Property Guru considered many options, was consolidation and the ease of use.

So compared to the other options, using Cloudflare to control this bot traffic, scraper traffic, didn't complicate their engineers' lives so much.

So control was intuitive.

Changes propagate in 30 seconds and analytics immediately available, etc. So Property Guru's engineering team also gets the best out of Cloudflare.

That means they not only use Cloudflare for bot management only, but also for all security and performance benefits, which was also a very happy news for us too.

Right. So the engineers are happy and presumably the sales and marketing teams are happy because they're not losing sales or having their hard work copied.

And then I know from us discussing this earlier, Jean, that there was some really tangible benefits that Property Guru saw after they switched to Cloudflare.

Do you want to just take us through those briefly?

Yes. So the main benefit would be the appropriate bot protection, of course, Property Guru was looking to have.

So we were able to protect them against bot attacks with the least false positive rate, which is great.

And their engineers also were able to control granularly using the intuitive control panel and analytics we have.

We were actually expanding this deployment to their wider platforms in other countries in ASEAN.

And we are in progress of further fine tuning.

Yep. So because of this whole consolidation and single pane of glass of control, one good thing was, one other good thing was they didn't hurt the search engine experience.

Right. And then thanks, Jean. And that ties back to what Naveen was saying earlier about good bots.

Right. Good bots were not hurt and the bad bots were being stopped, which is good.

Perfect. And Jean mentioned something about false positives.

And I was wondering if I could just throw to you, Naveen.

So just once again, in the interest of having this as a level setting conversation, can you just take us through what a false positive is briefly for maybe people who aren't so familiar listening in?

Yes. False positive, this concept comes in when we start dealing with smart bots.

If you look at the way or the attack vectors of bots, the bots try to mingle with your genuine traffic or a human generated traffic.

So in a very simpler terms, what the bot would try to do, a smart bot, what it tries to do is it will try to replicate a behavior of a human generated traffic so that the detecting tools will get confused whether a particular request, is it coming from a human source or is it coming from a malicious bot source?

Now, typically, when we put mitigation strategies, our intention is to ensure that malicious requests or traffic coming from malicious sources are being blocked while traffic or requests coming from a human generated or a genuine source is still allowed.

Now, the concept of false positive comes when you put a mitigation strategy, your mitigation strategy sometimes would even block a genuine request, a request which is generated by, let's say, a genuine source or a human generated source.

But for whatever reason, the mitigation strategy in place deems it is coming from a bot and it blocks the request.

The effect that this has is that while you are also able to maybe block malicious requests, but you're also now blocking genuine requests, the request that actually is directly has an impact on your business.

So, an ideal mitigation strategy should ensure that the false positive rate, which means the rate at which you are blocking genuine requests is as minimum as possible, but you are still able to block any malicious request that is coming from a malicious source.

Got it. Thank you.

And we've just been talking, I think, about an example of where, you know, Gene outlined where Property Guru really knew that they had a problem.

I mean, they were very clear when they came to us.

Rohit, take us through your example, because I know with the customer you worked with, they weren't even aware of what degree of bot traffic they were seeing.

Yeah, so recently we had a customer, Kate, and they were using our CDN and other Cloudflare products like the firewall and things.

But they had no visibility into the kind of split traffic, like how Naveen mentioned, whether, you know, how much is from humans and how much is actually automated traffic.

And this customer, in fact, is a leading e-commerce customer in India.

And they focus solely on cosmetics and lifestyle. And, you know, they need to ensure that it's only the right users who are able to log in and, you know, access and buy these products.

And so we definitely don't want to stop them.

But at the same time, we need to stop things like, you know, competitors and bad actors who want to, you know, do things like inventory hoarding, for example, during a sale.

So that things don't show sold out and, you know, users are actually able to buy them.

At the same time, they also need, you know, search engines like Google to actually show up in case there's a sale coming up and there's a product that's on offer or there's a price discount.

They need the good bots to pick up this as well.

But at the same time, stop the bad bots. And we were able to show them exactly the split in the traffic, you know, for a certain time period.

And when they saw that, they were really convinced and they said, OK, this is something that they definitely want to incorporate in their platform.

And seeing the ease at which we could just turn on bot management because it didn't require any specific scripts to be run or any sort of software to be installed or anything.

We could just turn it on from the portal and immediately it starts picking up the entire data.

And also, they were really interested in knowing things about how it works in the background.

So we did tell them, OK, about how a threat score is assigned to these bots.

And they were really interested in things like the heuristics and fingerprinting of it and how we identify these things like that.

So I think that's a really good use case.

And they've been probably using it for about a month now and they're really happy.

That's really interesting. So, I mean, this is an example of someone not – it's that unknown unknown.

So you don't even know what malicious traffic is scraping your site in many cases.

And they were able to use Cloudflare to get a sense of just how exposed they were by the sounds of it.

And then to also from an engineering point of view kind of geek out in a fun way around the risk scoring and maybe having some control over how they set those parameters.

Is that right? They liked having that access. The engineering team, like you said, they definitely like to tweak things.

So they like their own flexibility and things like that.

So that's one more thing that they really like, that even though it's a solution that we provide, they can really customize it.

So they can decide what action to take on these bots and how strict they need to be with them.

Things like at what time they need to take an action. Especially in e-commerce, the sales happens sometimes towards the end of the month sometimes.

And then sometimes during – in India, it's especially during festive periods.

So they can expect a spike in traffic at that time.

Probably more stricter rules at that time.

But also to Naveen's point, like you said, ensuring that the right users get in, are able to buy the products, and they need to have a seamless experience so that they return to be, you know, like returning customers, for example.

Yeah, I think we can all put up our hand as being caught in one of those cycles where for some reason a website thinks that we're malicious traffic and we have to go and continually revalidate our humanness and eventually we just say forget about it.

Okay, so that's India.

I'm going to now head over to Joo. And Joo, I remember you talking about a customer that had a slightly different background and a slightly different challenge that we helped with bot management.

Can you tell us a little bit about that customer to start?

Sure thing, Kate. So, like, I got a chance to work with a customer from a fintech and cryptocurrency industry.

So, like, cryptocurrency has been booming since 2017.

And there were a lot of, like, fintech companies dealing with exchanges and the trading of such digital currency.

But due to the nature of the business dealing with a lot of, I guess, currencies that have monetary value, so naturally it has been attracting a lot of malicious attackers from DDoS attacks, bot attacks, or even hacking attempts.

So, this customer based in Korea is one of the top cryptocurrency exchange operating out of Korea, but their business has been booming.

So, they have been expanding internationally to the Southeast Asian market and even more in the coming months.

But one of the problems that they have been facing is, again, just like a lot of the industry nature, they've been getting a lot of attention from the attackers.

So, some types of bot attacks that they have been getting, or credential stuffing, where a lot of the malicious bots are attempting to kind of have mass login attempts, which is, again, with the dual purpose of one is to try to, I guess, hack into stolen credentials.

And another thing is, again, if those kind of false login attempts accumulate, it'll have a kind of load, it'll add into a lot of load to the overall site itself, which increases their traffic and kind of, again, it's not good for load balancing purposes.

And another kind of type of bot attack that they have been facing is, again, content scraping, because a lot of the, apart from the trading itself, a lot of the users are attracted to the latest market data, like how much is the Bitcoin worth today, and then what is the live ongoing kind of trading and the rate that's happening at the moment.

So, a lot of the competitors are trying to just break into this industry in an easier manner.

So, they often try to steal those kind of valuable market information directly from the site itself.

So, this fintech customer of mine have been trying to kind of fend off from all those type of attacks prior to, I guess, meeting Cloudflare Spot Management.

That's really interesting history, actually.

I know cryptocurrency has been one of these, it's like a supernova, it was this bright spark of activity, and then there's so much competition.

And so, when they moved over to Cloudflare, so were they using another solution or they just realized they had this problem and they came straight to Cloudflare?

So, I think it's kind of a lot of common theme that we're noticing, but I guess the thing about bot is that the customers are not, in general, aware of that many options out there.

So, before, I guess, being introduced to the Cloudflare bot management, they knew that there might be some bot problems, but they didn't know the magnitude of the problem.

And another thing is they didn't know, I guess, are there any easy way of fending from those bots?

And I think, again, the manual way they have been thinking is, again, manually going through the traffic and trying to see if, like, I guess, it would have been more reactionary.

Like, if something happens, then they try to figure out, okay, how to increase the protection from bot attacks and so on.

But I think that's why a lot of their engineers were very happy when they were first introduced to Cloudflare bot management.

Right. So, it sounds like they first were able to, by coming over to us, were able to get a full diagnosis of just how extensive the bot traffic was and maybe just the degree of risk that they were facing.

So, from a risk management point of view, it's useful.

And then, also, just from protecting their competitive placing in the market, they're able to stop that traffic and also then get maybe a sense of what type of things are happening beyond the scraping and stopping the credential stuffing, which I think is also a good thing.

Very common problem with any site that's got logon portal, right? I mean, it's that notion of if you can brute force your way into a site, there's all types of information that you can get.

And we've got about five minutes to go. And I know, Rohit, you had also another example of a fintech company.

Did you want to just tell us a little bit about that company?

Yeah, sure thing, Kate. So, this customer, in fact, signed up directly with bot management.

And it's similar to how, you know, like what you said, because there's a lot of financial transactions involved.

Now, they're a leading payment provider, and they've been in the market for about 10 years now.

The thing is, they wanted to make sure that they already knew they have a bot problem, and they knew the risks involved.

Because since there's a lot of transactions that keep happening on this platform, they have a lot of customer sensitive information.

And that includes customer names, personal details, for example, credit card information.

And they really cannot afford any level of breach, because that will be an incident for them.

And their entire DevOps team works with us.

So, right from the get-go, they wanted to customize the entire thing. So, we had one session with them where we told them how the technology works.

And post-switch, they decided to set up all the rules themselves, test everything themselves.

And also, one more thing I can probably tell about them is, like the other customers who might not know they have a bot issue, they already knew it.

And they want complete visibility, end-to-end, because it's a financial company.

And they have a lot of things to report to regulatories, different boards.

And so, they definitely wanted all this information.

And anything like credential stuffing, like how you mentioned, is something they really cannot afford.

Yeah. So, I think that's a very good example, which is very similar to how the cryptocurrency one that you mentioned.

Yeah. Especially when companies, I mean, if you lose the trust of your customers by exposing their private information, the hit to you in the long term is significant.

And we've got about three minutes to go.

And I wanted to just throw back to Naveen. I'm going to make a statement, and Naveen's going to tell me if I'm right.

So, you're going to quality check me.

So, I'm going to say that based on conversations we've heard today, and also my own reading and what we know around Cloudflare, that if you have an Internet property, we can almost guarantee, in fact, we can guarantee that there is some type of bot action or traffic coming to you.

And, you know, 40% of Internet traffic is bot traffic.

And a portion of that is going to be great. It's going to help you to get your placings with search engines.

But a portion of that is also malicious and can be loss of IP that you don't know about.

It could be a loss of competitive edge that you don't even know is going on.

And then, worse, the things that you might find out after it's too late is that someone's brute forced their way into your system and they've got details or that they've been stuffing your shopping cart, which is another one we haven't talked about.

So, if you know you have a bot problem, it's easy.

You could come to Cloudflare and it sounds like we've got a very simple solution for people to play with and then enable.

And even if you don't have a bot problem, Naveen, would it make sense for someone just to come and speak to someone at Cloudflare and let us turn it on for them for a little while and they can see the traffic?

Absolutely, Kate. And that's the right suggestion that I would give because, as you mentioned, Kate, if you have any public-facing Internet property, you would probably have a bot traffic.

For me, the number one step to deal with bots is first, get the visibility.

Understand what is the trend of your traffic.

Get visibility to your traffic. And I know that there is a temptation, especially for some of the organizations, to say, look, we don't have any bot issues.

I'm pretty sure you will always have it. Like, with most of our customers, they always thought they never had a bot issue.

And when they got the visibility through our bot management product into what their traffic patterns look like, what percentage is coming from automated source, what percentage is coming from non-automated source, it kind of raised alarm for them saying, oh, we didn't know that.

So definitely visibility is absolutely important.

In fact, we are also going to enhance our product such that even if you don't have bot management turned on, we will give you that visibility.

We will tell you what percentage of the request is coming from, probably coming from bot, what is coming from human interaction source and what looks like bot or maybe human.

So if you ask me what are the two things that my bot management strategy or my bot mitigation strategy should incorporate, one is visibility.

As an organization, we should get visibility into what the traffic patterns look like.

And secondly, the mitigation strategy should be driven by your threat intelligence.

Ensure that there is your mitigation strategy is constantly feeding all the threat intelligence that is happening throughout the public network.

And then you are building up controls based on that threat intelligence.

All right, guys, let's get on with your day.

Thank you so much. Bye, guys. Thank you, everyone.

Have a great day today. Bye.