Banter with the Bots Team: Episode 1

All right, hello everyone and welcome to a brand new series we are calling Banter with the Bots Team.

We're really excited to do this. On the bots team, we always talk about different things that are going on in the world, whether they be current events or anything else that relates to automated traffic on the Internet.

And so we figured we'd set up a time to actually talk about it.

And before we get into anything, I want to do a couple introductions.

My name is Ben Solomon, I am the product manager for our bots team here at Cloudflare, and I'm located in San Francisco.

Abraham, want to give a quick intro? Yeah, hello everyone. Thank you so much for inviting me, Ben.

My name is Abraham Adberstein, and I'm part of the bots team.

I'm a systems engineer located in Austin, Texas. Yeah, so we are here to just talk bots today.

Abraham and I always say we like listening to tech podcasts or anything like that, where people kind of just chill and talk about what's going on in the world.

This is particularly relevant because there are a lot of things going on in the news right now that rely on bots.

So we'll get into that in just a second.

But our first item on the agenda today is actually talking about the PlayStation 5, and you'll see how this connects in just a second.

Abraham, I know we both looked at this when the PlayStation 5 launched a couple months ago.

There was huge demand. People across the world wanted the PS5. It was sold in a couple different outlets.

But in particular, there was a lot of demand on sites like Walmart.

Why don't you explain how that relates to bots? Yeah, for sure. So basically what happened was that I believe Sony gave an announcement about signing up to get a PlayStation 5.

Everyone signed up, so all those PlayStation 5s were out of stock.

So then the big retailers like Best Buy or Walmart started creating specific dates where you can go and buy a PlayStation 5.

The problem was when you were going to your checkout, immediately, in a second, that whole page would just get stuck.

Or when you're trying to get a PlayStation in the checkout, it would just suddenly disappear, or there would be an error in the website trying to process your transaction.

And what happened is that there's a full market of bots, and they're usually called scalpers, that go to all these type of retailer websites.

And they make huge orders. They can buy 3,000 PlayStation 5s in one single order.

So what happens is that as soon as those PlayStation 5s came out live to be bought by anyone, they ran out of stock everywhere.

And pretty much nobody got a PlayStation 5. The scalpers got all the consoles.

And the only way to get one was certainly on eBay, maybe double the price or triple the price.

And everyone was really disappointed after that. Well, and that's such a good point.

I mean, you know, when you think about an event like this, when some really high demand product is being sold, the reason that people want to use bots, and the reason that they want to jump in and buy 1,000 or 3,000 versions of something, is because they realize if they can take all of the stock, then they can resell it on sites like eBay or other places on the Internet, often for two, three, four times the price.

So bots are actually really engaged with this very expensive and costly market for a lot of people in the world.

It doesn't always have to be scalpers, right?

But in this case, it absolutely was. And the challenge is that if I'm a human and I go on Walmart, the moment the PS5 becomes available, I'm manually going to click all of these things, right?

Like I'm going to the Walmart page, I'm typing in PlayStation 5, and then I'm adding it to my cart.

But by the time I've done that, these bots have been able to routinely just automatically do all of these different things and check out and get those bots ready, get those PlayStation 5s right away.

So I'm wondering if you can tell us, I guess, a little more about what the fallout was from this.

Have any sites done anything to fix this situation in terms of like putting up protections around bots or trying to sell PS5s in different ways?

Yeah. So it's like a tricky situation because like you can start from first, like from the point of view, like is this like legal?

Does this affect economically the retailers or the customers?

And there's a lot of views regarding these because regarding these like legally, it's not something considered illegal.

Maybe it can break some terms of service from the websites, but the main problem is that you have a big customer base that just gets completely disappointed to the retailer.

But the retailer as well as all their products.

So it depends also on the type of product that you might be selling.

For example, you were saying like a ticket for a concert, that's actually very much more, it's more profitable for retailers because when you have concerts or music venues, you never know if you're actually going to sell all those.

So having scalpers guarantee that, like buy those tickets actually guarantees that they're going to sell more.

So it's about driving up the demand too.

If a bunch of scalpers show up and they purchase something, they can guarantee that there's not any stock left and create some sort of demand.

Right. But it depends on the market.

So for example, like if it's a music venue, they don't know if everyone is going to buy them, right?

So in that sense, no, they're not like, let's say another famous artist and someone gets all the tickets.

It's not guaranteed that they're going to buy all the tickets because maybe it's not a super famous artist, but maybe for the PlayStation 5, which was so famous, then that might have some backlash from the actually customer base.

And then that might depend on the retailers if they want to implement bot management or some type of detection system on their end.

I know for a fact, like Walmart does have some sort of their own custom, a low level bot management, like in-house made, and they serve CAPTCHAs and they did try to a little bit like ease or stop like the bots from just completely automatizing the full process and removing the whole stock.

So I guess it depends a lot on the retailers or also the type of products you're trying to sell.

Yeah. Well, and look, I think there's a couple other things that we can tie in here.

One is the fact that bot management, which is really what we work on, isn't the only approach here.

There are other companies, including Cloudflare, that offer solutions like waiting rooms where basically everyone who tries to show up for some big event gets queued into this line, right?

And you see this all the time.

I've had this when I try to sign up for popular events or try to purchase tickets.

I go to some website and I get thrown into this waiting room. There's sort of a loading box on the page and it says, hold on a second.

We've got a lot of people showing up.

We're going to let through a few people at a time. This is something Cloudflare is doing right now for its waiting room products that we've just launched.

And we're actually using this for vaccine rollout across the country.

It's, again, a very different situation, but a similar problem where you've got a lot of demand for something, in one case, a vaccine, in another case, PlayStation 5, but both are experiencing the same high demand issues and can potentially be plagued by bots when they do show up.

The other thing I want to mention here is from Walmart or Best Buy's perspective, I think it's reasonable to ask why they would care, right?

If they're just selling, say, 10,000 PS5s, they should just want to sell all of those.

And as long as they get the money, you'd think they would be happy.

But I think the other thing to consider here is there's a real customer pain, right?

There's customer impact. And even if you sell all of your stock in something, if you end up with 10 ,000 really angry customers or a million really angry customers who thought they were going to buy a PS5 and realized they had kind of been scammed by bots, that creates other problems for you as a company, right?

Yeah. No, yeah, for sure. I can definitely see how customers can feel cheated by the process.

And I think anyone who is somewhat in the retail industry should be at default trying to care about your customers.

Because, yeah, like you said, maybe you sold your stock, but then you create such a bad reputation for your company that you're not caring to give the product to the people at the fair price, that maybe your stock value goes down, or there's going to be less popularity to buying it from your stores.

And I mean, just from an economic standpoint, it's never good to have a monopoly where someone just has the whole product, tries to sell them.

So if you want to try to motivate competition between different retailers and provide options to your customers.

And then that could even dissuade later on even the customers themselves from buying the product that they really wanted, because there's just no way to get it.

So there's just so many ways that it could go wrong for sure.

Yeah. Yeah. And look, it's not all big businesses shouldering the cost to protect the consumer.

I think if you put something like bot management in front of your site, you're actually going to reduce the load that ends up hitting your origin and save money as a business.

So everyone kind of wins in this situation.

It's one of the reasons that we work so hard to try and actually block bots throughout the web.

But I want to pivot to something else we've talked about.

We see a lot of the same bot issues on things like social media, dating sites, right?

How do the bots on these sites differ from the ones that we were just talking about?

Well, yeah, they have their similarities and their differences, but it's an interesting fact.

In a lot of the dating sites and applications, a huge percent of the traffic is completely bot-based.

And the main difference, I would say maybe if you want to compare a scalper maybe to a bot that is trying to get inside a dating app, is perhaps the level of disguise that they use or complexity around them.

For example, a scalper that would just go to a Walmart store, they just have to fill up some signup forms, maybe change some user agents and make it be really fast to do it.

But maybe a bot that is actually more in these dating applications, they had to have an actual Facebook account with maybe some followers, some pictures.

Most of the bots actually have full profiles in dating sites, have full profiles, bios, pictures, comments, even in posts.

And even when they're inside the application, they tend to change some settings.

So we could say it's a more robust type of bot that tries to disguise themselves as a real user and use real user profiles.

So I would say that's one of the main differences between a Walmart bot or maybe a dating site bot.

And they both have also very different purposes.

One of the ones, a bot that is in a dating site might try to probably advertise phishing websites, inappropriate or phishing websites, promote subscriptions or unwanted services.

Or they're going to try to encourage the user to move to another application somehow.

While maybe a scalper would just try to check out, buy a product, and that's it, just leave.

Right. So look, it seems like, especially in the PS5 case, the bots are executing a very clear set of steps.

It's log into a website, add something in the cart, purchase it, and then rinse and repeat.

Just go back and do the same thing until they run out of stock. With things like dating apps and social media websites, it's actually a completely different level of sophistication.

These bots are not trying to execute a series of steps.

They're trying to impersonate and pretend to be other humans, which is ironically the whole challenge of bots on the web and what we deal with on a daily basis.

The problem is that from our perspective, we're trying to launch a product that will then help a lot of our customers protect themselves from these bots.

And the bots look like all of the other traffic. The bots blend in with the human traffic that's coming in.

And that's from a technical standpoint. But in the dating apps, both a technical and a UI, sort of a visual perspective, these bots have to appear as if they're actual humans.

And this is something we didn't see even a few years ago, let alone 10 or 20 years ago as bots are starting to show up on the web.

So dating sites are another big use case. I think we should talk about one more before we kind of move on here.

Another classic use case would be banks, right?

We've looked at sort of the very simple bots that would scoop up PS5s. We've talked about the sophisticated ones that jump on dating sites.

But banks see a whole different class of bots, which often deals with credential stuffing.

Yeah, for sure.

I guess there's different tiers and levels of potential harm a bot can do. When we start from Walmart to dating sites, but now to bank level, it's just a different level.

Mainly, I would say the main type of... Everyone thinks hacking around systems and companies is very complicated, like little programs or malware or attacks.

But most of the times, it's just bad configuration and mainly bad account management and password using.

And how it usually works is whenever there's a data breach in a company that left something misconfigured, they're able to dump all these passwords into the dark web or forums or even post it in Reddit or many websites.

And then what bots, what these credential stuffers do is that they just compile this massive terabyte sized list of passwords.

And they just try it in a variety of accounts.

And when they are able to get into one account, they can pivot to different accounts because we usually tend to reuse our accounts and also reuse the same type of passwords or pattern-based passwords.

And they're able to get into your bank account and they're able to make transactions or they were able to move things around, even double authentication.

And a lot of our processes today are not forced in many ways.

And it can be sometimes easier to even bypass those.

There's been plenty of cases where like 2Auth is not the best thing to, like it's not, it can be like easily bypassed.

It's, you know, it's kind of wild just to go back over this and really double click on it.

If I have an account at let's say bank A, right?

Some random bank. And I always sign into that bank's website using my username and my password.

If I'm not responsible, I, but look, most people do this.

I reuse some set of usernames and passwords across multiple different websites.

And all it takes is for one of those other websites to get hacked, right? To get compromised or leaked in some way for an attacker to then have my information and at least have a good guess that if I had an account at some other website that was compromised, bank A is a pretty famous bank and they should try those same credentials on this bank, right?

To try and get into my account. And no one's going to get it on the first try.

I think it's super unlikely someone could guess my credentials, even with an idea of something that I've used on another website.

But the idea here is that time is everything. If I'm a human being and I start to test and I go through and test one set of credentials, I probably won't get it.

If I go five times, still probably won't get it. But as you go 10, 20, hundreds, thousands of times, there's a good chance that other people can start to guess your credentials that actually get you into these online platforms.

And when you switch over from just having a human manually try these to actually having a bot do this at scale, that's when it gets really dangerous.

Not to mention the fact that we're no longer talking about buying a PS5 here or getting into a dating website.

We're actually talking about banks and financial information. Once you're in, you have access to someone's entire livelihood in a way.

So that's really scary.

What are some of the ways that we actually detect these bots on the web?

What are we looking for in terms of signals as we build out bot management? Yeah.

So basically, it's just like you mentioned. The thing of a bot is that it's just like a little monkey sitting behind a computer that runs in a dedicated hosting provider.

And it's just trying everyday massive amounts of passwords. Not necessarily all of them in one specific user, but across millions of users.

So statistically, you're able to get a couple of those users.

And then once you get to their email or whatever, you can just go move laterally.

But the signals that these kind of bots produce is exactly that.

Since they go across so many users, so many requests, that's when you start picking up a lot of signals.

And they're usually hosted somewhere.

They're not usually running on your computer. They're probably running on some Amazon instance or some GCP instance.

And they're just working all day.

So if we actually take a dive into the traffic, we can start computing aggregates or looking at how these bots kind of behave.

So then we can, for example, see, oh, well, we see requests coming from this specific address to this specific website at this specific time with these specific URLs.

So we try to create detection systems that look into that and immediately flag them and be like, hey, this is bot behavior.

Let's try to block this, rate limit this, or do some type of mitigation.

And that's also, I guess, the downfall of the bots. The bots are just programs.

They're programmable by a person with specific set behaviors. And we're able to pick on those compared to just normal, which are completely different to maybe a normal user who maybe just checks their account once a day or et cetera.

It's really wild. I have to say in my almost a year or so with the bots team, I've gotten more and more kind of pulled into this world of really getting interested in what bots are doing on the web and trying to uncover this a little more.

It's one of those rare things in security that not only is so important at kind of a low level in terms of how we technically implement things, but also is really interesting at a high and sort of abstract level as well.

Look, we could sit here and we could talk about how we're detecting all these bots and how we use all these different signals to try and find automated patterns in the Internet.

But it's fun that we can also then turn around and say, here's what this looks like in real life.

Here's an exact example of how this affects real human beings that isn't so detached from reality that it feels hard to even visualize in any way.

So now we've kind of gone over some examples in the real world.

These are very, very pressing issues and are things that will continue to happen despite a lot of the bot management systems in place.

Let's talk about maybe a bigger and even more relevant example, which is the stock market.

Obviously, those of us who have been following kind of the news lately have seen there's been a lot of volatility on the stock market, right?

Whether it's GameStop or AMC, plenty of folks are kind of shorting the market and then playing against the people who are shorting the market.

But bots, again, play a big role in this. And many folks use bots to actually sell shares of stocks and to do plenty of other things in the stock market.

Yeah, the stock market, it's just another playing field. You can pick pretty every area of the Internet or every area of technology and there's some bot doing something behind it and you would never have identified it.

And the stock market is just such a level of abstraction that is built up by bots.

And if you look at the sources, for example, you can pick up different sources, but some say 60% to 70% of the actual trading behind stock market is just algorithmic.

Others claim 90%. And I honestly believe it.

There's just such a variety of bots that can do activities from just simple, you can use a bot for yourself to just trade between certain caps or in thresholds.

You can have big companies moving the markets or making trades or trying to find the best bids.

I mean, the New York Stock Exchange have their own thing.

So it's just a complete full world dominated by bots. Again, there's this question of like, why?

If something is working well, then humans are able to sell stocks on a regular basis and make money and occasionally lose money as well.

Why even bring in bots here? I guess it's largely because of scale again. That bots can do something that humans cannot do, which is keep a watchful eye at every single second, be more reactive the moment something changes.

And then also to learn from some of the patterns on the stock markets.

So one of the things I saw is it's not so much that stocks are more capable than humans, but stocks can immediately jump in as soon as a particular stock starts to fall and someone wants to sell their share.

Is that part of it that stocks are just incredibly reactive and then immediately jump in response to different trends?

This is like parallel to what we were talking similarly to having a bot that tries to do credential stuffing on account.

You also have in the stock world, you could have a monkey behind a computer, which is like your bot, and it's constantly looking at a stock.

Stocks have been so volatile lately, but if a lot of people buy suddenly, stock goes up or goes down, you could be behind a computer and you could try to issue orders, but sometimes you have to be fast in a millisecond level.

And that's where everyone is putting their money, trying to create bots to maximize their profits.

If you have a bot constantly listening, it's constantly polling, it can pull the best beats, it can make the tradings in a millisecond timeline and maximize the profits.

So if you want to win in this world, like you had to start depending on bots and not just like a human behind a computer.

Here's a question though.

I mean, look, if I get a bot and I'm using it to sell all of my shares and purchase shares in the stock market, and you also get a bot and pretty soon everyone has a bot, does that mean that we've kind of leveled the market in a way that there could be no upward or downward swing because everyone has the exact same bot going at it and doing their best?

Well, I wouldn't, I don't think so because there's already a lot of, there's many websites that provide a lot of botting services to trade stocks.

Many are free, many are more expensive and big companies use them, little guys use them as well.

I think it just depends. There's still, it's almost an art working in the stock market.

The bot, let's say it's like a tool that might help you be more precise.

There's a lot of patterns, a lot of things that you will fall down and repeat it, but the bot is just going to be a tool to automatize it, make it faster, take out the emotion out of it sometimes.

But there's still a lot of art around it.

You could give everyone a bot and we could still, some people might do more money and other ones might do less because it still depends on which stocks you want to invest and what's your betting strategy, what's your thresholds.

There's so much customization around these bots that I wouldn't say everyone would be the same.

There's so much uncertainty around it, but bots definitely are pushing this, making more accessible to everyone.

Today, everyone has Robinhood and a lot of all these training applications and we're getting this wave of access into the stock market that we never had before.

I feel like the next stage is going to be the ultimate stage where not only you are in the market and you're constantly trading, but you have actual, everyone's going to have maybe their own bot to start making their trades.

I think that's where everything's starting to move now that we're getting our feet wet playing with the market.

Yeah. Well, and that's where it really becomes an arms race, right? Because even if everyone does have a bot, like you said, there's an art to it.

And so it's not the fact that everyone has a bot, that still leaves us in competition.

It's who has the best bot, right?

Who has the best underlying algorithm that's able to study all sorts of traffic and then make decisions when these kinds of things happen, right?

I've looked into, there's basically a practice called backtesting where a lot of folks will come up with different algorithms and then they will retroactively run those algorithms.

They'll try to see how those algorithms would have performed on past market traffic and market trades to understand whether or not they should use those algorithms in the future.

And I think this is probably, look, I'm no stock market expert, but I'm sure that bots will be used in this way to be trained on past traffic and then see which ones would have predicted all of the surges and then falls in the market so that we can then find one that prevails going forward.

And I imagine that's going to be a pretty sought after technology, right?

If someone gets the holy grail of bot algorithms for the stock market, everyone's going to want it, right?

Yeah. I feel very parallel to how, for example, Elon Musk is making trips to the space and rockets, all this accessible to smaller companies and pushing it so everyone invests their resources.

So I see it very similarly in the stock market where we're having all these tools, all these applications, all this world of information available to us that we're moving, it pushes them to the next step of how we do this.

Yeah, I wouldn't wonder maybe in a few years or less than that, for sure, that everyone is going to share different sizes of volumes, try look at historical data, use bots that make different types of bits or different types of cells to change the prices.

So it's getting there.

It's pretty amazing. I mean, look, as we see sort of the democratization of the stock market as well as other areas, I think more people are going to get involved and we'll see a lot more things changing as well.

We're kind of coming into the last couple of minutes here and I can't let you go without talking about some of the stuff that you've been working on lately.

Why don't we start with Bot Fight Mode?

Could you give just a really quick summary of basically what Bot Fight Mode does for our customers and who can use it?

Yeah, so basically one of the main ideals that we have here in Cloudflare is to make the Internet better, provide as much free resources to our customers and just generally provide better, make the Internet safe and faster and everything we can to make it great.

And we saw that we strongly believe that everyone should have access to bot management as a basic requirement to have a website.

Most of the traffic in the Internet is actually bot, I believe maybe more than 50% is bot related.

So we decided to create a new future as part of our bot management, which we call Bot Fight Mode.

And basically it's a free option that anyone who registers their website with Cloudflare, they can put it in front of their website and it immediately kicks in and starts detecting bots.

But not only that, it has the capability to work as a deterrent and as an offensive mitigation.

It provides highly competition challenges when it identifies bots and that really trains the resources that these bots hosted in hosting providers use.

So yep, and that's accessible to anyone who wants to try it out.

That's a pretty wild approach.

So this is literally for anyone who uses Cloudflare, regardless, you can be on a free plan and turn on Bot Fight Mode today.

You just go to the dashboard and I think we've got it under our, you'll go to the firewall tab and then actually find our tools sub tab.

You'll just be able to flip Bot Fight Mode on and we'll immediately start doing some work against these bots.

And just to really focus on the challenge component, the reason it's called Bot Fight Mode is because we are fighting back against these bots.

We're not just blocking them. We're actually issuing these challenges that Abraham mentioned, which are computationally expensive.

What that means is we are going to cause some financial pain, if you will, for people who are trying to hurt others on the Internet.

If you try and go after someone, we'll release this challenge, which then causes a lot of work on whatever cloud server you're doing and you will get a big bill.

So this is kind of Cloudflare is doing its role, doing its part to really go after malicious actors on the web.

And we want to do it in a responsible way as well. I would encourage everyone to check out our most recent blog post on this.

We're doing things like offsetting carbon emissions by planting trees.

So we're running out of time here, last couple seconds.

Abraham, thank you for joining me. We'll have to set up another one so that we can talk about this even more.

For sure. Thank you for inviting me.

See you tomorrow.