Banter with the Bots Team: Episode 2

All right, and we are live. Hi everyone and welcome to our second episode of Banter with the Bots team. My name is Ben Solomon. I am the product manager for bots here at Cloudflare. And usually what we do is we get together a couple folks from my team and we talk about the stuff we've been working on. It's usually bots engineers and me, but I am really excited because today we have a special episode. We are here to talk about Super Bot Fight Mode. This is a feature we launched during security week, so over a month ago now. And the really cool thing about Super Bot Fight Mode is it was an incredibly collaborative project. This was not just a bots engineering thing. It spanned a lot of different teams at Cloudflare. And that's one of the reasons that my two guests today are both not on the bots team. They're on really cool other teams here at Cloudflare. So I will let them introduce themselves. Mari, let's start with you. Can you just give us your role, your physical location, and then if you've got one, a fun fact. Okay. Hi, everyone. My name is Mari. I'm a product designer in the EMEA zone, which is Europe, Middle East, and Africa. Mind you, at the moment, it's mostly just London and Lisbon, so fairly smaller. I've been at Cloudflare since August last year, and I work on all things security. Awesome. Oh, yeah. Oh, fun fact. I play roller derby. Roller derby is basically a race on skates, on roller skates, where you're allowed to hit people off the track. And the point of the game is getting hurt, but getting hurt legally. My favorite thing about fun facts is I'm learning these as well. I did not know that about you, but we got to chat more about that. David, I will go to you. Same questions. Can you tell me about your role, where you're located, and a fun fact. Yeah. Hi, I'm David. I work on the London product UI team. As that sounds, I'm based in the London office, although I'm physically located in Brighton, England, so on the south coast. And my fun fact is that I was a big fan of pro wrestling when I was a kid, and I still am now. And when I was in my early 20s, I was like, yeah, let's give it a go. So I trained to be a wrestler for eight months. It was a crazy time. Turns out we got a couple athletes here. So as we sort of get into this, before we even talk about SuperBot Fight Mode, I think it's really interesting, the dynamic that the three of us have. We're all on different teams, and I'm in San Francisco as well. So we're on different continents in some respects. For you two, how do you work together when, you know, Mario, you're on the design team, David, you're on the London UI team. I guess Mario, I'll ask you first, what's it like working with David, and are you guys communicating every day, or is this just a monthly thing? Gotcha. Well, the design team in the EMEA region is actually uniquely positioned because we partake in the same rituals as the Louis team, the London UI team, which is what David is part of. That means that we have daily syncs in the morning where we chat about what we've done the day before and what we're intending to do. And it also gives us the opportunity to start iterating and working together as soon as we hear about projects, pretty much. So when me and David work together, it doesn't work in the usual way where I will receive a product's requirement, then I'll put some designs on paper and then hand them over to David, and he'll turn those into what customers will see. What works is, you as well, Ben, you're involved in the room from the beginning. We all do all of the parts from understanding what the problem is to understanding how we're going to output that in our products together. It's also really fun. I know almost nothing about design, and getting to sit in on some of these meetings with both of you as well as the other folks, you learn a lot in the process. David, I have kind of a related question, but not exactly the same for you. When we started this project, it's obviously very bots focused, right? Super bot fight mode, it's in the name. How much did you know about bots? And did you have to read up to before you really got started? Yeah, sure. So, I mean, I didn't know a whole lot about bots. I knew that that is what kept me from buying like a PS5 and that they scrape for tickets and for sneakers and stuff. But I didn't really know about the mechanics and the behaviors of bots. So that's what's part of, that's what's the really interesting part of my job is that I get to work with all these different product teams in Cloudflare and I get to learn about all these different fields of interest. And yeah, it was certainly a learning process along the way. The great engineers on your team obviously helped out with filling a lot of domain knowledge there. Yeah, it was a great learning experience. The PS5 thing is interesting too. We've seen so much throughout the web of when PlayStation 5s went on sale, bots just jumping in and trying to buy all these PS5s. So I get it. It's personal. It's personal. I want to, but before we actually talk about the process of building this, I want to show everyone at home what we actually built. So just in case, if you're not a Superbot Fight Mode user, you'll understand what we're talking about. I'm going to share my screen here and this is, we're going live here and just testing things. Mari and David, I'll use you to kind of check me. Can you see what I'm looking at? Yes. Great. So if you go to the Cloudflare dashboard as a free customer, a pro customer, a business customer, any customer, you will have some form of bot analytics and protection now, which is really, really cool. And today I'm going to demonstrate the business version. If you have a business plan, you don't have to pay extra for this. We've just given all of our bot capabilities to folks on business plans and pro plans across the board. So there's two different parts here. The first thing we launched is a version of bot analytics that is part of Superbot Fight Mode. And you can see it on the screen here. We are showing you chronologically all the different traffic that's come into your site, and then we're breaking it down by the type of traffic. So this purple region over here, this is definitely automated traffic. It's traffic we are really confident comes from bots. And across the board, you can see verified bot traffic. These are the good bots like Google, like Bing. If we move over here, you can see likely automated traffic. Traffic we're pretty confident comes from bots. And I'm looking at a demo zone here. So there isn't a lot of traffic to look at. But if I kind of throw this back, I'm going to go to a full week. So we're looking at tons of data here. You can see that during different time periods, you'll often get and very commonly get different types of traffic. So in this particular period, we had 10 requests from verified bots, as well as 20 from likely humans. So this is really cool. I'm going to kick it back to 24 hours just so we pare it down a little bit. We also will show you data like the actual detection engine that generated a particular bot decision. So you can see that, for example, our machine learning engine accounted for 40 of those different bot decisions that showed up. All these different sources, these engines are completely filterable in the dashboard. This is some of the cool stuff that Mari and David helped build. And then at the bottom, I've grayed out a couple things just because they're sensitive information. But we do show you the IP addresses, user agents, paths, all this good stuff so that you can add these filters and stack them up to get an idea of your bot traffic. Now, one other thing before we get into conversation here, if you click on configure super bot fight mode, we let you actually act on all this information because it's useless if you're just looking at it, right? So you get a couple different choices here. You can choose to block, allow, or challenge any of your definitely automated traffic. You can do the same for likely automated. And we give you other controls too. So you've got choices. And if you're not feeling super comfortable with aggressive bot protection, you can dial it back just a little bit. So I'm going to pull this off of my screen, and then we can get right into it. Give me one second here. All right. So we have talked about super bot fight mode. And it's obviously available to everyone who has a Cloudflare account. Or you can get the bot fight mode, sort of the regular version if you're on a free account as well. Mari, I'm going to go to you first. I came to you with a big vision for this. I knew it was a wide reaching project that reached all these different plans. Where did you even start on design? I think it's important to note, I wasn't even the first designer on this. We had someone called Maggie Yu. She's a very talented person. However, she was unable to continue the process until then. So when I took it up, it was pretty much, let's solve all of the analytics bits for the enterprise customers, which is kind of an important part of the product, and then see where we go from there. So to start off, we had to understand what are the types of analytics that people wanted to have, what kind of internal available data we had, and how would people even want to interact with this. So that was pretty much the starting point. Now, afterwards, we had to figure out how to get this really, really hefty products and very sturdy solution that we created for enterprise, and phase it down so we could democratize it and offer it to all of our different types of customers. So this is where super bot fight mode comes in. We try to understand what level of handholding and what level of granularity was going to be relevant and appropriate for each type of customer on different plant types, which was the really exciting bits of things. I heard that siren in the background, and I was thinking someone saw a bot. They're going to stop traffic from their site. This is super interesting. The other thing here is I remember going through the initial kind of spec process and saying, this is what I hope we can do. And I made some very bad mockups because my design skills are horrible. But when we started chatting about the different design approaches here, a lot of the things I was hoping we could do might not have matched the Cloudflare aesthetic, right? They might not look like what we already do in the dashboard, and we might not have components for them. What do you do when someone like me shows up or someone else and is like, I'd love if we could do this, but there's no clear template to fit it to? Right. I think the first thing we have to notice is that when someone comes to you with a requirement, the first question we'll ask is, do we need it? Once we have determined that, yes, we need this, this is an important thing, we try to understand whether there are other visual components or ways that we've addressed it inside our products that could apply to this. In the case of bot analytics specifically for enterprise and what we phase down to super bot fight mode, there were some things that didn't exist in our system. So we look outward. We see if there are other best in class examples. If we find anything, great. We bring those learnings in. But the fact of the matter is, once we decided what we wanted to do and how we wanted to do it, we work in conjunction and like in tandem with the UI team, namely David and Nick Downey and Beth, who were all working with us together and the component design team as well. So that means that it's not just something that is going to be used once in our dashboard. If we're going to create something new, we want to make sure that it's something reusable and something that could be relevant in other parts of our dashboard. So no one offs, but rather things that are systems, not just solutions. You're building things that can be reused in other places, right? They're going to outlive the products that they were meant for. I like that way of thinking about it. David, let me jump to you. Mari brought up the fact that there were a lot of different folks on your team who were involved with this. And I know in particular you and Beth Knight spent a lot of time working on the front end. How do you do that? Especially you're both remote, you're both contributing to this one sort of body of work. What's that like collaborating with someone else on a project like this? Yeah, it was an interesting process, especially working on like the initial bot analytics products as well. There's a lot of trying to find our feet there, kind of saying how the flow is going to go for this kind of thing. Mainly it's about communication and making sure you don't tread on each other's toes, making sure there's kind of as little overlap as possible when you're working, just so you don't run into costly problems with conflicts and whatnot. But yeah, it turned out to be a great test of our communication skills and I think we worked pretty well to get this done. Do you find that having someone else to kind of bounce ideas off of is helpful for creativity in general? Or is most of what you're doing just kind of implementing a lot of the things that Mari has helped design? Sure thing. I mean, Mari and Beth and I were kind of bouncing ideas off each other the entire way. It's never like, as we said on software, it's never like just a handoff process. It's a constant case of collaboration and bouncing ideas off of each other and iteration and kind of, oh, does this work? We'll try to test it out. And then if it doesn't, we go back to the step before and say, okay, what other direction can we go from there? It's a really interesting and fascinating process, especially remotely. Cool. The other thing that has always struck me about your work and the work that Beth does is front-end work is a little bit deceiving. It's the first thing that customers see when they show up in the dashboard, but it relies on all of these other pieces that are kind of built below it, right? And this has got to be an incredibly difficult thing to start building, right? When sometimes you start working on visual dashboard components and none of the underlying structure is there yet because it's being built in tandem at the same time. How do you manage that? Yeah. It can be difficult, especially if there's a bit of a delay between the requirements I need and being able to actually get started on it. But at the end of the day, it's about just kind of effectively communicating with the engineers that you're working with and being like, okay, this is what we're kind of expecting. This is what would work for us. And then being able to tell you, okay, well, this is what we can give you. And then from there, hopefully you can mark out or design something around that makes sense and requires as little adaptation as possible when those systems do become available. I think I want to transition to talking about something that you both have mentioned so far, which is bot analytics. Bot analytics is, we should say, this is not our first rodeo together. The three of us have collabed on a couple of things in the past, in particular, bot analytics, which was last fall. And part of our task with this was figuring out a way to take those kind of enterprise analytics, the very, very specific detailed visuals and bringing them down in an appropriate way for our self-serve customers, right? Making sure that this fit their daily needs. Mari, how did you think about scoping things down in a way that would be useful and not just limiting features, right? Got it. Yeah. I think the question that we tried to answer was really how simple could it make it to customers to sort out what was their automated traffic from what wasn't. And when it came to configurations, what was the level of just one clickness that we could apply depending on their account level type? So really, so they could see what the analytics look like, however simple, and they could see what the configurations were doing in real time. So. Got it. Is that generally your approach that with self-serve customers, they're looking for these kind of one click solutions that are sometimes don't require as much background and training, but can accomplish just as much? I think it's part of the way we do things at Cloudflare to democratize all of the things that we have. Sometimes we offer the same things, but with less granularity and with less options to go into the nitty-gritty of the configurations. So it really is about how much granularity do you want to see? Then you go up plans or go down plans. So we tend to follow that kind of framework. Got it. What was the most difficult part of kind of adapting analytics to self-serve customers? I think the most difficult part was understanding just how little or how much did they actually need. So when it comes to super bot fight modes, we still give customers a fair bit of information when it comes to details about the automated traffic. However, unlike enterprise, we don't give them the opportunity to look at concepts like bot score or creating firewall rules from those analytics. We just give them simple controls to respond to the numbers that they are seeing. So yeah, I think the biggest challenge was understanding how much they needed and also making sure that everything was clear. When you're talking about things that are being offered, they're similar, but there are different offerings for different levels of accounts. There's a lot of content that comes in. So there was a lot of thoughts put into every single word that we put on the screen. So when we talk about design, we're not only talking about what do the graphs look like, but we're also talking about the entire content of the page, be it the data or be it every single word that goes in it. You were mentioning it's really important to make sure things are clear, right? Whenever anyone is using the dashboard. And the easiest way I can think of to do that is to actually test it with real customers. How do you do the testing process? Are you building tons of different mock-ups and then who are you testing it with, right? How do you go out and find the right base? We have a lot of very effective customer proxies inside Cloudflare. So this would be accounts, what is it called? Success managers. You're talking about customer success. So we're actually working with the different contract customers to see what works best for them. Correct. So if we don't have direct access to customers, which can be a little bit tricky, we can talk to these people who deal with the customer problems on a day-to-day. They are really like the first line of defense and offense when it comes to tackling any issue. So we're very lucky that we can talk to them and get their understanding of things as we are doing things, rather than waiting until it's done and then finding out whether it works or not. All right. So David, I think we've kind of got a picture of the process here, which is there's a lot of back and forth, right? Mari will come up with some designs and we'll give feedback and work through what may or may not work in the dashboard. Once you feel like the mock-ups are in a good place, how do you start constructing all of these different plan versions in the dashboard? I think, yeah, so we kind of build from, we have a set of established patterns and established use and context in the dashboard. So we kind of build from there basically. We'll find the bits that are easiest to put together in a way, and then anything that needs changing, anything that needs adapting, anything that's specific to that product, we'll kind of add from there. In the case of a super bot fight mode, because we already had the enterprise bot analytics products available to us, it was really just a matter of finding the most common elements to each of the plans, kind of adapting them as we go, and then branching from there and making any plan differentiated features available from that point. So, yeah, it's really just about branching off of the solid core of the product and yeah, that's pretty much it. Is there anything you learned from your experience building bot analytics that changed how you approached super bot fight mode? That's interesting. That's an interesting question. I can't say there really was. It's a very similar data set and we kind of had a great idea of how it behaves after doing bot analytics. So, scaling down from that was pretty straightforward. I think it was interesting to see the kind of view that you would get when you did pair that granularity back and what would be available to each customer. I could really see at that point kind of what the use case was as we kind of went between plans. I imagine that must have been a tedious process when you sort of have this enterprise product that's fully built and maybe you build a different version of it for business, but you have to keep reconstructing a different version for each tier. Did you find that difficult, frustrating? Was it fun getting to kind of pull different features and different levers in different ways? No, no, no. It's always fun to put that kind of thing together for sure. As I said, I like just seeing kind of like the different views that you can achieve it for different plan types and kind of how we accommodate for different users. It's always interesting to see. Yeah. Mari, let's go back to you for a second and talk about some of the design work that went in throughout the process here. I remember getting links to Figma mockups and being able to look through the entire flow of SuperBot Fight Mode. What tools do you use? Is Figma one of the main ones and what else is there? Right. Well, it really depends on what phase we are in. If we are in a problem definition, project discovery phase, we tend to use a whiteboarding tool called Miro, but really anything goes. If we actually had a whiteboard, we'd use that as a tool. But mainly we use Figma for its collaborative aspects, meaning that loads of people, anyone in the organization really can join in and see things real time. And also for the facts that people can leave comments. So it creates a good place for async feedback loops to happen and also a unified source of truth when it comes to at what stage we are at and where we're trying to go. Were you using the same tools before the pandemic back when everyone was physically in an office and able to talk? Yeah, pretty much. I mean, the pandemic has made some things more difficult and some things easier. I'd say when it comes to the collaborative work that, for example, me, David, Beth and Nick Downey have put together doing this, it actually has helped because it means that there's no limits to the amount of people that fit in a meeting room. It means that you can just share your dashboard at any time and work together without having to pull a chair to the table and having to like scooch next to the person. So it was an exercise in shedding the ego of what you're doing, not letting yourself be, I don't know, put off by making mistakes in real time and really, really taking it seriously that things are collaborative. And it's only then that we can make things twice as quick, at least for the bot work. We achieve things in two days that I would normally not have achieved before the pandemic in a week, just because we could have quickly and anytime just jump into a room, sort through the problem without me having to walk halfway through an office, go sit next to someone, awkwardly ask if it's OK to show them something. It made things simpler. There's something really interesting there, which is ironically, even though we are now further apart than we ever were physically, this has become a more collaborative process in some ways, at least from a design perspective. David, have you found the same thing on the UI side? Yeah, totally. I feel like kind of the scenario where, yeah, in an office where you're either kind of crammed into a room or, you know, squished around a desk or whatever, I think that's kind of a, it feels like a very one way kind of flow. You know, someone's usually kind of doing the leading and then everyone else is kind of sitting around. Whereas with tools like Figma, with the design iteration that we do, everyone has access to this like living document of work that we can kind of contribute to and leave feedback on. So, yeah, I think it kind of, you know, yeah, it becomes like a truly collaborative process this way. I always love getting those Figma links. For me, we're also in different time zones, right? So you guys are many, many hours ahead, usually eight or nine hours ahead, depending on the time of the year. And when I wake up and I get a Figma link that shows all this cool stuff that's been brought to life, it's really, really exciting for me. So I always look forward to that. The other thing I want to ask you, Mari, about design is sometimes the changes that we make involve other products, like moving around the physical location of the bots tab in the dashboard, right? And so there's all these other stakeholders who say, well, my tab is here, right? If you move yours here, mine's going to get displaced and go somewhere else. Who do you talk to when there's a design decision like that, that involves many other different teams? I'd say over communication is key. And by that, I mean, getting people so tired of hearing you say the same thing that eventually they start thinking about that thing themselves. So if there is a problem such as whose tab comes first, first of all, we try to understand whether there's any data that can help us make that decision. Even before I go and talk to someone, do we know why the current order exists? Is it based on customer use? Is it based on we wanting to feature something? Is it alphabetical order? So trying to understand the reasoning behind the decision that led us to the points that we're at and trying to align everyone that has skin in the game around what is the thing that we're trying to achieve. It doesn't matter what system is there now. If we want to go somewhere else, we can do it together. But if you decide to do something and not inform other people, it's a recipe for disaster 100% of the time. Got it. Got it. David, whenever you get a bigger change like that, that maybe resonates throughout the dashboard, is that a concern at all? Does it affect how the rest of your team builds UI components? Yeah. We generally don't like to tread on each other's toes. I don't want my changes to affect something else that someone else is doing. So there is consideration for that, totally. We all have the chance to review each other's changes before they get merged. So that's a great opportunity to point out any kind of incoming mistakes that might be happening or any areas that, oh, well, this actually will be changing soon. So don't rely on this being here. You're going to have to change this later, that kind of process. Yeah. We also go through a rigorous testing process of making sure everything works before it gets released out to the rest of the world. And the other favorite thing that I have besides getting a Figma link is getting sort of a new build, a preview link that I can test out for UI components. David, how do you guys put those together? I want to paint the picture for everyone at home. You can sort of build a preview version of what we're working on and then send me a link that I can then test out in the dashboard. What's the process for that link? Yeah. So that's part of our integrated deployment system. It's a great tool for testing kind of partial features with internal stakeholders. So say if I have the analytics for a certain product, but the control plane isn't quite there yet, or we're missing a few data points, then I can still kind of show work in progress, get early feedback on things, especially from people who have frequent communication with customers or who have customer use cases that they're looking to meet. So it's great to get that kind of early feedback, really tighten the feedback loop there. Got it. And it sounds like there aren't any other tools that you're using in particular. Is there anything you want to call out here? No. Outside of the engineers, like bread and butter, terminal, and my editor, no, we don't really use any specific tools to build these things. Main tool is just, yeah, everyone around us and the people we communicate with. With our last minute or so that we have, I want to ask you both a quick question, which, well, I'll go to Mari first. Mari, what was the most difficult part of Superbot 500 to build? The most difficult part was getting alignment with all the teams that are involved. There was a lot of work that involved us understanding how would we really sell this to customers, which meant that we had to be aligned with marketing teams. And that was a new team for me to even meet. So I'd say stakeholder management across different teams and across the org on a multi-time zone world was definitely a challenge, but one that we conquered. Got it. David, with 15 seconds, if you could add a feature to Superbot Fight Mode, what would you add? I want a real-time bot fight mode video game built into it so you can fight bots in real time. That's what I'm hoping we can get to. So this is not a promise, but everyone at home, stay tuned. There may be some exciting video game components being added. Thanks for joining us. Thank you. Thanks.