How to Prepare for Black Friday

Hi everyone, my name is Paolo and I'm a Solution Engineer at Cloudflare. I'm joined today by my colleague Vences, which will help us through the session. So together we want to spend some time discussing about peak period and Cloudflare configuration. So obviously we are getting really close to peak trading time. We have events fast approaching such as Black Friday and we want to understand a bit about what we can do to obtain the best performance and reliability from your e-commerce platform. In one of the previous sessions we had our colleagues Dion and Simon that explained a few key points around caching and around logging and today we're going to continue this discussion and experience a few more areas within our discussion here. So what we want to do today is something a bit dynamic. So we will be putting ourselves in the shoes of our own users and we will be asking questions about this topic and in particular we will try to put ourselves in the shoes of let's say someone responsible at e-commerce business about being prepared for Black Friday. So I will be starting as the user and I will be greeting Vences with some questions trying to extract some information from him. I'm sure he's going to love it. So I'm going to start now and I'm now going. So as we get let's say closer to the peak trading days I'm starting to wonder what I can do to protect my infrastructure and specifically we have an API which is heavily used by our mobile apps and also by some partners. We noticed that there are peaks of traffic hitting this API and we want to understand if we can protect against threats. Is there anything that you would recommend me to look at for this scenario? Yeah sure. Thanks Paolo for asking that question. So maybe before talking about how to protect your API and what feature I'm going to talk about, something you need to notice before going to any kind of protection is to make sure that the endpoints where you're running your API is in proxy mode under the DNS tab. So this will be the first step that you need to do. Then when it comes to API endpoint protection, so we have a feature called rate limiting that could help to protect your API against specific threshold. So the way it will work is you will customize a specific threshold on the dashboard and you will make sure that traffic will not go over that specific threshold. Yeah that sounds likely because we noticed there were peaks of traffic coming from very specific IP sources so I think it could help. Do you have any best practices for setting this up? You know how can I set it up and monitor it? Yeah sure. So just before going on the best practices, just if I come back to your use case about API protection for your device or your application that are currently used on your device. So some nice feature around the configuration. So it's highly customizable under the dashboard and as you can notice on the right hand side, so there is a sneak peek of what you can configure and you will notice that you can customize the response. So here what I will recommend for you is to use the custom JSON where you can specify any schema that you're expecting to receive on your application and that will be understood by your application to display any custom message for your user. So when it comes to now the recommendation and some best practices, so the first one I will give you is to run the rate limiting with different thresholds. So usually we recommend to create three different thresholds with the action as simulate. Just to notice the simulate is only for enterprise customer but you can create those different threshold and notice which one has been raised and as soon as you found the right threshold, then you will be able to start and for them and maybe block the request if it's go over that one. You can also keep one as an emergency. For example, you create a rate limiting for your API endpoint and you can disable it and when you need it, just enable it and in less than 30 seconds the action will fire. Cool. Yeah, that would definitely be useful. I think especially when you mentioned the JSON response, obviously we have our mobile developers that require a specific response in order to take an action. I think it would help them and it seems like I'm able also to review and inspect the outcomes before I enforce an action which is good. In general, I was thinking we discussed the API now but we also have an e-commerce website where users can log in and they can look at their orders and they have their personalized information. Is there anything I can do to protect, for example, the login and registration endpoints for my webpage? Can I do anything similar to this? Yeah, you can definitely use rate limiting for that. As you can notice on the advanced criteria, you can define which kind of method you want to apply the rate limiting. For a login page, as you mentioned, the post will be definitely the good fit. In terms of traffic matching specific URLs, you will need to put here the URLs that are matched on the post request. So, this will be my first recommendation when you will go for protecting a login webpage. So, using a rate limiting web source different method. Yeah, cool. Obviously, then I can review that in my overview for the analytics and see how it's doing. That sounds quite useful. If we move, let's say, the scenario a bit now and we think one thing I've noticed, for example, and I'm reading as well, I read a lot about some sites which are a victim of credential staffing attacks and perhaps where there is a more sophisticated approach is not just one IP which is hitting my endpoints. Sometimes I see requests coming from different actors and I'm not really sure even where they are coming from. Is there anything that we can do to help with this more generic case? Yeah. So, when you will come to more advanced scenario, like you mentioned, like credential staffing, you need to have also more advanced tool on Cloudflare. And we are for that as a bot management. So, bot management is another enterprise customer and it will help you fighting against those kind of attack. So, maybe we can define together what is credential staffing. So, as you mentioned, credential staffing is the fact to reuse credential from previous breach and using or using a dictionary attack to guess a password. So, those kind of attack is often originated by automatic source. Maybe another one could be interesting for your e-commerce website is the content scrapping. So, content scrapping is the fact to steal content from a webpage and creating a mural of that website. So, as you came closer and closer to Black Friday, maybe you can have some deviation of that attack with price scrapping could be also another thing that could eat your website. Yeah. It does definitely sound like scenarios that are really relevant for me. Maybe my competitors wants to look at the price that I'm putting some of the items on and try to get a better one. And how does the bot management help me? I mean, I understand the concept behind the attacks, but what is bot management doing to help me with all this? Yeah, sure. So, for the bot management, so how is it working? We will assign a score for every request that we go through Cloudflare. So, as I said previously, you need to have the endpoint proxy on your DNS tab, and then every request will have a score between one and 100. One is for most likely automatic traffic, and 100 will be rated for requests, which is most likely human traffic. So then, how we define the score is based on three different engine. So, the first one is heuristic engine, which is a base of known criteria that makes the request tell us that the request is automatic. The other one is the machine learning. So, because we are doing at the scale, so we have a lot of information, and we gather all those information to have a better security, and we can learn from that and protect better your property. And the last one is a behavioral analysis. So, the behavioral analysis is in charge to look at requests against some baseline, and to evaluate that request against the baseline, and see everything that are not as usual. So, those three different engine will assign a unique score, as I said, between one and 100, and is the way our bot management is working. Cool. Now, that makes sense. In terms of like operationally, how I understand I get a score, and I can play with the score, but how do I configure this? Like, do I need to learn a new tool, or how does it work? Yeah, great question. So, everything is integrated into the dashboard. So, to configure those rules, you can either use the firewall rules. So, as you can notice here on that slide, on the right-hand side, there is a bot threat score, which is the score I just mentioned before, and you can play with that score and add new criteria into your firewall rules. So, basically, you will leverage that score inside the firewall rules to create rules that will block or challenge requests based on your needs. Another way to do it will be via workers, but I will not cover that one right now, but it's something to keep in mind. Yeah, I will ask you that one another time. Okay. In terms of like best practices, I mean, I can create the rules in the WAF as usual. It seems like I can use the same tool I'm already used to, which is good, but how do I arrange those rules to achieve maximum effectiveness? Yeah. So, when you can run best practices and also analyzing. So, usually, we recommend to go with two different steps. So, the first one is the detection steps, where you will create those rules that will apply for a specific path or for a specific subdomain or any other criteria that you want to create. And then, you will put those rules in log mode, and we will use the analytics platform or either the logs that you are redirecting on your infrastructure, as Dion mentioned during the first session. And you will learn or discover based on that, and you will adjust your whole to be more specific or to exclude some specific traffic that you know it's a good bot for you. Cool. So, second step. So, as soon as you discover, so now you know your traffic, you isolated what is good and what is bad for you, then you will be able to put those rules in action. So, like block or challenge as any other firewall rules. Yeah. Cool. That makes sense. How do I oversee all of this? Like, what kind of visibility do I have on this setup? Yeah, good question. So, if you go on the overview and the firewall, so you will get here a nice dashboard that will show every request that are matching your firewall rules. And if your question is also like, right now you don't have this add-on as a bot management, and you want to get an overview on what's going on, if there is like any bot issue, et cetera. So, we release a really nice dashboard and there's a bot traffic where you can get those analytics and see if bot management will be a good fit for you. Yeah. I would definitely go and have a look now and see if I need to turn this on before Black Friday. Yeah. Great. Cool. So, now we will change the world. So, I will be the client. So, you will be on the Cloudflare site. Is that okay for you? Yeah, that's fine. That's this time. Yeah. Okay, great. So, during our last Black Friday, so we noticed issue because some bad actors managed to bypass our firewall. And we were able to... So, those actors are able to DDoS our infrastructure directly. So, do you have any recommendation when it comes to my configuration on my dashboard where I can maybe spend time and make sure that everything is set up correctly? Yeah. Yeah. That's a good point. And it's actually one of the first thing we discuss with all of our clients and all of our users in general is make sure that your region is configured in a secure way. And I would say the first step, which we briefly discussed as well before, is when you're proxying traffic to Cloudflare and you are setting up, let's say, the rules, you need, first of all, to make sure that your role is orange cloud. So, essentially, that is behind Cloudflare. And that means that if a user is connecting, they will be seeing the Cloudflare IPs that have been assigned to you and not, let's say, the origin IP that is behind that only us we know and we can go to. In terms of that traffic, so the lack of traffic that is going from Cloudflare to your region, what we recommend is to make sure that you are allowing only the traffic coming from Cloudflare IP ranges. And this can be done on your load balancer or your firewall, depending on your infrastructure. We publish those IP ranges online. They are available on our website. We also have an API where you can get them. And I would also recommend you to review our telephone provider. We have a provider there which is used for all the configuration in general. And specifically, it has a resource for getting this information in a very automated way. Oh, great. So, yeah, Terraform is definitely a good one. So, my team use Terraform to configure everything. So, definitely a good thing I will look at. So, if I understood well, so the first thing you recommend is to have subdomain proxy to make sure that Cloudflare is securing and also doing performance on that specific subdomain. And I can also restrict IP on my origin site. Is there any other configuration I can put in place on top of the IPs that you just mentioned? Is there any other thing that I can do? Yeah, absolutely. We have a couple of things and we will focus on one in specific. So, if you think about the route of the traffic, so we are going from the user to Cloudflare and that's typically encrypted. We're going to look at that in a sec. And then also you have the link from Cloudflare to the origin. We already set up the allow listing for the IP ranges, but you can think someone could be creating some spoof traffic, maybe trying to pretend they are Cloudflare in terms of the IP. And what you can do on that is to configure the authenticated origin pool feature that we offer. With this feature, when we make a request to your origin, we will also include a certificate in the request itself. So, what that allows you to do is to verify that certificate when you receive the request and make sure that it's indeed coming from us. In that case, even if someone is trying to spoof the traffic, they will not have the ability to also include that certificate, let's say. Yeah, great. So, any other thing maybe? Is there another step I can take? Yeah. So, first of all, you can use a generic certificate, but you can also set it specifically for your zone or your hostname. We also have another way that is going even a bit further. You could completely lock down your infrastructure and accept traffic over Argo tunnels, but maybe we will talk about that in another session. Yeah, okay, great. Yeah, definitely interesting. So, now I know how to configure my origin to make sure that the traffic is coming from Cloudflare. So, now let's move to the different area. So, I see on your schema that there is a TLS, the first TLS uncheck between your client and Cloudflare. Do you have any recommendation on what I need to take care on that end? Yeah, that's a good question. Indeed, we really have a lot of configurations available in the dashboard to fine tune and tweak that first leg between the user and Cloudflare. I would say, although there are a lot of configurations and I recommend you to review all of them, I would cast your attention on the ones I have now on screen, which are really easy to understand and quick to deploy as well. The first one, minimum TLS version, pretty much says what we have. It does what it says, sorry, and it allows you to enforce a minimum version on which you want to accept an incoming TLS connection. This is really good. For example, there are some regulations such as PCI compliance, which require you to have a secure version of TLS. We know that, for example, TLS 1.0 is dated and is not really safe any longer. So, you can quickly just change that and suddenly, all your clients will be connecting only over TLS 1.2. Another one which I would call out is the TLS 1.3 toggle. Obviously, TLS 1.3 is the latest incarnation of the TLS protocol. It has a lot of benefits, both in terms of security and also in terms of performance. Before, you were telling me that you have a mobile app. So, one of the benefits of TLS 1.3 is that it removes some of the, let's say, handshakes required in order to establish a secure connection. This is really helpful if you are on a mobile and you have maybe a spotty connection, the data connection is not great, and it really helps to have those extra performance. Finally, we also have a toggle that allows you to bump easily all the traffic from HTTP to HTTPS. So, if I, as a user, try to connect over HTTP, we can automatically redirect you to HTTPS. I would recommend you to look at those settings, first of all, just because it's probably something you can sort out before Black Friday is released. Cool. Yeah, it's definitely some option I will look at more closely. Any specific recommendation, like can I check what kind of TLS version is used right now by my client? Is there any way I can get information on Cloudflare? Yeah, we have a graphic that essentially shows you the breakdown for each TLS version. In general, before changing these settings, what I would recommend is to make sure that, first of all, you try them on your staging environment. Secondly, you review your traffic to see if you have a lot of traffic coming from a client that needs TLS 1.0, for example, for some reason, maybe it's a legacy integration that you had in play. Also, for enforcing HTTPS, make sure you try that first and make sure you don't have any mixed content warnings. We can help with that as well, but yeah, those are the main things I would highlight for you to check. Yeah, great. Thanks. Just cautious of time. I know that he only stayed like four minutes. Do you maybe quickly speak about a different use case? I have some freeze in term of my infrastructure because we are more close to the Black Friday. Is there a new way to work and do some change on the request that's going to Cloudflare? Is there anything I can do? Just maybe quickly introduce that. Yeah, absolutely. That's a fairly common problem. There is always someone that is running to the dev team the day before Black Friday and ask for a last minute change. I think what can really help in this scenario is our serverless platform. If we look at Cloudflare Workers, for example, what you can do is rather than go and beg your developers to break the code freeze, which they may not like, you may want to set up a Cloudflare worker and do that change on the edge rather than on your region. If we look at the we have here on screen, you have a products API that used to give products information, such as the title of the product and the price and so on. You maybe want to change the way this API responds. For example, the API currently responds with prices in US dollars and you need to change it to be British pounds. You could do that change on the origin and upset your developers, let's say, or maybe you can set up a worker on the edge. This worker will be globally deployed across all of our data centers and we will respond from the closest one with respect to the user. For example, this worker could be set to run only on the API path for your API. We would forward the request, we will get the response, same as usual, and then we can tweak the response in the worker code to do the currency conversion and send this back. It's really a quick way and to wrap it up, if you have any problems with it, you can turn off the route and your traffic will continue to go as normal. You can experiment quite easily and do last minute changes in this way. Great. I think we covered a lot of topics during the session today. It was very interesting. I will look at more different workers, definitely a nice product that will help me when I will come close to Black Friday. Just to wrap up the session today, thanks everyone for joining us during this Cloudflare TV segment. We wish you all a happy peak season. As a reminder, we covered during these two sessions, the first one was more around caching and logs, and today was more about rate limiting, bot management, how to protect your origin, and TLS optimization. Finally, we covered workers. So thanks a lot, Paolo, for being here with me during this segment, and we wish you all a happy peak season and see you soon on Cloudflare. Thank you, guys. See you soon. Bye.