Transform Rules: "Requests, Transform and Roll Out!"

Hi, everyone. Welcome to another Cloudflare TV segment. I'm your presenter today. My name is Ming. I'm a solutions engineer at Cloudflare. Today, we will talk about transform rules, which is a feature that we wrote out not too long ago. We have published a lot of blogs, talk about the detail about transform rules. If you're interested, please read at blog.Cloudflare.com. Today, we want to utilize this TV segment to help you to understand what are the benefits that transform rules with Cloudflare can provide. All right, so let's see the agenda for today. Today, we will talk about what are the transform rules and what do transform rules work. In the end, I will show you some common use cases that our customers want us to implement transform rules. And if we have some time, we will do a demo so you can actually see what transform rules can implement. All right, so without further ado, let's get started. So, what are transform rules? Transform rule is one of Cloudflare features that can offer to modify HTTP requests to help your application perform optimally. So, what does this mean? When you're developing an application, it's not a one-time job. It is a continuous job because your application is always evolving, your infrastructure is always evolving, and your content is always updating. Because of those, your requests expect a specific input when going to origin in order for the application to perform properly. So, special input with requests might include URL normalization, URL rewrites, pattern modifications, and those can be provided by transform rules. And you might have a question about, I saw my origin server already does that job. Why do I need Cloudflare for this? So, today, if your origin server do URL normalization, URL rewrite, or header modification, it actually does add a lot of overheads to your origin server's capacity, no matter if you're hosting on prime in the data center or on cloud, which let's say GCP, AWS, or JORG. So, it does add a lot of cost to your bandwidth. This sometimes is very expensive, and sometimes it's very hard to manage. So, Cloudflare offers a proxy service. We can sit in the middle between your eyeball traffic, which is your Internet user. So, basically, as an Internet user with your domain, it will reach out to Cloudflare, and then Cloudflare will do all the configurations that you implement with us, and then send good content back to origin. So, when Cloudflare can offload those features off from your origin server, so we do it in the middle, so we have 200 data centers across the world in 100-plus countries, your end user's request can actually result to the closest Cloudflare pops. So, using Cloudflare for transform rule with those functionalities, it can definitely improve your performance and security. If today you're a Cloudflare customer, you might have a question like, oh, I saw Cloudflare Workers already do those functionalities. Why do I need transform rules? Why Cloudflare has a new feature called transform rules? That is a very good question. Cloudflare Workers act as a serverless compute, which is a very powerful product that can help you to modify HTTP requests, response, sometimes do bug redirect or do conditional response. We can also do, like, token authentications. We can actually serve the static content for you. But using Cloudflare Workers for those functionality to modify requests is a little bit overkill. So, that's why we have transform rules, because Cloudflare Workers require you to write code in JavaScript. So, when a customer comes to us saying that they want those functionalities, as I see at Cloudflare, we almost want to always propose transform rules instead of workers, because Cloudflare Workers is for more complex settings. So, all right. So, that's the definition about transform rules. So, let's highlight this slide. What are the key points about transform rules? So, transform rules is to modify HTTP requests. So, let's dive into each individual functionalities. Let's do URL normalization first. So, what is URL normalization? URL normalization is to normalize HTTP requests to a standard format, which allows you to predictably write security rules and filters. So, what does this mean? You can see that here there are two URLs listed here. One is a normal URL. So, the normal URL has normal pattern. It's just basically www.example .com slash login. So, this is a login endpoint, which is very critical to protect, to avoid, let's say, brute force attack. So, you might have, let's say, relimiting rules and fiber rules to protect particularly this endpoint. On the other hand, we also have encoded URL. So, what are encoded URLs? So, encoded URL is using special characters to encode the actual letters on your normal URL. So, we can see that here we are encoded L to percentage sign 6C. So, both endpoints lead to the same content, same destination, same endpoint. So, today, if you have a relimiting rule to protect this endpoint, particularly which is slash login, and at the meantime, you have bad actors on the Internet that issue massive requests using this encoded URL, it can actually bypass your relimiting rules for this endpoint. So, what can you do? So, Kaufer, as a security first company, we don't want that to happen to you. So, today, we are announcing URL normalization to apply all the Kaufer plans. It is very easy to implement. Here, you can see the dashboard screenshot here. We mainly just, you go to your dashboard. We are currently proxy your domain. It's just two settings. One is, do you want Kaufer to normalize the incoming URL? If you turn this toggle on, that means when end users issue an encoded URL, it will result to call Flutter's POP, which is the closest POP that's close to the SMU. Then, if you turn this on, that means we will transform the request from an encoded URL to a normal format. So, that is when you turn this normalized incoming URL on. So, once we transform the request to a normal URL, and then we will apply all the Kaufer rules, relimiting rules, performance features, and security features to this particular URL. Now, there's another settings asking you, do you want Kaufer to normalize URL to origin? If you turn this off, that means after we're scrubbing traffic and we apply performance features and we need to go back to origin. So, when we go back to origin, we will convert it back to the encoded URL. So, that means when we go to origin, it's still using the encoded and original URL, if this URL is encoded. So, your origin will always know what is the original URL, and this is the default settings of our Kaufer account. If you want to override it, let's say you want to turn both on, that means both in the front side or in the later side, we always convert it back to the normal format URL. And, of course, you have flexibility to turn both off and one on, one off. It really depends on your preference. That's everything about URL normalization. It's very easy to set up. You just go to dashboard and complete those tasks, and that will be it. And, of course, it's API available. All right. So, let's dive into the next functionality of transform rules. Transform rules also do URL rewrites. What is URL rewrites? So, URL rewrites is rewrites URL a path or a query string based on the logic of your application. It is not a URL redirects. You know, rewrites and redirects is almost like twins. Sometimes they look similar, but they're quite different. When customers reach out to us saying that they wanted URL rewrites, we always want to double check with the customer by mentioning the difference between URL rewrites and redirects. So, what does URL rewrites do? So, URL rewrites does not change the end user data facing. But a URL redirects actually changes your URL and user facing with a 301 or 302 status code. So, we want to mention the difference to customers so the configuration that we recommended actually meets customers' expectations. So, transform rules do URL rewrites, does not do URL redirects. We have another feature in Cloudflare called page rule, and we can forward URL, and that is for URL redirects. But in this particular feature that we want to talk about today is doing can do URL rewrites. So, what are the functions that URL rewrites can do with transform rules? So, we can do static rewrites. Basically, you will rewrite to a fixed string, fixed URL, and we can also do dynamic rewrites, which you can do a computed string. So, we rewrite to a computed string when we go back to the origin server. So, since Cloudflare offers a lot of product features, whenever you implement a functionality with Cloudflare, you always want to know where does functions that I implemented taking a place in the life of the request when my request traveled into Cloudflare. So, for this URL rewrites, we will proceed before Cloudflare fiber rules, before page rules, and workers. So, you can actually know that, oh, I want to rewrite the URL, but I want to rewrite before fiber rules, page rule, and workers, et cetera, and then transform rule is definitely the solution that you want to take. All right. So, let's dive into an example. So, let's do a static URL rewrite example. When you visit www.example.com with this cookie and you want Cloudflare to rewrite this URL to a www.example.com when going back to your origin server. So, there are two conditions. One is www.example.com. This is the host name. And also, a need to pass the cookie. And once those two conditions meet, and then we will rewrite to slash root one instead. So, this can definitely help you to do an A-B testing when rolling out new features or content. So, let's see how you implement this rule. Here, you should see my Cloudflare dashboard. So, you can configure all the Cloudflare core services within Cloudflare dashboard, and everything is available here, and also is available on API. So, for customers that are familiar with Cloudflare dashboard, we have each individual tabs here. For transform rule, particularly, it will be under rules, which is a filter icon. And there's a sub tab called transform rules. Once you click on that, it will ask you, do you want to create a transform rules? Click on the dropdown. There are two options. One is URL rewrite, and the other one is modify headers. So, we are talking about rewrite URL. So, let's say click on URL rewrite. And then once you click on the URL rewrite, it will ask you to fill out a couple of fields to help us to identify when we should rewrite the URL. All right. So, I have this pre-config on my account to save some time due to we only have 30 minutes. So, let's see what the rule looks like. So, first, we will need to give a name of the rewrite rules for us to easier to identify the rules. And then you can add a lot of filters to it. So, for this particular one, I want to use whoever visit www.haloming.cf. Haloming.cf is my zone on Cloudflare. And with this cookie, version equals to week one. And then I want Cloudflare to rewrite the URL to a static pass. So, you can see there's static and dynamic. For this example, we do static. And I want to rewrite to week one. So, that is actually match the requirement that in the slides. For query string, I want to reserve the query string. All right. So, once you save the rules, Cloudflare will apply this rule to match all the requests and match those conditions. But today, if you want to build a more complex transform rules for URL rewrites, you can definitely add more characteristics you can define. So, let's say we can say that the requests come from a certain AS numbers or with certain cookies come from certain countries, continent, the source IPs. There's just a lot of user agent. A lot of characteristics you can define. And if you don't want to use a dropdown or regular expression is working for you, you can definitely add a regular expression here instead of using the dropdown. So, you have flexibility to do both. All right. So, let's see how does this rule do for the request. So, here you should see my terminal. So, on the left-hand side is my local terminal, which is my laptop's terminal. On the right-hand side, it shows means.com. This is my server. So, this is my server's terminal. So, let's see. I want to grab a log from my server to any path that was v1, because we actually rewrite transform to v1 in the v1 path. So, I want to grab the log to v1. So, now I put this command. Basically, I want to tell the log for v1. And then at the same time, I want to issue a request, curl request on my local terminal. So, basically, I'm curling this www.hollowmink .cf with this cookie. So, this request should trigger the transfer rule. And transfer rule will rewrite my URL to slash v1, which is what we are looking for here. So, I press enter. You can see that the request went through. And you can see I have a get. I have a log record here with v1. You notice there's 404, because I don't actually have any content for this path. It's just mainly for demo purpose. So, this 404 is expected. But you can see that transfer rules works for the rules that I just implement to call for this account. All right. So, let's go back to our slides. We did a static URL example. So, let's do a more complicated dynamic URL example for rewrite. So, let's do this. So, this is a dynamic example. The requirement is when any user with any URL with this www.example.com with a cookie of version equals to week two. So, two conditions. Host name and cookie. And also with a query string of page equals two. So, three conditions. Host name, cookies, and also this query string. And it has a slash global in the beginning of the URL. So, four conditions. So, this is host name. This is a cookie. It has to have this exact query string. It has to have slash global in the beginning of the URL. And then I want to rewrite the request by replacing the slash global in the beginning of the URL with week two. So, basically, we're changing the path. And also update the query string of new page equals two. So, there are two places that need to be changed. The path needs to be changed. The query string needs to be changed. So, let's see an example. So, with the previous slide's requirement, we will see that when any user with a www .example.com, let's say this is a long URL and it has global as a master path in the beginning. And it actually has a query string, which is page equals two. With a cookie and version equals two week two source. Those conditions have to be matched. So, global in the beginning of the path and page two as a query string and also version equals week two as a cookie. Cloudflare will rewrite the URL to replacing global to week two and also replacing the query string. So, this query string will be a static query string. So, another example is www.example.com. And this is a long URL. It has global in the beginning, similar to the first one. And also page equals two as a query string. With the same cookie, I want Cloudflare to rewrite replacing global to week two. So, change different paths and also change the query string. All right. So, I'll give I'll pause a little bit for you to digest examples so we can dive into how to implement those. All right. So, let's see how you implement them. So, go to Cloudflare, back to Cloudflare dashboard. So, this you can see, again, it's under rules tab and there's a transform rules sub tab. When you click on rewrite URL. So, it's going to be the same thing. When you click on it, it asks you to fill it out. So, for that particular example, it's actually a dynamic rewrite. So, I have this preconfig here. It's on. So, let's click on it to see what are the details in it. So, first, we need to give a name. So, the name is rewrite URL paths and query strings. So, the condition will be a host name equals www.halloween.cf. So, this is my zone proxy by Cloudflare. And with a cookie. So, the cookie is version equals to wait two. So, this is the condition. And also, with a query string equals to page two. So, that actually matched the condition of the examples. Of course, you can add more filters by clicking on and or. It depends on your preference. And also, regular expression builders, you can definitely do regular expression instead. So, we want all the requests that match those conditions to rewrite. Today, this example, we will do a dynamic rewrite. So, we will rewrite dynamically. So, basically, you will fill out in the regex replace and with URL path here. And so, the global path will be in the beginning. So, you want to add this sign here to indicate that global is happening in the beginning of the URL, which as a folder. And I want to replace global with slash wait two, which is we change the path. So, global update to wait two. For query string, I want to rewrite query string. Of course, the query string can also be static or dynamic. It depends on your use case. For this demo purpose, we will do a static rewrite. So, it rewrites new page equals to. So, once I save this through, it will actually just drop for the rewriting for dynamically. So, let's go back to my terminal. Let me just clear my terminal. All right. So, instead of looking for wait one, let me just clear that. We were looking for grabbing wait two. Remember, we're changing the path global to wait two. So, I'm telling the log to looking for wait two. So, let me just issue a curl. Let's do just examples here. So, on my local terminal, on the left-hand side, I will issue curl with this URL. So, as you can see, that URL has a global in the beginning as a path and also page equals to as a query string. And I also want to issue with this cookie. So, which should trigger the transfer rules that I implemented with my coffee account. So, I entered that. All right. So, you can see the transfer rule actually taking action. When I tell the log and the log, you can see that we will get. So, coffee rewrites the URL slash wait two. So, global is no longer there. So, it changes to wait two. For the query string, you have question mark new page equals to instead of page equals to. So, let's do another example. Let's issue another URL to see if it works as the same. All right. So, let's do. So, I will be replacing different URLs since it's a dynamic rewrite transfer rules. You can see that. So, this URL, I want to issue a request. It also has global in the beginning. And also, this is different URLs and this. So, different paths and also has a query string page equals to. So, those two requests share some similarities. That's why I can trigger the transfer rule for URL rewrites. Global in the beginning and page equals to as a query string. Also, you need to pass the same cookie. So, it matches all the transfer rules. All right. So, once I press enter, you can see coffee will already rewrite this global to wait two and also the question mark query string also updated. So, again, this is as expected because I don't have actual content for it. All right. So, let's dive into the last functionality of header modifications. So, this is another feature that transfer can provide. What does header modification do is add and remove request headers when going to origin servers. So, you can definitely use this feature for string or non-string static or dynamic request header values. So, this is actually the most requested feature that our customers use in call fair workers. Let's say you can use workers at arbitrary headers or dynamic headers, but today you can do it with transfer rules. And also for enterprise customer, if you already purchased our bot management products, we can issue a bot score to each individual request of your zone. So, we can actually add a bot score as a request headers when going to origin. So, as some of you might know that call fair bot management work as a scale. So, basically, each individual request comes to call fair, we will issue a score from 1 to 99. So, the higher the score, the more likely human traffic. The lower the score, the more likely bot traffic. So, let's see how it works for headers modification. All right. So, header modification can add static headers. So, basically, like arbitrary for a cost bar. So, this is static headers. And then we can also add dynamic headers that can expose the user's geolocation. Let's say if your applications need to utilize a user's geolocation to do some functionalities, we can definitely add those user locations or any dynamic headers. Let's see how you implement those. All right. So, back to the dashboard. So, again, rules filters, transform rules. To create transform rules dropdown, there will be modified headers. So, once you click on that, you will see the couple of fields similar to the URL transform rules, and you can definitely just fill out your settings here. So, I have a pre-config rule set. And here, click on the wrench icon, and you can see the given name, static or dynamic headers. I want to set those headers for whoever visits this domain name that I currently proxy by Kotler. So, I'm adding a dynamic header. So, it can add a static header, dynamic header. You can also remove headers, of course. So, setting a dynamic header, you can give it a header name. And also, so the value, for example, this one is adding an AS number. So, you can add this dynamic value here. And another example is I want to add a boss score. So, boss score, which is for enterprise customers who purchase bot management products. So, this value will be available for you. I want to add boss scores and also client continent, which is client geolocation. And I also want to add a static headers. So, basically, arbitrary headers to indicate that this request may come from Kotler or you can use it for a different purpose. So, once I click on save, and let's see what does it do. So, currently, this endpoint, I'm hosting with an HTTP server. So, HTTP server can reflect all the request headers that Kotler is sending to my origin in the UI in the browser. So, let's see. If I visit this, my zone with this path, which is my HTTP endpoint. So, those are all the headers that Kotler is sending to my origin. So, this is not client facing because this is request header Kotler sent to origin, not client sending to Kotler. All right. So, let's see. We added a client AS number, which is here. Oh, we have a boss score. Indicates 99. I'm perfectly 100% human using the browser issue request. So, that's accurate. And we also have continent. We add this headers and also bar, which is arbitrary headers. So, that's how header modification works. So, that's wrap up all the functionality of transfer rules. I hope you enjoyed the session. I appreciate you follow along. I'll see you next time.