🎂 The Future of Page Rules: putting more power in the hands of users

Hello everyone, and welcome to Cloudflare TV for this session today during Birthday Week. And we're going to talk a lot about rules, rules and rules. My name is Michael Tremante. I'm on the product management team here at Cloudflare and I'm joined by three other colleagues of mine who are also product managers Alex, Sam and Matt. Before we dive in, maybe a quick round of introductions. Sam, let's start. Let's start with you. Sure. So, yeah, my name is Sam. I'm on the product management team here in London and I look after the group product manager looking after everything, cache protocols and a whole bunch of a whole bunch of rules. Awesome, Alex. Thanks. Thanks, Michael. My name is Alex. I'm the product manager for cache team here at Cloudflare. You want to go, Matthew? Yeah. I'm Matt. I'm product manager for FL. I still struggle to describe what FL is, but it's everything. A request for Cloudflare will hit and we process. Every single product is built on top of a phone, Right? Okay, great. So before we dive in, a quick reminder to whoever is listening, if you do have any questions, please email live studio Cloudflare or TV. If we get to it towards the end and we have some extra time, we will try to answer those questions live. Otherwise, we will definitely follow up via email later after the session. So let's start with the topic for today. It's all about rules. There's been a bunch of blog post announcements about a lot of new products and features available in the dashboard from today. Matt I remember back in the day we were solutions engineers and we were advising customers on how to essentially configure settings with Page Rules, but I guess we've outgrown that. What is some of the feedback that has led us to sort of rethink Page Rules? What were some of the major pain points? Yeah, I think as you said, solutions engineer. So on the sales side, talking to a lot of enterprise customers, when we shipped Page Rules ten years ago now, it was great. It's held up for a long time, but the main feedback we were getting was one, there was an end of Page Rules people were running out of 125 on Enterprise we had CMS asking for more constantly. Even at an email today after the announcement asking for more Page Rules. And the other one was we could only match on a URL path. Yes, you could do wild cards in that, but customers wanted to match on more so. Ever since you shipped the firewall rules engine and customers were able to configure firewall rules based on all of the cool things such as assigned GEO headers, cookies, people wanted Page Rules to do the same. So we've had that constant feedback and yeah, finally we're in a in a state to release and talk about all the cool things we've released today. Right. And just to be clear, these new things already in the dashboard. If I'm a self-serve user and I log in right now, will I see it? Yes, it is available now. API, TerraForm and Dashboard today. If you log in on the little section on the left, it says Rules, it'll expand out. You'll see all of the rule sets that we're talking about today. Yeah, I think the Page Rules to add to what Matt said, have been the powerhouse of a lot of the layer seven configuration of cloud services. So I'm really excited by this. Actually, I've logged in myself on my zone and I started playing with them because it was well overdue, a complete overhaul and revamp. So I guess lots of new things have come out in the context of rules. And one of the big things that I remember customers always asking about is how do I do a redirect and how can I be done with Page Rules? And Page Rules did have some redirect functionality and today we launched some correctly around dynamic redirects. But even before we talk about the actual product launch from today, may I ask you what is, what is the redirect at a high level so everyone can follow along? Yeah. So the easiest way to kind of think of redirects are basically Cloudflare telling your browser to go to another page or into the website when you ask for one. So you can ask, for example, dot com slash login and it could tell you actually go to apple.com slash secret slash logging because that's where it actually is for example. So, you know, a redirect is basically just telling you to go somewhere else and that page lives somewhere else essentially. Got it. And these pages can be both. Be still behind cloud four, even outside of cloud for potentially, I assume. That's correct. Yeah. So. So these could be you redirecting to the part of your own website or it could be redirecting to like a support desk you have on Zendesk or an e-commerce site on Shopify or anything. So they are sending it back to the, to the, to the actual actual browser, to the actual client and telling that client to then request something else. So it's all completely transparent to the client themselves. They know where they're going and what they're accessing and what the location is. They're kind of browsing. Got it. And I guess and I remember with Page Rules, we had a forward sort of dropdown feature where, as Matt said, only based on your URL, you could match against a specific URL and forward to some other URL. I think we also allowed choosing the status code that was permanent or temporary. What? What? What's new with the new system and with the new dynamic? Redirect rules. Yeah. The big interesting parts of redirects actually on both sides. So typically when we kind of speak about speak about redirects, a lot of the requests tend to be on the on the matching side of it, people think of it. So being able to do a country based redirect so visitors come from a certain country to your kind of example dot com, then you redirect them to example dot fr, for example, if it's from a French region and there's that kind of match, that whole myriad of functionality, that kind of expressions we support now with the ruleset engine being the driving force here so people could choose anything in that expression that it's a use that as a trigger for performing a redirect from the country. It came from things like the AS number, the IP address, the presence of a header. So there's all of that side of the house in terms of when we actually do a redirect. But the other part of it, which is which is probably more impressive from a consolidation perspective, you know, for people looking to consolidate redirects into dynamic redirect rules is the kind of action part. So once we've done a match, then what do we do? I think that's where a lot of value is kind of living in this specific product, because as it's built on the ruleset engine, there's a ton of fields. So things to do with the kind of geographical location like the city, the country, the continent, there's other fields to do with how we've passed the asset language header. So the browser sends a header to a website that says I prefer English, and then it's kind of English, I'll have English us and so forth. We can put all those fields. We can basically use those fields as like Lego bricks to create the new redirect. So it allows us to do some pretty cool things like saying if the browser requests French then redirect them to example dot com but splice in from spliced in there accept language between the kind of host and the path so then the origin can actually make that content load localized. So that's the really cool part is you no longer have to have to be prescriptive. Like if you see example dot com, send it to example dot com slash while you can now basically kind of put almost like a pattern in place and then things come in in this pattern and they go out in the other path. And so when we when we kind of design this and we spoke to customers a couple of months ago, we were looking at like HD access files from Apache. So people doing this on premise, we were consolidating maybe 203 hundred rules down to maybe one or two dynamic redirect rules. So that's the power is just taking this super prescriptive, uninspired implementations and just whacking it in one rule and let us do the do the hard work. And is this is our customers getting access to some regex type functionality to implement sort of the pattern matching here or is it something else we're exposing? There is, yeah. As always. There's kind of levels based upon the kind of subscription plan. So full blown regex is kind of enterprise in business plan. This is all at the zone level. Dynamic redirects also level. So business and enterprise plans get kind of what we call full fat, the full gambit of regex, but we do have and we have created and we want to carry on creating kind of bespoke functions for when people use regular expressions, for when you don't really need to use a regular expression. So we notice that a lot of people were using regular expression, but just checking when something matches, Does it start with ABC and does this URL start with ABC? Does this your iPad start with ABC? And we kind of saw that over and over and that's bad for the customer because you have to write these really horrible expressions. It's bad for us because you have to do a really complex evaluation when we could just do something much more streamlined and CPU friendly. So we created these two new fields to start with called starts with and ends with, and they're all in the UI today in the Rules Builder. So you can click on URI Path, click on the middle button, which is like the operations, you can say you are right path starts with log in and that's available for all plants. So we're starting to try and again look at those use cases for regex and condense them down. And then as always, just try and drive them down steps so everyone gets Access, And I might have a use for starts with an ends with the web. That's the beauty of the single engine. Right. Okay. And then I know we've had bulk redirects for quite some time. Just so I guess we're aware. What is the difference between the bulk redirect function and the new dynamic redirect function. Yeah. So, so the, the use case behind bulk redirects is basically, you know, where you want to match on. And so, you know, the URL you're looking for. You know where you want to send people to when you see that. And that's kind of a simple redirect. You know, if it's example dot com slash a send it to example co dot uk or example dot com slash b and you prescribe pretty much prescribe what you're looking for and what you're sending it to. So the use case book redirect solves is we've got a lot of them like a lot of them. Can we push all of that to the edge to, to Cloudflare like Cloudflare handle that workload. And when we launched it, we kind of built it up to I think it was up to maybe 250,000 book redirects per account. And then sometime in April or May, we increased that from quarter of a million to about 8 million redirects per account. So customers can just upload these really huge lists of from to status code basically push all that workload to the edge and then we can just take care of it. And dynamic redirect conversely is for you've got a bit more dynamism in there. You need to use regex to dynamically add the country code or the locale or the whatever it may be, and it's only for certain scenarios. That's when you would use a dynamic redirect to be fine grained, but also use one rule to maybe do what you would do 1000 with if you have to kind of spell them out. So the really nice thing now is customers can use both or either they're kind of completely included. They can just kind of play around and see which fits their use case best. Awesome. And so besides redirects, the, the other extremely complex component of Page Rules is caching strategy. And I still remember trying to debug caching strategy with edge cache TTL origin cache TTL and Alex, we now have cache rules. So, so what's, what's new with this new implementation? Why should customers start sort of jumping in and using them straight from today? Great question. I think that the answer here is probably rather similar to some of the other answers that we've gotten today that cache rules differently than Page Rules allow and put the sort of precise triggering functionality at the fingertips of users. So previously on Page Rules, customers needed to use parts like dynamic or wildcard parts of URI patterns in order to trigger rules such that when a request for that pattern would come in or sometimes like a full URL or something, that that's when the Page Rule would trigger. And oftentimes those Page Rules would be various caching settings like, Hey, when you see this URL for this image or this path come in, I want to cache that response for longer maybe than what I said at the Origin or I just like it is easier to do it at Cloudflare. It's easier to do it at an intermediary level than in the origin. But what has changed with cache rules and with a lot of the rules functions today is that we've taken sort of this page out of the last book and we've allowed for there to be additional criteria on which you can trigger these rules. So there's a bunch of request headers now outside of simple URI paths that you can use to trigger the cache rules. But the cache rules themselves are rather similar. There are edge cache TTL. So how long Cloudflare will maintain things? There's things that we can pass downstream to browsers. You can do things like indicate that we should serve stale or not. The origin error pass through. So if you want to pass a error that your origin is throwing back to the end users, you can trigger that on Cloudflare. So it's sort of the who's who and the well known rules that you've come to know and love on page rules, but with more precise triggering functionality on cache rules in the future. The future is looking bright for all of the rules functions. It's going to be easy to set and to trigger additional rules that are in flight. Now, you know, this is a beta version that we've released today. Love to get the feedback from the users about what they'd love to see next. Some caching functionality, some origin functionality redirects that maybe they don't have available today or something that they want to trigger it on, but they don't see a way. And so I think that, you know, we released this product and the suite of products with the idea of like, Hey, Page Rules has been ingrained for the last ten years of Cloudflare maybe longer and it's time is ripe right now to try and reevaluate that and to provide people with more optionality when they can trigger rules with simply pushing a button. We all know and love Workers and everything else and provide you like a good testing ground for a lot of features. But you shouldn't need to write a bunch of JavaScript to get a simple functionality that you could use with just a single push of a button. And so that's the idea. But I think as you were describing them and coming up with a few things that I think I can do now that I couldn't before. So that's exciting. Now, one more question. There's I know caching has 1000 different options. I wonder, did you have any challenges when it came to the design process of the rule builder? Because at least in the world and you know, the standard Page Rules, you just put your filter expression and then you do an action with cache, you sort of have 1000 actions you can choose. How did that play out? Yeah, I mean, as I think we're learning a lot and as we're getting sort of this initial round of feedback, there's a bunch of interactions that are bizarre and there's a bunch of interactions that maybe don't really make sense. And so there are, you know, safeguards that we're looking to put in and reevaluate as we get some additional usage. And so there are there are going to be sort of like weird interactions as we provide some new functionality to people, especially with the optionality afforded to users related to doing something super powerful like caching things for longer or shorter or whatever. And so in order to make sure that we best safeguard people from those sort of bizarre, bizarre realities that they can get themselves into, if they are trying to do something super funky, then we are putting safeguards in and reevaluating how these different inputs that people can set can be a foot gun in some situations. So we have some safeguards now and we're actively monitoring how people are using these things so that we can better, better help and get people into a good state in the future. So I guess the answer to the question is yes, but that's sort of just like the nature of the beast. And, you know, it's always nice to get some feature requests on how we can make things better for people. So, so challenge to the audience. Try and build a cache rule that doesn't make sense with the new rule. Okay, great. So there's so many new announcements. Next one, configuration rules. So, Matt, over to you again. I believe configuration rules is now a much clearer, cleaner way to disable some of the other Cloudflare features that are running on the proxy. Is that correct? And I guess what all have we built inside of configuration rules? Yeah. So configuration rules is everything else. So what you knew in Page Rules that isn't ported to origin rules that we discussed later or cache rules or dynamic redirects lives in configuration rules. So turning off Mirage or enabling rocket loader again on the same sort of edge rules engine path. So everything that you probably have gotten Page Rules an easier and better way to do it in configuration rules again, built on the edge rules engine product. Got it. So can I turn off pretty much most things now that were available in Page Rules already in configuration? Or is some gotchas still to be aware of? No, they're all there. There is some things that, as we said, have moved out so you may be looking for oh I caching or dynamic redirect still, even though they've moved into their own things. But this is the same functionality that you're used to that we've moved in and that again allows us, as Sam says, we can open up filters, we can add things to it in the future. People ask for disabling certain features or enabling certain features within Page Rules, we're able to do this a lot easier and configuration rules. The most recent one, I would say is Zara's obviously one of the products that we brought into Cloudflare, and so that was in the Page Rules is one of the latest ones that's already available in configuration rules. But it allows us to do things like that where you want to test features first rather than just on URL. Again, use cookies countries to enable features or disable them before going big bang and enabling it for all of the users. And their all. The role filtering and matching capabilities are exactly the same as the rest of the rules. Products at this point, right? That's correct. Yeah. And again, that's the reason why we're able to talk today and sort of not build this out over many months or years is we have in Cloudflare that one edge rules engine. And we're all able to sort of build our products on top of that. So we you know, it's Matthew RCA always goes like one plus one equals three. That is the one plus one equals three products. We're not going away and going, okay, let's build our own way of building configuration rules or cash rules or anything else. We can already use that engine that's there, build on top of it, build our own what we call internally phases. So each one of our products is a phase, and that is then gated to what filters we want or what actions, and allows our engineers to really iterate quick and build a lot quicker than they were able to previously, especially with Page Rules. So yeah, again, kudos to you where I don't like you to give new kudos on sort of the firewalls, firewall rules side of things, but it's a team effort. But yeah, it's made everyone sort of product lives easier and we've have other products that aren't using the rules engine but see how powerful it is and now migrating into it. So there will be more announcements and you know, and again, it gives familiarity to the customers. They know how they can use it. In one use case, they have that knowledge and are able to use that knowledge in our products really simple and easily. Awesome. Yeah, and I think that's actually sort of the hidden superpower here. Someone using the WAF will move over to cash flows and they'll immediately get it right. And then the next one up is origin rule or origin rules, which have been in API only beta for quite some time. Very quickly. So before we jump in, what, what are origin rules? And then secondly, what sort of learnings have we had from the beta and what have we added now with this announcement? Yeah. Sorry. Your rules are. And the idea behind them was when we were kind of looking at Page Rules and kind of how we would do this. This thing like two years ago. We looked at it and we said, well, these like, masses, these ones are cash related. So they're like, cash rules. These ones are redirects, so they're redirects. These ones are config. So they kind of configure and we were left with a few outliers and two of those kind of main outliers were something called resolve override and something called host had to override. Both of them are currently only kind of enterprise features. We're doing our best to try and find safe ways to stack them, but those kind of those two and what those two are essentially. So matching traffic. Send it somewhere else. Send it to a different DNS, all kind of resolve it specifically to a different address. And when you send it to that different address, we can also allow you to change the host header. And the host header is kind of what these origin applications use almost as like a like a key. So they kind of look and say, no, nothing matches or yeah, I've got something for this and then we'll kind of accept and they'll root internally. So we kind of thought, well, those two things together constitute like an origin rule. So basically determine, determine where to send this traffic to based upon, again, the same the same fields. We've spoken about this last half hour. So we kind of put it together as that. And the kind of story behind origin rules is if you need to change where traffic goes to and you to change how it looks when it gets there, then just implement an origin rule. Right. And that is just to make sure everyone understands is different from a redirect rule because the redirect is sending the browser somewhere else while the origin rule is sort of invisible to some extent. browser, right? Yeah. Yeah. So there's kind of three here, isn't that there's the redirect which tells the browser go somewhere else, basically sent a request to somewhere else, explicitly does URL rewrite, which just basically changes the URI path, but it sends it to the same origin, the same server. So someone could request log in, we rewrite that and we send it to slash secret, but it still goes to the same end point, the same, the same server, the same VM. And then there's like you say, host header resolve override, which could be send it completely somewhere different on the Internet. We proxy it and it's somewhere completely on the Internet differently and again, change the host header. So when it arrives there, the host had to kind of match you something it's expecting, whether it's the S3 bucket or an Azure website or whatever it may be. Have there been have you noticed some sort of patterns on customers using these features, like some common use cases that tend to be frequent? Yeah. The main one is where people kind of want to back off specific generally file libraries like image libraries to like an S3 bucket on kind of one of the major vendors. So websites examples dot com again they could have example dot com slash images and that could be where all their assets are, their icons are their kind of marketing collateral is and they don't want to host all of that on their website builders. So what they do is they, they'll have like an S3 bucket and then what they can do is use an origin rule that says if the traffic matches URI path starts with Images then override the results that you resolve. the DNS to go to the S3 provider and then once the traffic gets the S3 provider, I want you to override make sure the key the host header actually matches the name of the bucket. So when that request slash images slash logo jpeg kind of gets there, the S3 provider goes oh, the host header is example, dot com, dot S3 dot, whatever. And it kind of knows how to get it to that bucket and back again, basically. So backing off the kind of static assets to a cloud provider, to an S3 provider is, is super common, particularly in Page Rules and we'll probably see similar to it to origin rules. Yeah. And I'm trying to use cases I've seen and one that comes to mind is when you type in your URL rewriting where you have maybe your marketing website under your main domain, but then you want your blog under slash blog. But that blog is maybe a WordPress site hosted on a different server, and that's super useful to achieve that with a couple of clicks now, right? Yeah. So you've got you've had all these announcements today, I think I have four or five different blog posts that I need to catch up on. What is what is next. I guess with the future of all these rules, what can we expect in the near future? So yeah, I'll say I'll take that. The next phase of this, as it were, is switching gear from ship, ship, ship. So we've been focused for a while now on getting these products out the door. You know, today, as you say, we've just launched four brand new products in one day out to beta. We're going to listen for the feedback. We've talked to the customers, kind of see what's missing or kind of what the problems are, why it doesn't quite fit. Once we've addressed those, then we're going to kind of switch gears and we're going to think about we've got millions and millions of Page Rules out there right now that have been accrued over the last ten years. We kind of want to end of life Page Rules, which is why we're doing this. So how do we kind of shift that humongous volume of Page Rules into this new suite of products? And how do we make that as frictionless as possible? You know, and that's not an easy task to do, as you kind of know, with the migration. On a much smaller scale. So it's not going to be easy. But the nice thing is we've been building in Cloudflare, we've been building a lot of cool things for customers, which we think we can possibly leverage ourselves and we'll kind of go into that more when we kind of talk about it. But what we want to do is just say, Hey, we don't we don't want to say, Hey, you need to migrate all this stuff yourself by whatever the date may be, because that's sucky experience. Instead, we'd rather find a way to say, Click here and we'll do it for you and we'll test it for you. So that's kind of what we're going to start thinking about now, is how do we kind of get this to be the only way of doing? The host had to override all the only way of doing cash, you know? So that's the next mountain we have to climb. Now we can see before really. I'm really tempted, but I won't put you in the spot and say, Well, when will customers say goodbye to Page Rules migrations? As you said, we're going through one with the web and it's definitely we're sort of trialing all different patterns. Click here and it's done, or move your rules over by certain date and it's definitely hard. Our topic. Okay. So first of all, thank you, everyone listening and joining us today. Of course. Please do go and check out our blog. I think today probably we've done a new record. There were 12 blog posts in a single day. All of them are really, really exciting announcements. There's still a lot more to come for Birthday Week. If and when you do have any questions, of course, feel free to open a support ticket. Or if you're an enterprise customer, reach out to your customer success manager and Alex, Sam and Matthew will happily jump on a call, including myself, to show and explain and see what you can build with Page Rules and beyond that. Thank you very much. Have a good evening and see you soon.