Latest from Product and Engineering

Hi, I'm Jen Taylor, Chief Product Officer at Cloudflare and I am thrilled that you're all joining us for another episode of our phenomenal Latest from Product and Engineering. I'm here with my partner Akram. Hi, I'm Usman Muzaffar, Cloudflare's Head of Engineering. It's been a couple of weeks since we've done one of these. So I'm actually, I was happy to see it back on our Friday calendar and some absolutely critical key teammates joining us today. I think we'll just go in the order that they are on my Zoom screen. Angie, Constantin and Nick say hi in order. Angie. Hi everyone. I'm one of the product directors and I'm here to talk about billing. Great way to end the week, Friday. Hi everybody. My name is Constantin. I manage the billing team here at Cloudflare. Hello, how's it going? I'm Nick. I'm the tech lead for the billing team. Awesome. So I always love having this team on because I so appreciate the work that this team does to really help make that interface between the products we build and getting them into the hands of our customers more seamless. And now, so Angie, I know you guys have been working on a pretty significant improvements to our entitlement system. Can you just like back it up? Like what is an entitlement? Like what does it do? And like why does it matter? Yeah. So in the world of SaaS, when a user clicks a button to buy something, it's not like you're actually receiving anything in the mail. You're basically getting a service. And when you purchase a subscription, there's a bunch of different indicators that define whether or not you have certain features, how much of a certain feature you have, and that's all something called an entitlement. So basically think of a subscription as a package of all these different features that roll up to define like how that product is going to go ahead and work. Awesome. Now, obviously you guys are building an entitlement system. We have our whole suite of products. How does that interface work? How does that relationship between entitlements and product work? Yeah. So we have something called a catalog, which basically defines, here are all the different groupings of the products that we have. And depending on who the customer is, they could be getting a different type of subscription. So we have something, pay those customers, other people that are buying their services on a month-to-month basis. It's at will, so they can go ahead and terminate it in a single time. And then we have contracted customers that have a pre -negotiated rate and the pricing can vary. And the terms of the level service is going to be much higher than the pay-as -you-go because they're getting a higher level of service. So all of those things combined are things that are built into that catalog. And as we're shipping new features and products on a pretty high cadence, the different engineering teams need to be able to plug into that catalog so that they can go ahead and launch it and sell it accordingly based on the different customers that we have. Angie, you've been here for quite a while and you've seen this system evolve. What were some of the early big steps when Cloudflare went from a very simple plan to actually having some of these more complicated projects and how they were built? What were some of the inflection points back in the day? Yeah, so when I joined, which is five years ago, there was no such thing as entitlement. Everything was just hard-coded into the basic plans that we have today. And as we evolved over time, we realized that we didn't have enough flexibility for our customers. One of the analogies that I always use is if you bought a pro plan, you had automatically 20 page rules. But as our users were using the page rules, they ran into a wall. They couldn't buy more. So we had to introduce the concept of an add-on, which means that we need to have a way that we can override the hard-coded values that we had. And so that kind of introduced the concept of entitlement. And that was quite some time ago. And so we have different entitlements. There are some that are just an easy yes, no Boolean statement. And then there's others that need to identify what is the next amount of origins or page rules or some quantifiable number that users can go ahead and configure. So it's evolved quite a bit over time. And we've kind of hit this wall in terms of where we're able to fit. And obviously, as any person in engineering realizes, there's a lot of tech debt that comes into play. And we realized we had to do a massive overhaul. And we were finally able to get to that this past quarter. That's fantastic. So Konstantin and Nick, so product managers are dreaming up all kinds of new products. They're dreaming up new ways to build them, new ways to limit them. Like, oh, you should be able to pay for this with a coupon, and you should be able to add 10 more of this. So Nick, as you start hearing all this stuff, and you were there from the beginning and started to actually think through how we model this, what did the beginnings of this system look like? Just from ground zero, when was the first time you said, like Angie alluded to, this used to literally be baked into the product. Like, okay, now this has to be factored out into its own system. And it needs to be some way of literally saying you are allowed to do this. This product is giving you this capability. Talk a little bit about the early days of entitlements. Like what was Hello World of the entitlement department? Angie gave it a lot of nice color. In the world before our entitlement system was very, very sad. Because when it came to- Transparency, folks. That's what it's all about. Yeah. It's hard work getting new features out. Because when it came down to it, billing is a complex domain. And there was no abstraction in the product for that. So when it came to answering questions that other parts of the business had, like how much of this feature does this customer have? You needed to grab an engineer and just start looking at code in order to answer that. Because it was encoded in the product. There was no interface between them that was keeping it abstracted. So entitlements was the introduction of that abstraction. So let's talk about the abstraction. So abstraction is all about hiding details. So what is the detail you're trying to hide? What is the surface area you're trying to expose to product managers and engineering managers? What is the world you wanted to present? And what is the world you wanted to hide that was below the abstraction? Great question. It comes down to keeping product engineers from asking the wrong questions. Whenever a product engineer needs to consider, does the customer pay the bill this month? Which plan are they on? Is this a one-year term? Is this a month-long term? These aren't the questions that product engineers should be asking. You got to boil down to its finest and most atomic question, which is this feature. How much of it do they have? Do they have it at all? And then that question gets answered by the entitlement service. And then on the other side of this abstraction is billing doing its work, charging customers. This customer didn't pay their bill. And unfortunately, we have to revoke service or something like that. And we get to sit on the other side and control all these knobs and switches in order to keep our billing service providing service to customers and making sure that any changes in that are abstracted from the product and the product can remain blissfully unaware of how all this sausage is put together. That's fantastic. Konstantin, I wanted to zoom up a little bit and just sort of be architecturally. So Nick is describing a service, something that will provide a surface area. You interact with it. It's got an API. It has some state. But how does that then connect as the person who is in charge of the billing team and owns the relationships with all the other engineering teams? What happens next? So this bit gets baked in there. How does it actually change what the product is allowed to do? And how is it consulted? And how did you think of your sort of human interface with the engineering teams as you're building something like this? Yeah, it's a good question. This allows other product teams to think of how this specific offering, this specific product can be offered to our customers. How much, as Nick alluded, how much of it can they have baked into something we call also a subscription, which we haven't talked about. But that is the higher level of abstraction that kind of hides the entitlement under it. It allows engineering teams to think of how they can bill or product and engineering teams to think of how they can charge and how we can bill for that specific service or that offering the platform has to have or give to the customers. And do so in a way, as Angie alluded, that is flexible. We don't have to hard code anything. There doesn't have to be a ceiling to the offering, so to speak. But more importantly, also do so in an incident prone way so we don't cause problems or outages for customers. We can be scalable and service can be turned on or turned off when needed in a fast, reliable manner. Entitlements as an API in the back end of things and just as an entity should be and is a critical system, but should be an afterthought. Should be the quietest system in the company that produces a lot of value, but doesn't cause any damage, so to speak. That's basically the idea that we had in our mind when we did a bunch of work in the last two quarters, spearheaded by Nick and the team, to make this a reliable system and a system that is very flexible to allow products to be creative with how we offer our products and how we can charge for those products as well. The next question is that, to engineer the engineer, like Nick said, there's an API, there's a data model that I'm going to interact with. What does the entitlement system look like to the rest of the Cloudflare customer -facing team? Where else does this thing show up? Do we have custom interfaces that let our internal employees, our customer -facing teams interact with entitlements? Can you turn on beta features for customers? Yeah, we do have what's called an entitlements UI, which is a UI GUI system that allows you to see what kind of entitlements a customer has turned on or turned off. You can set individual entitlements that way, but as we've been thinking through this and making things be cleaner, more scalable, and less sad, we also have something called the subscriptions UI, which allows us to go a level higher and product teams can think of customers, what subscriptions do they have? Under those subscriptions, they can actually drill down and see, well, if I really want to know what entitlements they have. Subscriptions are kind of like a container. That's the customer-facing thing. A subscription is basically a catalog of entitlements. Is that kind of how it works? You got it. That's exactly how it works. Subscription is like a bucket or a catalog of entitlements underneath it all. Yeah. Got it. Got it. I think one of the things that people who watch this kind of want to think, what does this look like on the inside? Here you're hearing, it's a product. It's an internal product. It's got product managers, engineering managers, tech leads, and a growing audience inside Cloudflare that needs to interact with the system and manage it. It has all the needs. It's just what happens. It's not something that our customers interact with directly, but it's how we are able to provide the customer-facing. I know Jen has a million other questions for Angie about how all this works, but I want to nerd out with Nick here for one more second. The interesting thing here is, and Angie touched on this, is at the beginning, it was like, hey, is it on or off? We built a new feature. Do you have it or not? She said the word that comes to mind, which is Boolean. It's just literally down to a one or a zero, and it's stored in there. But then you introduce the very first motivating example she was talking about, like, okay, we've got the thing called page rules. I'm going to give you 20 of them. How do I know when you've used up your 20th one and now you've hit the limit? And then there's way more complicated products. So Nick, when you started to think about how you model this, what did we reach for? How did the model of entitlements evolve over time? It did start out as a simple integer store, like pretty much. It was a KB store. It was like, this customer has page rules set to 25, and then we wanted to encode more things that you can see on a pricing page. So like Cloudflare's pricing page right now, you're going to see that nice little grid, features off to the left, plans off to the top, and in the middle is like what these are allocated to. And basically, entitlements is just an encoding of that per customer. And so if you look at the page, it probably has this right now, you'll see numbers and you'll see checkboxes. So in order to encode that, it needs to be polymorphic. It needs to be able to store Booleans and max counts for certain things. But we also have to do this in a way that is more strongly typed, like page rules set to a Boolean doesn't make sense. So we have a catalog of all these features, know exactly what type there is, so we can reject any bad input, which doesn't happen very often, but it's nice to keep it consistent. In between max counts and Booleans, that's served most of our use cases so far. We have a few other exotic types, like enum number ranges, where you can have like a set of numbers and then like a range dictionary, which like min and max, but those have more limited use cases. And most are just served between the bools and the max counts. And at the end of the day, from the product engineer's point of view, it's still just allowed or not, let me know. Well, when it comes to entitlements, we are the rule book, pretty much like we are, we tell the services what the customer has, and then the products are responsible for enforcing that as well. So when it comes to enforcing those limits, we also get to kind of put that complexity where it belongs in the product itself. So entitlements is this really nice complexity sorting machine where everything is kind of tucked in its right spot and we can scale the company without having to worry about all these weird interactions between our systems. And we can unlock creativity when it comes to thinking through how do we want to, with such a broad portfolio of things. And back to you, Jen, here, like your team keeps dreaming up new ways of building things and then new ways of billing on them. So, you know, what are some of the interesting things that came up there? I guess, yeah, Angie, please go ahead. Sure, yeah. So back again, back many, many, many moons ago, the notion of billing was very just simple. We wanted to offer something called a prepaid subscription. So that means that for on a reoccurring basis, you're going to pay ahead of time for a full month. So that's a pretty simple idea. But then we got to the idea of usage fees. We had launched things like Argo and load balancing and stream and workers, and the number of use cases just kept like exploding. And there was no way to anticipate like how much usage a customer would actually have. So we had to adopt the idea of something called billing in a rear. So that means that we're going to charge you for last month's usage, the actual amount that was used. And so that was like a completely different concept that we had to consider. And then the units of measure that each of the different products had were different. And then the tiers could be different. So some of the products have a free component, some of them don't. And then the pricing also differs. So those are all just like different variables that we had to quickly shift and adopt towards because not all products work the same. Some are in a request basis. Some of them are on minutes for stream. So they're all vastly, vastly different. And then we also have things like add-ons. So on top of the base plan or subscription, there's additional features that you could buy. So that complicated the logic. So there could be a prepaid component with an add-on and billing in a rear is all for a single product. And load balancing is probably the best example that has all that complexity. I mentioned in the top of the hour that we have the pay-as-you-go customers as well as the contractor customers. The billing and the pricing is all different too. There could be some kind of built-in cap or maybe there's an overage. It just all vastly differs and it complicates billing. Outside looks very easy. It's just a factor you just multiply by X and then you're done. But there's all these business logic rules that we have to incorporate when it comes to billing. That's what you're saying, that caused a lot of pain across the product and engineering organization that as we grew and had to adopt more homegrown solutions and became creative, our code base became more difficult to manage. And therefore we couldn't scale as easily. So that's another reason why we've decided to clean things up in this area. Yeah. And why investing in billing was a big thing. And especially over the course of this, you know, coffee is getting bigger and bigger, signing more and more customers, like the volume, the amount of success here was still growing straight up. And so part of it is an incredible independent of all of this is normal technical. Yeah. It's funny, you know, we think of Cloudflare as such an innovative company and kind of figuring out how to get the product, innovate on the product and then figuring out how to get the product in the hands of customers quickly is also a key part of it. I think sometimes from the outside, people underestimate the amount of innovation and investment that needs to go into these mission critical systems. Because as you were highlighting Konstantin, like there definitely were, there were moments where sort of building on top of the legacy system and doing everything that was bespoke actually was slowing us down. It was making it as actually harder for us to get product into market. And as we do this with anything, entitlements including, we're always asking ourselves, we need to build systems that will be five to 10 years resilient down the line because customer basis is going to continue growing. And I remember conversations with both of you guys about how do we make things self-serve, right? Because the other thing that I hated seeing was billing being the dependency on, there's enough complex dependencies already, right? And we've got, you know, we all remember the careful reviews where we're going through status, like, oh, we're all done, just waiting for that. And I remember talking about, we have got to make it so the team can do this themselves. So you're not in the crosshairs of shipping new features for a second. What did we do there? How did we make that a self-service thing? I think that's a great, great point or segue for Nick to brag about it a bit and how this is more self-serve than it was ever before. And people can just go and do what they need to do. Yeah. The entitlement system is obviously there for the benefit of the engineering teams that are there to design what their products look like from a feature perspective. So when it comes to it, exchange of needing to understand what their features were and put it all together for them was like this huge friction point in getting products out of the gate. So what we started with was the ability to kind of define these features. And it's a really simple workflow now where engineers from the other parts of the organization can just come by, add some code to this one repo. Once it gets merged, it's out the door, it deploys automatically. So you have the features, but then you have the entitlements that are already established for customers who have perhaps already bought this product. And this brings us to a process that was until recently very much the same friction, which is called a backfill, where you have all these customers who have bought the probe zone. And our product managers want to set up features for the people in the probe zone to have. So the idea of mass rewriting their entitlements in order to actually give them this feature took a billing engineer staring at some SQL, trying to figure out how to do this. A few more engineers reviewing it to make sure it wasn't going to blow things up and then hitting the release days. So you're looking at like a week to actually turn this around. So we expanded on the same concept and we're able to establish a configuration for all these products that we have. And now engineers can come by and simply redefine what the pros subscription looks like from an entitlement perspective. And then same deal. Once it gets merged, automatically deployed, we hit go and it will automatically give it to everybody. And this is a forward looking project as well to make sure we can improve the reliability over time. So it was a really successful project. And now engineers can turn a backfill in and out inside of a day. They come to our chat channel. They're like, I need to do this. I'm like documentation in, out, merge, go. We're at the dream for the entitlement. And what I love about this story is you guys still have control. There's just enough of oversight to make sure that it's still locked in, but the tedium and the toil has been completely ripped out. And also the knowledge is expanded. Everybody else gets now how the system works. And that was so critical. When I think about some of the big changes that happened in the pandemic used to be, there was a billing pod. You could just sort of saunter over to Nick and Constantine and hang out with you. And as we became more remote, became more distributed, streamlining this was so critical for our growth. Yeah. Now we're trying to find other reasons that people come and hang out with us. We were just joking about what topics are we going to talk about? I was like, we should talk about tax and like, no, we're going to put you out of your sleep. I was like, no way. No way, man. These are fascinating problems. I'm not, I take, I accept that. I'm going to do Nick and Constantine talk about tax for two hours. Yeah. I mean, here's the interesting thing for me too, about billing, like probably more than any other portion of our business. You guys are really at the crosshairs of a bunch of outside influences and requirements and forces that you don't necessarily have a huge amount of visibility on. You don't necessarily have a huge amount of control, but you need to move forward with pretty quickly. Like, you know, Angie, it sounds like the landscape is changing a bit in India and it means it's going to present new challenges for the team. What's going on there? Yeah. So we just got word that for reoccurring payments, which is the majority of our subscription business for our pay-as-you-go customers, they're changing how reoccurring payments are going to work in India. It's great because what's happening is for these reoccurring payments, you need the customer needs to basically verify that these transactions are great. You know, it's good for security. It's a great step. These are regular changes by the government of India. Yes. This is a regulatory change. This applies to everybody operating in India. Right. Correct. And so, you know, now we have to figure out how is this going to work? You know, we need to make sure that we're not disrupting service for our customers because we're not able to collect these payments. We need to be able to figure out how we can invoice these customers, how we can try to charge these customers and how to collect money from these customers. So, you know, it's not just billing as a black box. There's multiple vendors that we need to work with. There's different levels of applications and we need to coordinate all this work. And, you know, when regulatory changes come like this, it just adds another level of complexity on top of all the other products and features that we're trying to ship. So it definitely makes it a very exciting and fast changing environment for us. So, you know, if you look at these kinds of changes and you think about it, OK, so now you have to get the customer involved in what was just sort of a seamless sort of reoccurring process. Like what where do you start? Like how do you start on a problem like that? Good question. Yeah, this is where we're a little bit of in a pickle because we're trying to figure that out like right now, it's this is a real problem that we're dealing with. And so, you know, we're trying to partner with the different vendors. We're trying to really get solid answers in terms of what are these different questions that we have, because it's it's definitely gets complicated when especially if you're not in that country, like you're not familiar with the details of what these laws mean. That's when we need to really rely on our partners and figure out how we can collaborate and work together. Yeah, we're looking at it immediately right now as we speak. We have a meeting scheduled for next week to jump in. And this is what billing, like you guys mentioned, billing is about reacting fast, but hopefully also responding intelligently to problems. And we're going to put together a nice product requirements document, thanks to Angie and the rest of the product team, and then turn around the solution because the impact is quite high due to this change. I want to go back to the thing Nick said earlier in our session here, which is of abstraction, of having ways of hiding details in some places and exposing flexibility in others. So like, maybe question for you, Konstantin, you look back at where and how we've been able to do this, how have we been able to keep up with, it's not just India, the changing requirements are all over the place, tax requirements, new product requirements. What are the attributes of the Cloudflare billing and entitlement architecture that let us keep up with this rate of change? Like, how is it that we didn't, where were we flexible? And almost like, what advice would you give a startup founder who is putting their data in their billing system? Like, where should the flexibility be? And where can you take the shortcuts? That's a good question. Well, first thing I would advise to anybody who's starting up is to make sure that they put a nice robust spec together, specifically technically, and try to not cut as many corners as they possibly can. And afloat as much logic that we, in the beginning stages of billing journey, we took it upon ourselves to create systems that at times fulfill functions or replicate functions of our vendor partners. And that was a bit painful at times, more than we should have done, right? So that goes to thinking a bit more about the way we solve problems. But then something else that comes to mind is scalability and redundancy. Something we had in the beginning that Nick put together, orders worker, which is a system that is able to turn around orders really quickly, resubmit them if something goes wrong, and essentially make sure that the customer's orders go through, is a system that we put together fast, and then evolved over time, we built on top of it as we scaled. Thinking of things like subscriptions and making them moldable, if you want to think of it that way. And subscriptions as an envelope on top of entitlements, so that you have multiple levels of modeling. Thinking how that fits into the picture and how it interacts with entitlements, and then thinking as well outside of that, how does their product catalog look like? Because it grows over time, and products that come into play may be different in the way that they're structured. And how also all of that together gets released into our production environment, and is released in a way that is also error prone, can be done at any point, without any pain. I would say that those are the problems we've been trying to solve and build on top as we went. I don't know, Nick, do you want to add something to this? Yeah, I suppose I've always envisioned the world, the billing world, inside of a company as two worlds, where it's just like you have enablement, which is our entitlements API service, and then billing, which is our subscriptions API service. And then they have a nice line drawn between them. For the entitlement side and enablement, I think getting some custom solution probably isn't that bad of an idea. We have a custom solution, and it fits our needs really well. But on the billing side, I don't think there's... It's not a good thing to do not invented adhere syndrome. There's companies out there who do this really good. And when you get down the rabbit hole, you're going to find something that you need to do to fulfill what a billing system properly needs to do. And you're going to be like, oh, we're a billing company now, because we're implementing all this billing logic. So let some vendors do that and build some good relationships with them. Yeah, that's great. We want to have a good relationship with them too. I always like to say our mission is to help build a better Internet, not necessarily to help build a better billing system. Right. Wow. We're at time. I can't believe we had so many things to say. I told you we could easily go two hours on tax if someone asked us. We didn't even get to tax. We ran off the tax. It's going to be awesome. But Angie, Konstantin and Nick, thank you so much for joining us. So much. And thanks for all the amazing work. You have unlocked so much productivity at Cloudflare. And I'm so pleased with all the progress. Thanks everyone. We'll see you next week. Take care. Bye. See you. Bye. Bye.