Latest from Product and Engineering

All right, hello and from the West Coast, this is the latest from product and engineering.

I'm Usman Muzaffar, Cloudflare's head of engineering. Jen Taylor is out today, but I'm very pleased to have some of our other great teammates with us.

Angie, Martijn, why don't you guys introduce yourself? Hi everyone, I'm Angie Kim, one of the product managers here and working on billing.

Hi everyone, my name is Martijn Gonlag and I work together with Angie on our product for billing.

And as you can guess, today's theme is billing, which is a very deep topic.

And if you are not used to thinking about how complex SaaS billing can be, especially for an infrastructure product, especially for a company that has a wide number of products and we build them in different ways and we try to make sure that we have a lot of different ways of making it easy for customers to pay for stuff and our product managers to have creative ways of making it possible to get value from customers.

So I'm really thrilled to spend the next 30 minutes talking about billing.

So why don't we just start, Angie, give me some orientation.

What does the billing team even own? What do they do at Cloudflare? Yeah, so when people come to me, they're like, okay, so I've got a customer that wants to pay their bill.

Like, can I just give you their money? And we're like, well, it doesn't quite work that way.

So the best way I can explain it is, and I'm sure a lot of people can relate, especially in the times of COVID where everybody's sheltering in place.

So imagine going out and doing your own online shopping.

So you have to go into a website, enter in your payment information that gets saved somewhere magically in the ether.

And then you start picking what items you want.

You go ahead and make your selections. You click a button and then you get charged and then somehow arrives on your doorstep.

So it's not too dissimilar from what you face from retail shopping, but it's a little bit different.

So there is a whole collection of user information.

So we have to create a token that anonymizes the information that we have about the payment.

So we'll collect things like your name, maybe the location from which you're paying for it, as well as your payment information.

So that all gets stored. And then there is the actual process of making the order.

So from there, we go ahead and put a bunch of things into a transaction and then it gets sent off and we go ahead and issue payments as well as invoicing.

So those are like the very high level pieces, but there's a lot more system level things.

So there's the whole idea of a subscription. So that's something we go ahead and create as a package.

And from there, there's a whole bunch of features that get entailed with it.

And then there's the reoccurring payment that happens.

So we have to collect money from customers on a reoccurring basis.

And then what's the worst case scenario? We are unable to collect payments.

And then we did try a couple of times before we decide whether or not we're going to continue service with you.

So that's kind of like at a high level what we do.

And already my head is swimming, like we've got so many different components just listed there in the first few sentences of explaining what the billing system does.

So Martin, what has your involvement been in all this and how have you been working with Cloudflare billing in the last year or so?

Sure. So I'm primarily focused on our pay-go business, which means our pay-as-you-go business.

So customers that come in and want to pay with a credit card or PayPal account.

And I'm working on a variety of different projects, which can be actual customer facing changes, such as allowing them to add a backup payment method.

But it could also be internal customers, such as supporting our finance team to enable taxation in certain countries.

So what's a backup payment method? What does that even mean, for example?

Sure. So I mean, by default, people will use a payment method. But in a SaaS world where you have a recurring subscription and you have websites tied to that subscription that you want to make sure stay online, if something goes wrong with your payment method, we don't want your website to go down.

And so what we did was offer customers the ability to flag an additional payment method as a backup.

So in the event that we detect something went wrong.

Like more than one credit card, for example. Correct. Yeah, more than one credit card, more than one PayPal account.

Right. Yeah, because, you know, so exactly.

And I want to get to this because I think this is one of the unique challenges of Cloudflare's billing system is that we have both an enterprise, a classic enterprise business where it is contracted business, but we also have a pay-as-you-go business, which just like Angie, just like you were saying, is as straightforward as you visiting an e -commerce site.

Just type in your name, your email address, slide your credit card, turn on some features.

But, you know, unlike worst case, if Amazon's, you know, if I'm in my favorite e-shopping site, whatever, you know, if my credit card's expired, at checkout, it'll just tell me, hey, this is expired, you know, please enter your new number.

But at Cloudflare, people often will set up websites.

I remember I was a Cloudflare customer for years.

I probably logged into the dashboard once, you know, set up a lot of infrastructure and just sort of let it forget.

And I am counting on Cloudflare staying up.

So, you know, things like backup payment methods are critical because we absolutely want to do everything possible before we even consider, okay, maybe this is really, really, fully, truly out of business.

And like there's the customers no longer interested in service.

So, Angie, I think first question to you is like, if you zoom all the way out, what are the differences between a self-service billing platform and an enterprise billing platform?

And like, what are the big challenges you had to think about as we were like designing a platform that's going to work for both?

Sure. So, with our pay-as-you-go business, we built an infrastructure that was pretty seamless in the fact that it was ubiquitous.

Regardless of who you were, the pricing was always going to be the same.

The terms are going to be the same. The features are going to be the same. And there were different levels that you could purchase, but it was pretty much the same for everybody.

With our contract customer zone, there's a little bit more customization that needs to go into it.

And there are premium different levels of services and features that are available to those enterprise customers as well.

So, trying to package something as well as being able to figure out how do we piece all these different features together to a solution that makes the most sense for these customers is very different.

So, there's a lot more hand-holding, a lot more human interaction involved.

So, it's a little bit different. And also, the pricing could be different as well.

And the terms could differ. So, with the pay -as-you-go business, it just runs such a machine.

We've got all the services and all the features just running on its own.

With the enterprise customer zone, there's a lot more, again, hand-holding just because there's contracts that are involved.

And so, humans need to go in and actually fill something out and sign it, whereas our pay-as-you-go business is just a click of a button.

So, it's just much more manual today.

And to admit, and everybody else I think in the organization would admit to this as well, we haven't really done as much diligence when it comes to automating that process.

So, what we're focusing on at least for next year is trying to figure out how do we bring the two experiences together so that customers, especially start from our self -serve business graduate into our enterprise level, but they get the same level of service and expectations that they have from when they were a free customer, essentially.

Yeah. And it's actually two classic engineering design centers, right?

One is simplicity for scale.

So, you want to make it as simple as possible. So, you can just crank the numbers up, 27 million zones altogether, I don't know, hundreds of thousands of zones that are self-service.

And then, like you said, that enterprise business, it's way more configurable, way more number of knobs and switches and authorizations that have to happen.

Martin, have we had to work with that? Can you think of an example of some of the automations that we've done recently to help streamline some of that?

Some of that stuff that used to be more manual and now the billing platform and the product itself makes it easier for Cloudflare employees to sell things and for self-service people to configure things?

Yeah. I mean, the billing team has done a tremendous job in taking a platform, a billing of a platform so that the engineering teams can come to us and pretty much self-serve grade subscriptions nowadays.

It used to be a very manual process, would take several weeks, months, depending on what you actually wanted to do with the subscription, what type of billing you wanted to do, what type of features you wanted in it.

And all of that is now pretty self-serve to where they can go into what we call a catalog and they make a pull request and then there's additional few components in the background that do come looking into it because we actually have to collect money on subscriptions.

But yeah, it's pretty self-serve nowadays. So, that's amazing.

So, that means if engineering teams with the same ease and the same kind of tools that they're used to, like using Git and creating pull requests, can actually provision billing sub-platforms for their products.

And now it can be something that will flow into the rest of the infrastructure so that customers can sign up for products and pay for them and get notifications of them, set up backup payment methods and all that great stuff.

Yeah. And it also opens it up for some of the things Angie was hinting at for what we're trying to do for the next year, allowing more of that automation.

This may segue into another interesting conversation, but we have not just subscriptions, but subscriptions encompass entitlements.

And one subscription may have a dozen entitlements associated with it.

And it used to be historically that a human would have to go in for an enterprise customer and would have to set those entitlements.

And so, those are the kinds of things that we want to streamline by making those features more self-serve.

Yeah. Let's talk about the word entitlements. Angie, what's an entitlement?

Why does it come up so often when we talk about billing? Yeah. So, entitlement essentially is just an individual feature that can be part of your product.

So, again, depending on the level of service you have, we're going to increase the amount of quantity that we've got for different customers on the different tiers.

So, this is a way that we can abstract a lot of the billing information in terms of like how much you get charged, how often, all those rules and abstract it out so that the different engineering teams can just say, do I need to give this feature to this specific entity?

Yes or no? It's just a simple yes or no question, or it could be a max count value.

So, that way they can figure out, you know, based on the contract that they have with us, whether they're paying a month-to-month credit card or through a check or through a contract, they just need to know what feature am I supposed to give to this customer and how much of that feature am I supposed to give?

So, an entitlement, if I go into engineering speak for a second, an entitlement might be as simple as a boolean switch that says you are authorized to use feature X.

Correct. Or it might be you get nine of these or 24 of those, you know, load balancer pools or rules of certain kinds or something like that.

So, and the entitlement is the engineering infrastructure which is guarding the control of the availability of this feature.

Exactly. And that explains the next question which I had, because if I look at some of the recent projects, this is the whole title of our session, after all, is the latest from product and engineering.

And, you know, I was looking over some of the recent billing projects, billing support for interns, billing support for Project Galileo, which is a free service that we provide that we're very proud to provide for, you know, for organizations that, you know, do good and charitable work.

And, you know, also billing support for things that help with things like Cloudflare helping with election services.

Why is billing involved?

The whole point is these were supposed to be granted services. Why is your team engaged on these projects, Angie?

Yeah. So, there's a lot of like different initiatives, like you had mentioned, where we want to provide, you know, just services to everybody.

Yeah. Whether or not you can pay, not pay, just overall social good reasons.

And the reason why billing is getting involved is because we can't just enable services and not justify why potentially they're not paying for us, right?

So, we have a bunch of KPI and there's a whole, we have a whole BI team that's just focused on making sure that we're counting things correctly.

And from an accounting perspective, we need to make sure that we're catching all the revenue as much as possible.

So, we need to justify why certain customers are getting certain services for free and not paying for us.

And this is also a way that we can light down, lock down, put more audit controls as well, so that as we're going through and provisioning these types of requests, we can determine, oh, the interns are going to get Cloudflare for free for a year.

So, when that year ends, that we can go ahead and turn it off. But if we didn't have all these mechanisms and flags in place, we wouldn't, we couldn't justify why, you know, who these customers even were.

Yeah, exactly. So, like this, all those giant list of use cases that Martijn was referring to, like all that, like keeping accounting of that, regardless of how many actual dollars or cents are next to the number is all still part of billing.

So, it's very interesting, like billing owns a really complex cross -section of all the way to UI facing pay, like literally the page where you type in your credit card number, the backend systems that then tell the product whether something is on or off, and then also data and analysis systems.

And so, one of the things, Martijn, you alluded to is like different products want to be charged in different ways.

One of the abbreviations that came up a lot when Angie and I were just talking, we both joined Cloudflare in 2016, that we certainly became a word that we heard a lot was UBB, stands for usage -based billing.

So, let's talk a little bit about that, Martijn. What do you mean by usage-based billing?

What is the usage that we might be talking about? And why is that a completely different animal than, say, a plan-based billing?

Yeah. So, I mean, the way that we used to bill for most of our services was a flat fee, and there's certain things included with that.

But as we expanded our product base, a new feature became available that don't get included with plans.

And we need a way to charge for those.

And in some cases, it makes more sense to actually charge for the usage for those features.

So, in those cases, we need a way to count the requests or the bandwidth that was used, for example, or the number of worker requests for a worker's product.

And then we need a way to actually bill a certain amount for those requests and influence that to the customer.

And so, that whole process flows through, and that's what we would call internally our usage-based billing pipeline.

Yeah. I was struggling for this. There's analogies to this in any other kind of telecom, right?

Where it's like some things you sign up for a plan, but then you also might get charged for a certain amount of minutes or for very expensive kind of calls.

And part of this is manifest in a term that shows up in billing circles a lot, billing in arrears.

Angie, what does billing in arrears mean? And why does that suddenly make the engineering and product behind this become more complicated?

So, one of the interesting things about the way we bill is we bill on an anniversary date.

We don't bill everybody the same day of the month.

So, the reason why it complicates things is that my anniversary date could be, today is what, the 30th?

But yours could be the 15th. We should be talking about Halloween also.

That's why, in case, for the audience wondering what's going on with the black backgrounds.

Exactly. So, I've got the gross candy background. That's the name of the pipeline.

But yes, it is the 30th. So, if a new customer signed up today, they would get billed on the 30th.

Is that right? Correct. Yes. And then I could be billed on the 15th and Martine could be billed on the 1st.

So, everybody has a different billing date.

And when we talk about billing in arrears, you're paying for the previous period.

So, my billing in arrears period is going to be very different from yours.

It's going to be very different from Martine's. And so, trying to keep track of that, it becomes a little more complicated.

And so, this is because we have some usage-based components trying to count out and tease out, okay, this is a service period that Angie had used and this is her billing date and trying to count every single day all the different requests and queries that had happened during that time period.

It becomes very customized to the individual customer.

And so, yeah, that was a very big change in mentality and, you know, internally just trying to rock that in terms of how we're going to communicate that to customers.

And then we had to build a platform with the data team so that we can go ahead and aggregate those different items and be able to count it and reliably get that information back before we had to invoice the customer.

So, it took quite a bit of time for us to get to the point where we could do something that now is pretty easy.

Easy. It's like an issue to us. But I remember when talking to another engineering leader, you know, the team was like, usage is in arrears, service is forward-looking, and that's normal.

Think about it, you know, and I was like, oh my god, that is normal.

But wow, that's a lot more work. Like, it's one of those things where you just take for granted and then when you ask around industry, you know, how big is the billing team of your SaaS company, it's always bigger than you might suspect because these problems are really tricky and they're really important to get right.

The other thing I wanted to ask you about, Angie, and Martin, jump in, please, is we had a really big milestone last year, just over a year ago, when Cloudflare went public.

And in some ways, it's a milestone on a very long journey.

But one of the things, it was unique in the sense that, unlike all the other engineering teams, billing suddenly had a lot of work and a lot of things that needed to double check and potentially replace some components as we were getting ready to go public.

What were some of the challenges that we had to solve from an engineering and product level as we were getting ready to become a public company, Angie?

So, one of the things we had to do was we had to take a step back and look at the infrastructure that we had in place and the different vendors that we were using.

And we had to make sure that we're in compliance with all the different regulations that are required.

So, we did an audit and, you know, with the partnership of our CFO and the accounting team, we realized that we needed to switch vendors.

And so, a lot of that was ripping out of the plumbing that we had so deeply coupled into another application and trying to simplify things.

So, that obviously is not easy while you're trying to maintain business and deep collecting money.

That's right. All of those jokes about replacing the engine while the plane's in flight are 100%.

So true. So very true. And we had to start figuring how do we migrate new customers onto this new platform.

So, that was one thing that was happening in parallel.

And then, there was a lot of just data hygiene.

So, we had been running up under this model for about four years, I think, since we had gone through the last migration.

And obviously, during that time, we just accumulate a lot of garbage.

And trying to clean all of that up was no easy task.

So, again, it's situations when you're talking about the interns in Project Galileo.

We had a lot of customers that were giving service. And we had no reason or understanding why.

And then, we had collected a lot of information from customers.

And we were also embarking on collecting taxes from customers as well. And we realized in order to collect taxes, we need to have good clean data.

And when we're looking at addresses, we would have people from New York, New York, or NY, New York, the big Apple of New York.

It was just like classic data cleanup problems. And I remember engineering writing lots of tools and diff tools and just as you would any other data hygiene project to make sure that long before we got anywhere near all of our systems were in place.

And I'm pleased to say that that was, engineering and product worked well on that.

Yeah, we hammered through it. We got through it. I want to switch gears again.

One of the other things that's interesting about Cloudflare is we have customers all over the world.

Martijn, how does having customers all over the world affect what some of the responsibilities the billing platform has?

Angie touched on the word tax. There's other rules and regulations that apply in different places.

How does that affect an international global, a service like ours, which is used everywhere?

What do we have to do to the billing platform to make sure that it could handle customers in every corner of the world?

Yeah, I mean, it could mean anything from integrating with different vendors that we can actually charge tax.

Using tax is a good example. There are certain countries where you have to integrate with the actual local government and deposit invoices or send invoices directly to them.

It could be anything from that to, there are certain countries where we can't do business legally.

And we need to make sure that we don't actually accept money from people from within those countries.

And so there's a lot of different bits that go into actually making sure that before things get through a billing system, before we actually start to collect money, that we filter that out and we block any fraudulent charges and things like that as well.

That's great. Ultimately, the mission of the billing team at Cloudflare is a platform because we want, we're just like anything else at Cloudflare, where you own a service that we are then building other products on top of.

Angie, what are some of the other examples? If I were to survey all your peers on the product management team, what are other features and other kinds of things they want to be able to do when it comes to billing?

So one of the things that the onboarding team has mentioned was they want to have a more guided onboarding experience.

So right now, today is you go pick a plan, end of transaction. Then you go pick Argo, that's another dispute transaction.

The onboarding piece isn't as fluid and doesn't bring all those services together.

So one of the challenges that we're trying to solve for is how do we pull all those things into a seamless experience?

But it's not so easy because all these things are interconnected and trying to orchestrate the back ends and the success here and dependency that we have there before we can do this.

It's quite a feat. And so that's one of the things that we're looking forward to solving for, because I think it'll bring a much better experience for our new customers.

Another piece that we keep facing is just being able to package and price things in different ways.

So again, we've got lots of awesome products, lots of different tools and features.

And what we're trying to do is build a solution to customers, but we're bringing them individual components.

So again, just thinking smarter about how do we provide a different experience for customers that just want to focus on security?

Those that want to focus just on performance.

How do we bundle those things together in a way that makes sense to customers?

Yeah, exactly. And so interestingly, when a product manager says, I want to be able to bundle this product with this feature, or I want to create a package, or maybe that's even coming from the executive level, it should be super easy.

We should be able to charge people X dollars a month and give them access to all these things.

How do you even start to tease about that? Walk me through a little bit like our audience through what does a billing PRD look like?

What goes into it? Because it sounds there's a lot of stuff that has to coordinate with other engineering teams that you may or may not have any control over.

So how do you even approach the engineering and product challenge here? So I think you touched on it perfectly.

So people think that it's just so easy. We just collect money.

It should be straightforward. But it comes to just a lot of just going through different use cases and understanding fundamentally what it is that they're trying to offer.

Yeah. I think that's the biggest thing.

And then we just go through like, what about this scenario? What about this scenario?

Just because they don't see all the edge scenarios that we're so accustomed to.

And especially because of all the... We've learned a lot as well with different product launches.

We've had a lot of things that are built tied to plans.

And it caused confusion because it was a huge component that people weren't accustomed to.

And then they would turn things on and, oh my gosh, I have a $10 ,000 bill.

I didn't expect that to happen because I thought it was just a business plan.

So a lot of it's just like communication, collaboration. And yeah, when you look at the PRDs, some of them seem like it's just in a different language because it's terms that people aren't accustomed to.

That's kind of what I wanted to highlight, the word arrears and UBB.

It's almost like there's a whole sub-language there.

And I'm always impressed by how deep the if-then-else... It's never about the happy path.

It's always about the exception path. Martijn, does a war story come to mind or not even a war story, but just a tricky subtlety of an example of how some of these products can have interesting interactions and where you as part of the product and engineering team have had to think through important corner cases?

Yeah. I mean, I think most recently, our firewall team had an interesting use case where they wanted to be able to give certain functionalities based on the highest plan type in the account, which again, seems like a pretty straightforward question, right?

Like just show me what is the highest plan level of my account for any given zone.

But when you're working in a billing world, it's a little bit different.

And so those are some of the unique use cases that come up where we then have to go and think about, okay, well, how do we actually go back and provide that as a service to the teams and not just in a way that it works for that one team, but also now that it becomes available to other teams who may want to benefit from that.

And I think that really stems from Cloudflare has evolved as a company.

We're building new products. We're innovating on a daily basis. And the same is true with how we provision and sell those products, right?

We're looking at different ways that we want to package things.

And that's why those needs arise, right?

That's why we constantly have to think of new ways to actually make products available to customers.

That's great. Angie, just as we only have a couple of minutes left here, for any other billing PMs who are out there working at a SaaS company that's got legs and is growing, or even if I reframe the question as advice you would give yourself four years ago when you joined Cloudflare, what would be some of the things to keep in mind other than hold on as the rocket ship approaches whatever 200 Gs it felt like as we were preparing to become a public company?

I would say never assume you're done. Things are always going to change, especially as your business evolves, the company evolves.

If you don't have more needs coming in, then something is wrong.

We thought we had settled on a vendor. We're happy.

We're good. We're status quo. And then immediately thereafter, we realized, oh my goodness, we've already outgrown it.

So every day, it's just an incremental reiteration and just expect again that things will change every single day.

Martijn, how about you? Biggest learning? Probably always triple check things.

As Juan, our CIO, put it right, measure twice and cut once. That's really what we're trying to do.

We want to make sure that we don't overcharge our customers, that we allow customers to actually purchase.

That would be some of the feedback.

I think we're still learning. And that would be what I would give myself a few years ago.

We're continually building brand new products. And I'm so proud of everything we've achieved so far because it is an extraordinary customer base of so many different sizes and an incredibly diverse portfolio.

And the Cloudflare building team, a relatively small group of engineers has such a central role.

And it's been so much fun to work with both of you and the rest of the team.

I think that's good. Amazingly, as always, we hit 30 minutes before I know it.

But I just want to thank you, Angie and Martijn. Thanks so much for joining me to talk about billing.

And we'll have you back again before long to talk about some of the more recent stuff you're working on.

But with that, everybody, happy Halloween.

Happy Halloween. Happy Halloween. Happy Halloween.