Latest from Product and Engineering

Hi Jen. Hey Usman, how's it going? Good. Welcome to another edition of the latest from Product and Eng.

I'll start. I'm Usman, head of engineering. Jen, who are you?

I'm Jen Taylor, head of product. We have two of our favorite people here with us today.

Can you guys go ahead and introduce yourselves? Sure. Hi, I'm Dina. I'm the product manager for SSL TLS at Cloudflare.

I've been here for about two years now.

And I'm Max. I've been working on the SSL team for several years, but I've been at Cloudflare for almost nine at this point.

Nine years. We're just marveling.

That is really something. Max is one of the people who helped explain how all this stuff worked to me, explain to me how all this stuff worked.

And Dina was of course also, last time we had Dina on was because of DNS because she was the product manager for DNS as well.

So welcome both of you. I think, you know, as usual, we'll just start from the beginning.

Dina, why don't you just help us understand a little bit?

Like what is SSL? Why does that acronym roll off the tongue so fast and so frequently at Cloudflare?

We're constantly talking about SSL.

We have a whole team called SSL. And so what is this thing? Why is it so important?

Why is this a big part of what Cloudflare does? Yeah. So SSL is a protocol.

It is the reason why traffic on the Internet stays encrypted. It's the reason why you can access your bank account without someone stealing your information.

And it does all of this by encrypting sensitive messages with public key cryptography.

But beyond that, what it allows you to do is it's really important when servers are talking to each other that the server they're talking to is who, in fact, they think they are.

And so they also help validate that. So overall, they keep browsing online safe and make sure that the identity of servers who they think it is.

Yeah. So it's both encrypting things, which is like putting a lock, but also making sure that the lock is actually that you are securely talking to the correct website.

Because it doesn't help if you're securely talking to a website that isn't pretending to be your bank.

That's not a better situation at all. So to do this, how do you make a website secure?

Like it's now so de facto that in front of us, in the years that we've all been working in Cloudflare, big major browsers like Chrome now highlight the opposite.

It used to be they would make a big fuss to show you that something was secure.

And we were always teaching users like look for the green lock and look for the name of the bank in your URL bar.

And now we're past that.

That is now considered table stakes. And in fact, the browser highlights when it's not secure.

So Dina, how does this happen? If I have a new website, usmansdonuts.com, how do I make it secure?

What do I have to do to eat?

Forget Cloudflare for a second. How do I ensure that my visitors know that they're hitting the good usmansdonuts.com and not the imposter jensdonuts.com, who's trying to sell jelly donuts.

Steal his donuts. That's right. How do I make sure that my visitors are visiting the right website and that it's secure?

Yeah. So the way it works is there are certificate authorities.

They issue these things called certificates.

And a certificate essentially allows you to have a public and a private key.

And so the certificate is the public key portion. And so when a server is talking to another server, they essentially present each other with credentials, with signing algorithms, and they encrypt the messages that get transferred between them.

But how does a certificate work?

Where does it come from? And how do we get them? And how do we associate them with our customers at Cloudflare?

So at Cloudflare, we partner with certificate authorities such as DigiCert and Let's Encrypt.

And so we issue for our customers certificates on their behalf.

And so when you're proxying traffic through Cloudflare, essentially your eyeballs are hitting Cloudflare's edge.

And so there's a thing that happens called TLS termination. And so that's where we are going to serve the certificate on your behalf and encrypt the messages between the eyeball and us, and then hopefully from us to your origin server.

Now, Max, this is something we've been doing at Cloudflare for a long time. Can you take us into the way back machine?

How did Cloudflare get into this? It used to be that certificates were expensive.

They were hard. As Usman pointed out, very few people used to have them.

The world has shifted. And what has that journey been like for us at Cloudflare?

Sure. So putting on my Cloudflare historian hat, I'm going to- Cloudflare archaeologist.

Archaeologist is a better word. Basically, when I first joined at Cloudflare, only Cloudflare's paying customers got SSL certificates.

And we've touched on the reasons why. It's because it was expensive and difficult to provision them back in, say, 2013.

Over the years, we realized that certificates are more important than just what paying customers need.

So in 2014, Cloudflare rolled out with our product that we call Universal SSL.

And the whole goal behind that was to make certificates, SSL certificates available and ubiquitous for everyone on the web and starting with Cloudflare's customers.

So every single website that signed up for Cloudflare got a universal SSL certificate for free with no additional actions needed on their behalf.

And I just want to interrupt for a second.

At this time in the story, I was not at Cloudflare. I was at my own little startup, which was perpetually broke.

And we had a website. And I was blown away when I got the email from Cloudflare saying, we will give you an SSL cert for free, which would easily run in the $300 or $400 just to get one for a year, a cheap one.

And it's interesting to note that you might be like, well, what are you paying for?

It's just a giant string of numbers. You're paying for those certificate authorities, which is a third-party company, to basically sign and say, yep, this is who they say they are.

It's sort of the Internet equivalent of a notary public and saying, this is who they say they are.

And I'm signing this.

And the browser vendors and clients can then validate and say, sure, OK, so if DigiCert or Let's Encrypt or whoever says this is really a certificate, then we've got it.

So this is why it was a monumental decision to say that we're going to give universal SSL to basically the whole Internet.

It was just amazing.

Yeah. And that's exactly right. So what most people tend to skip over when they're talking about certificates is the fact that a certificate makes some sort of claim about identity.

That is, the server is who they say they are. And it's up to the certification authority, what we'll probably refer to from here on out as a CA.

What they have to do is they have to receive the order that we give them. Then they have to go out and prove identity to some extent.

In the simplest case, we do what's called a domain-validated cert.

And that basically is if max.com is out there on the Internet and I order a certificate, the certification authority has to go say, OK, does he actually control max.com?

And then once they prove that I do, and hopefully I do, then they're able to issue that certificate and give it to me.

And then the client, so say Dina goes to visit max.com, she can then verify independently that, OK, it was issued by the certificate authority and I trust them to have verified this claim of identity to some degree.

And that's how Dina can trust that she's really visiting max.com and not- So every computer has a list of what's known as root certificates that they trust.

And that's what the certification authorities manages are these roots.

Interesting. OK, so we came out with Universal, SSL in 2014.

But then we've done a bunch of really, I think, interesting things kind of between now and then.

And like, just as we like, so where are we at now? How many certs are we issuing a day?

It's a big number. Yeah, I know on a slow day, it's probably hundreds of thousands.

On a big day, it's millions. And some public certification authorities like Let's Encrypt like to tout those millions of numbers.

And we're in a similar ballpark. It's kind of amazing to think about that back when I joined Cloudflare, the Universal SSL pipeline was a little slow and it could only do maybe tens of thousands per day.

And over the years, we've since scaled that to handle this millions number.

That's like three orders of magnitude difference.

And again, it means, and I think the important thing to remember here is this means someone says, I want a certificate.

We get that request, create that public and private pair that Dino was talking about, send the public one to a completely different company, tell them to sign it, sign their name on it, get it back, then ship both of them to the Cloudflare Edge and start serving it.

And all that has to happen in a way that is seamless and easy for the site administrator to know that, okay, this is actually happening.

It's not stuck. It's not blocked.

It's not getting... Because if it doesn't show up, no security for you at the edge of your website.

And so it is extraordinary that we've been able to keep this growth going.

Yeah, that's exactly right. Not only does it take us being able to have all these customers and then orders on their behalf, but then we need certification authority partners that can handle this deluge of orders.

And then we have to then download the issue certificates and then deploy them to all of our data centers around the world, which is not a small task.

What do we do? Do we go out and buy?

We're just like, oh, great. Let's get more servers. Let's get more hardware.

How do we respond to the scale? Sure. So there's been a few fires in the past where we realized there's a doomsday clock.

We're like, oh, we're ordering so many certificates.

It's consuming a lot of space on disk. Because at the end of the day, a certificate is basically just a file with some fancy cryptography saying that here are your keys.

And then all of this can be proven back to the issuer.

So we can just optimize how we save files. In the past, we were just doing a very no brainer sort of thing where we would take the file and just ship it out to the edge.

And that was that. Why not? That sounds like a perfectly good place to start.

And that works. But over time, you realize, well, there's a lot of things about these files that are literally identical.

So then we can get smart about only sending the pieces that are different and then doing compression on the things that are different and then using a different file format so that the actual file itself is smaller.

And these are all incremental improvements that we've done over the years.

And one thing that we've done most recently is we used to store all of these files on every machine.

So that was just a lot of disk space.

Now we're starting to be a little bit more smarter about how we actually allocate which hard drives and which machines actually get this data.

So there's like a space optimization.

There's a time optimization. There's a delta of the change optimization.

It's actually one of the most, from my vantage point, one of the most interesting things at Cloudflare is to see these teams that are all scaling infrastructure on multiple axes.

And the team that owns that configuration value key store sort of swiveling their chair and going, hey, SSL team, we need you guys to tighten the belt a little bit here.

And SSL team's like, why? It looks great to us.

Like, yeah, of course it looks great to you. Like, here's the thing. It was one that I love about the teams at Cloudflare because they are doing everything you just said.

They're thinking about tightening the belt. They're thinking about kind of how to make it faster, how to make it more efficient.

But they're also innovating at the same time.

Dina, can you talk us through, like, we're moving to the notion of an advanced certificate manager.

Like, what does that mean? And like, what innovation are we adding at the same time that we're tightening our belts?

Yeah. So after we launched Universal SSL, that worked for most of our customers, but not all of them.

And so we launched dedicated certificates, which allowed you to have your, back in the day, we used to share certificates amongst different zones.

And some customers did not want to be on the same certificate as others.

And so we gave them the ability to have their own dedicated certificate, which they could order.

It could have not just your APEX in the wildcard, which the Universal cert had, but it could have up to 50 subdomains.

But then as we've been making all of these optimizations, we've decided to move to more of an advanced certificate manager kind of system, where instead of paying for each dedicated certificate, which certificates are pretty cheap, so there's no reason why we should be making you pay for each one.

Instead, what we ask you, what we do is the customer pays for this advanced manager.

And with that, they get up to 100 certificates for their zone. They get lots of subdomains covered, but not just that, but it gives you the ability to tie in a few security knobs, and they're really easy to do to increase your security posture.

So some of these are, you can control the cipher suites that are used for TLS.

So for example, if you only want the most secure cipher suites, you can now choose, you can now indicate that.

You can choose which certificate authority you want to issue the certificate.

You can also decrease your certificate's validity period to 14 days, up to 14 days now, which is really important because the shorter your validity period, the kind of less time that it's possible for like the key or the cert to be compromised, and it decreasing the validity period reduces the risk.

And it's something that overall the whole industry is moving towards.

But yeah, that's advanced certificate manager. We're going to continue to build things on top of this.

Another thing you can do is if you want Cloudflare to take care of your private key, but you want to go and get a certificate from your own certificate authority of choice, Cloudflare can, you can ask Cloudflare for a certificate signing request, which you will then take to the CA of your choice and get that certificate and upload it to Cloudflare.

But we continue to build things on top of this advanced certificate manager.

So more power, more flexibility, more control, more scale.

Yeah. Now, so I want to kind of shift gears for a moment because, you know, when we talk about scale, I think the other thing that this team has done that has been so amazing and frankly, I think transformative for the industry is the way in which we've helped SaaS applications actually ensure that they're serving encrypted traffic.

Can one of you talk a little bit about what we've done from a product perspective there?

Yeah. So a couple of years ago, this wasn't just like a small company HubSpot came to us.

And so they were trying to build out a whole certificate issuance pipeline for their own customers.

And so they have tons of, they have millions of websites on them and they want to give their customers the ability to stay secure and keep their traffic encrypted.

But building a whole certificate issuance pipeline is not trivial at all.

You need to integrate with a certificate authority. You need to book a database and a system that can issue, that can renew these certificates.

You need to make sure that you're able to meet your, for example, if you have a bank customer, you need to be able to meet their requirements.

And so even a company like HubSpot was having a hard time kind of building this out.

And so they came to us because we were leaders in the SSL space at the time.

And so we built out SSL for SaaS, which essentially allows a SaaS provider to issue certificates, not just for themselves, but for all of their customers that are currently configured to point to their application.

And this has been incredible through SSL for SaaS, we're actually keeping 10 million websites that are not directly on Cloudflare encrypted and safe and overall increasing the security of the Internet.

This is a really interesting problem because it only showed up after the SaaS revolution took off, right?

Because it used to be that if you're a SaaS company, right, that you, so, okay, so, you know, I've got, I want another company to be able to use, you know, my donut shop, it might be sponsordonuts.com slash Max's cafe, right?

And, but that, who wants that?

Like we want, you know, Max wants to be able to put it all under his website.

The fact that it's powered by my SaaS provider is in the noise, but if, but there's now there's a technical challenge, whose website is it?

Because if it's Max's website, the provider is a company like HubSpot, which like in HubSpot's case, or, yeah, you know, providing great marketing services or, you know, or Shopify or Zendesk or all these companies that have powered the SaaS revolution that have, but we want them to look like they are coming, you know, from the primary, from the customer.

Now you've got three different players.

You've got the eyeball zone, you've got the zone that's providing the service, and then underneath the covers, you've got Cloudflare, and yet it's got to, it's all got to fit into the certificate pipeline, the technology that a browser and a certificate authority needs.

So, very interesting problem. Very simple to explain on one end.

It's just like, look, just make it secure and make it work.

Like what is, what are you guys talking about? Like there's nothing to discuss here.

It's very obvious what success looks like. And yet, you know, three different companies interacting and trading certificates and pipelines underneath the covers is really interesting.

Well, and if my experience at Cloudflare has taught me nothing, if it sounds simple, will you explain it?

Part of what we do is really, we translate the complexity into simplicity.

And I'm kind of curious, Max, because you were, you were there on the ground floor when we were doing this, you know, what were some of the challenges we faced as we thought about extending this, this SaaS pipeline and offering we built for ourselves now to other SaaS providers?

Was it a heavy lift? Was it a light lift? Like what were some of the challenges?

That's a great question. I remember when we first came up with this idea that a SaaS provider can sign up for Cloudflare and then extend Cloudflare to all of their end customers.

I didn't really understand it. You're like, draw me a picture.

Like for me, I was like, when somebody first told me, I'm like, slow down.

Can you draw me a picture? Like, yeah, sorry. Right. And internally we had this goofy name for it.

It was just called managed C name, which who knows what that stands for.

But basically what it was, was SSL for SaaS. Well, without the SSL part, originally it was just for SaaS.

Cloudflare for SaaS, right.

Yeah, right. Yeah. There were no certificates involved because that was the additional difficult part was, okay, well, if you're not signed up for Cloudflare for directly, how do we know to place an order?

And then how do we get it to validate?

That was the tricky part that we ended up solving later on. But to get back to Jen's original question, this wasn't an easy thing.

The original managed name component was the shortest possible path to success.

And then we realized that was very inflexible.

So we kind of had to start over and do the whole SSL for SaaS thing from the ground up.

And now customers explicitly tell us what their downstream customers are going to be.

And that's the way we know, okay, we need to order a certificate for this host name.

And then we also need to go get it. And then there's a few different ways in which they can actually validate it, whether manually or automatically, but the choice is theirs.

And I think that's one of the things that SaaS providers really like is there's a lot of flexible options as to how to integrate this product.

And one of the more, I think, interesting things that Usman touched on just a moment ago was that it's one thing to be able to have Usman's donut shop be the provider for all the donut shops in the area.

It's another thing to brand it appropriately.

So that my cafe has all of my fancy, I don't know, baseball branding.

And then Dina has her lava lamp branding and it all just works out.

Yeah. Yeah. Well, and Dina, I know that kind of even more recently, I mean, obviously SSL for SaaS has been, it's been just a phenomenal to see the growth of the product over the four years that I've been here.

But more recently we've actually rebranded this ourselves and we're starting to actually think about sort of what is possible to build on top of that.

Can you talk about that a little bit? Yeah.

So we're kind of doing two things. We're rebranding SSL for SaaS as Cloudflare for SaaS.

So what we actually realized was that SSL for SaaS was much more than just the SSL piece.

When you're essentially configuring the SSL for SaaS, you're not just issuing a certificate for your end customer.

What you're also doing is you're extending all of the benefits that Cloudflare is giving to the SaaS provider.

So that's the DDoS protection, any performance boosters, any firewall rules, et cetera.

You're extending those to your end customers. And so now your end customers, so now your end customers stay fast and secure and they don't have to worry about reliability, which means that you don't have to worry about it as much.

But not just that, but we're essentially taking care of their whole infrastructure of the SaaS provider, but also of all of their customers.

And so Cloudflare for SaaS kind of encompasses of just, there's all these challenges that when you're building out your SaaS application, such as setting up an origin server or keeping your customers' traffic encrypted, low latency, et cetera.

And so Cloudflare for SaaS essentially takes care of all of these so that you can just focus on building out your SaaS solution and we'll take care of the rest.

One of the cooler things that I think also integrates well with Cloudflare for SaaS is the whole bring your own IP.

So with that, your customers almost have to do nothing at all to onboard.

And that's super powerful because if you already have a SaaS service and you have your own allocation and all of your customers are already pointing to that, it's easy just to have Cloudflare start announcing those IPs.

And then magically Cloudflare turns on for everything and they don't have to change their DNS records someplace.

And you don't have to worry about chasing down all of the individual cats that are your customers and getting them to like, you need to update it by this date, otherwise things are going to break and so on and so forth.

It's pretty magical, to be honest. That's not something that we used to do a lot in the past, but our addressing team, which aren't on this show today, they're really fun.

It's a good crew. How does this intersect, Dina, with Cloudflare for Pages and some of these other newer stuff that we're working on?

So again, like I have often said, part of the success looks like becoming a platform.

If you know you've built something right, if suddenly everyone else is showing up on your shoulders, uninvited.

So what else is built on top of this? Yeah. So I guess in a way, we are built on top of ourselves, but Cloudflare Pages, for example, is built on top of SSL for SaaS or Cloudflare for SaaS.

And so every time you make a Pages website, we're provisioning the certificate for you in the background and we're giving you all of these enhancements so that your website can be super fast, super secure.

It's served from all 200 locations and it's fully essentially hosted on Cloudflare, which is incredible.

If you didn't have some of the advantages that you talked about with the ACM and some of these new features, you could still do this, but it was a much bigger pain in the neck.

So walk through a little bit, what did you have to do otherwise?

If I wanted to change the Cypher suite right now, you've got it down to a knob in the dashboard.

What was involved before to have to switch that around?

Oh, that's a great question. So before we introduced the API and the dashboard settings is basically whoever needed to change this would have to submit a support ticket.

And then we would have to manually make this change at our edge.

And then every time the certificate changed or basically every time the certificate changed, we would have to either reapply or do that.

And it was just a bear.

It was just very untenable and it didn't really scale. Giving this control to our customers via a pretty straightforward API is much easier and requires no interaction and they can change it whenever they want to.

Yeah. So lesson for budding startups and growing companies everywhere.

If you see your own staff doing something that's tedious and no fun, and you know, the customers on the other side of that conversation are also doing something that's tedious and no fun.

There's your product opportunity right there.

Streamline that whole thing, turn it into a product.

And I love that this went all the way to even things like the exact cryptography algorithm that's used in the certificate.

It's just a checkbox. It's just a button.

And that in the backend turns into roll the new cert, provision it, and it automatically will get updated on the lifetime validity things that are...

I can't believe how fast this time went. We've already hit the 27 minute mark.

Just wanted to thank both of you for joining us. This team is always going to keep working on incredible scale and incredible new stuff here.

So thanks both of you. I think the last thing I just wanted to say was we're also working on an API shield though, Dina.

So just in the last couple of minutes, can you talk a little bit about what API shield is and how that also is built on some of the same tech here?

Yeah. So API shield allows you to keep your API and points protected.

And so this integrates with our MTLS, mutual TLS system.

And so it essentially allows you to provision or revoke client certificates for your APIs and keep those encrypted.

And we could spend a whole other talk on that because that's in the other direction, right?

That's rather than the server proving I'm really who I am, that's the client saying, no, I'm really who I am.

And which is unusual because normally you think of a website, the whole point of a website is to distribute information.

Why do you care who's coming to it? But it does matter if it's in the Internet of things, if you don't want something to pretend to be your thermostat when it really isn't.

And so there's a whole other universe.

And so we'll save that whole section of talk for a different time, but thanks both of you for joining us.

And it's amazing to see how much work we've done.

And Max, I can't believe like three orders of magnitude, we're going to keep seeing this growing over the coming years.

Yeah. Here's to a hundred million more. A hundred million more certs.

Here we go. Over a billion serve to be like McDonald's, right?

I'll get a hat. I know that'd be a good hat. We'll get those. All right.

Thanks a lot. Have a good weekend. All right. Thank you. There's too much that goes into creating high quality video today.

That's just simply still too hard for many of our customers.

Most cloud providers don't actually provide a turnkey solution for video.

They provide bits and pieces of the equation, but there's no provider that provides an end-to-end solution from rendering to streaming.

They'll provide bits and pieces that now you have to kind of cobble together to build an amazing product.

Our focus now is how do we simplify and streamline that by providing a deeply integrated, simple and easy to use solution.

A big part of what we do at Cloudflare is as we focus on helping build a better Internet, is take complicated things and make them simple.

And to enable them to just literally be able to go to Cloudflare, to log in, to point their video asset at Cloudflare, and then on the other end, be able to pull a player out of Cloudflare and place it wherever they need to be able to deliver the video.

And that's it.

There's a triplicate where you could do something either well or fast or cheaply.

And so we're striving for all three because we really need it. We need it to be really good because otherwise why would anyone use the service?

You got an entire Internet out there, use something else.

We need it to be fast because people have no patience.

And we need it to be cheap enough that we can stream to millions of users without it becoming uneconomical.

So you have to get all three and Cloudflare is a really important part of offering all three.

If you want to deliver a video to anybody on the globe, there really is no better network to put it on than Cloudflare because we can guarantee the highest quality experience to somebody who is in New York City and someone who's in Djibouti and someone who's in Sydney.