All right. Welcome back to Security Week. I'm your host, Patrick Donahue. Hopefully you caught our kickoff session earlier today with Michael Tremonti. If not, we'll be posted soon on Cloudflare TV so you can go rewatch it there. Gives a great preview of the week. I'm joined today by Dina Kozlov, who is the product manager for SSL Team. She just published a blog post on a blog for Rt.com, so go check it out if you haven't seen it yet. Ssl TLS is a team near and dear to my heart. I used to manage that team probably five six years ago, at least now. So let's let's start with the basics. What are some TLS certificates and why are they important? How are they used? Yeah, everyone really excited to be here. So certificates are essentially the core of security on the Internet. They do. They do multiple things. One of the most important things that they do is they authenticate a website, meaning that when you go visit some domain and it has a TLS certificate, what that certificate is saying is that you are, in fact visiting the correct server and the right website. The other thing that they do is they help Web traffic stay encrypted between clients and servers. And so overall, the really important to keeping sites secure and safe online. Great. So. So how does one get a certificate? I remember when I when I first got one many, many years ago, I want to say it was gosh, late 90s, early 2000s. I remember having to send some articles of incorporation of a company and actually mail this back to a company and wait a long time to get it. Presumably that's not how they're issued today. Can you just talk me through what that issuance process looks like? So since then, the process has sped up a lot. It's almost instantaneous. But you have different organizations called certificate authorities and certificate authorities are responsible for issuing certificates to applications or domain owners. And so what happens is, as a website owner, I want to get a certificate, for example dot com. The first thing that I need to do is proof that I own the own example dot com because it shouldn't be getting certificates for websites that are not in my control. And so the way I do that is through a process called domain control validation. There's a few different ways that you can prove your ownership of a domain. You can either do it through DNS records, you can do it by serving HTTP tokens, some certificate authorities send an email to you and you validate it that way. But essentially, once you've completed the domain control validation and shown that you do in fact own this website, the certificate, the certificate authority then issues the certificate to you with Cloudflare. What happens is we're the proxy for websites, and so our customers will essentially prove that they own their domain to us or to the certificate authority. And then we will go and we will fetch certificates from our different CA partners like digital cert and let's encrypt and then we will take that certificate and serve it at our edge so that any clients that connect to Cloudflare's Edge are then that connection is encrypted and all of those websites have a TLS certificate. Yeah. Really important that these certificates are only given to those that can demonstrate control. I think I read about a hack a few weeks ago or maybe a month ago now where somebody was able to hijack control and get a certificate issued and sort of bypass that. And then obviously, if you have a certificate, you can intercept traffic and decrypt traffic. And so really important to secure that. I know we've got products that help cars check those methods from many parts around the world, but we'll save that for another day. So what about so you're issued this certificate. Is that a one time thing or do you have to do that periodically? So every certificate comes with an expiration date. So you do have to periodically renew your certificate so that you always have a fresh one that is served at the edge. And one of the reasons why actually short lived certificates, meaning certificates for 14 or 30 days that expire in 14 or 30 days are much better than long certificates like your long certs. Is that certificate and that private key is only valid for that amount for that period of time. So if there is a disaster and someone gains control of your private key actually by when that certificate expires is when their ability to use that private key also expires. So we do encourage our customers to use short lift certificates and we are integrating more and more with certificate authorities. We actually default to 90 days certs, which means that essentially you would renew that certificate every 90 days and get anyone. Yeah, I think it's great to see the ecosystem, the web system, web PKI ecosystem get to a point where everything is automated and you're not mailing things around or doing manual validation procedures because to your point, like being able to get certificates issued quickly and frequently is really great for hygiene, especially when you want to deprecate old protocols and things like that. So it sounds like you're reissuing when certificates expire. What else would you reissue for? There are other reasons why it makes sense to do that. Yep. So the good reason is certificate renewals when they're set to expire. There's a few bad reasons why we would want to renew certificates. One of these is, for example, of a vulnerability. So for example, Heartbleed is a vulnerability that was exposed a few years ago. It allowed anyone with an with a vulnerable version of openness, SSL. If an attacker made use of that, then they could gain control of someone's private keys. And so, in that scenario, you essentially want to roll the private keys onto a new set, which also requires that you issue a new certificate. So that is one example. Another example is just aside from a vulnerability, someone gaining control to your private keys, you would want to instantly get new keys, new certificate. Another reason that we've seen a few times recently is certificate authorities. If they find an issue in their system, they are required by the Certificate Authority Browser Forum standards to revoke any faulty certificates, either within 24 hours or within five days, depending on how bad the issue that they discover is. And so we saw this recently. Let's try to reissue about 2 million certificates. Cloudflare customers were not impacted by this, but actually about nine months ago, one of our cars did have to do a mass certificate revocation because they found that they were reissuing certificates on old domain control validation tokens. And so we had to reissue 5000 certificates immediately. And so with the 24 hour and the five, either 24 hours or the five day window, that's a very short period amount. That's a very short period of time in which you have to get a new certificate or else the certificate authority will market us revoked any and any browsers or clients that check for those certificates. They will no longer be able to essentially serve your website unless you have a new certificate ready to go. Yeah. That I remember. I think Heartbleed was 2014. I think that was the year before I joined Cloudflare. And I remember we had a very large back then revocation list. CRL was sort of the way of the day sort of before CSP became a lot more prevalent, no CSP stapling and was quite a burden of serving that very large revocation list. And then these days, a lot of browsers don't even check that right. So being able to quickly re-issue and move to a different certificate is clearly important here. I think just to fill everyone in on what Cloudflare has done to protect against, as you mentioned, open SSL had this vulnerability. We of course moved to boring SSL, a sort of smaller code base, a bit more hardened and tested and a project run by Google. Obviously, we've also moved key. So the Heartbleed was getting the bits of the private key a little bit at a time. I just remember we had this challenge where we put up a challenge site and said, try to get the private key for this hostname. And we were thinking, you know, this might take days or weeks for somebody to do. And I think it was like within 24 hours I might be off a little bit, but somebody is able to extract the private key. And so moving those private keys out of that process and doing something that we call Key List, that you obviously spend quite a bit of time working on doing key lists, maybe even on that machine or in different locations. And we can do that today by doing it in other parts of the world, right where we can serve keys from. And customers can keep those keys themselves if they want on their own infrastructure. So a lot of work that's been done to kind of harden against things like Heartbleed. But of course, there are things that are going to happen this year. Right. So you just took us through a couple of those things. If a Cloudflare Scale today, we have millions and millions and millions and millions of certificates that we're managing on behalf of our customers. How long would it take to kind of reissue all of those certificates? So our team did an estimation. It would take about weeks to reissue 45 million certificates. And one of the reasons why it would take so long is even if our pipeline system can handle the re-issuance, we rely on our certificate authority partners to be able to issue as well. And so there's a lot of different kind of components in the system that need to work to be able to reissue it that scale. And with 45 million, that's just not it would cause that's just too high of a scale for our systems. So yeah. Yeah. No. I remember when we were originally renewing all those universal SSL certificates, we were getting rate limited by we worked with some different partners or fewer partners back then and just being able to take a dependency on a third party, an issue that you can only go so fast as the underlying ca and so clearly not something that we want to have to do or wait weeks. If there is a particular issue, if a CS is compromised, our customers expect us to react a lot quicker than that. And that's why they use Cloudflare to make that easy for them and take that burden away from them. So what would happen if we weren't able to reissue those, what would actually happen from a security perspective? So either in a compromise or going to see a revocation, your application domain is left vulnerable. So either attackers gain control of your private keys and are able to serve the domain on their behalf and you're not able to kind of regain control. Or if you're left without a certificate at all, then all of the traffic to your website is insecure. And so any onlooker can get information like credit card bank statements. And so you definitely want to keep that information encrypted and secure. And to do that, you have to have a TLS certificate that is valid. And so you would essentially be left insecure or just completely down because you wouldn't want to serve insecure traffic to your customers. Neither of those sound like good options. So let's jump in to your announcement today. What are we doing kind of proactively to to avoid that scenario? Yeah. So I'm excited to talk about backup certificates. So essentially the team started thinking about this a few months ago of if we had to reissue all 45 million certificates, how would we do that? Since we're not able to do that reliably or we don't want to wait until a disaster to do that? What we're going to start doing is ordering a backup certificate for all of our universal certificate packs. So essentially every time we order a new pack, either you on board a new zone or a renewal comes up. We usually just get another we usually just get one universal SSL certificate, but now we're going to get a backup certificate and this certificate is going to be issued from a different CA than the primary one. And so this is to help with cases like CA revocation. If that comes up, it wouldn't impact the backup certificate. And the other thing that we're doing is we're wrapping the backup certificate with a different private key, then the primary certificate. So that, again, in the case of a key compromise, the backup certificate is unaffected. And so then in an event of a disaster, our team can quickly deploy backup certificates to the edge. Preventing impact. Very cool. So... Sounds like we're. We're starting this process now or we're about to start it. Can you just take me through? I didn't quite catch. Who are we doing this for? Is this Universal SSL customers? No, we have other products. Advanced Certificate Manager, that sort of thing. Like who is going to be benefited from from this? So the end goal is to have a backup certificate for every Cloudflare Support cert. But to start we have our Universal SSL product. That's the product that gives all customers free TLS certificates. And so we have two kinds of customers. We have customers that use Cloudflare as their authoritative as their authoritative DNS provider. And we have customers that only use Cloudflare as their proxy and use a DNS provider that's elsewhere. And so to start, we're going to issue backup certs for customers who use Cloudflare as their authoritative DNS provider. And we're going to issue backup certs for all of the universal certificates we do in the future. Want to support our customers to have their DNS elsewhere. The only caveat there is that issuing backup certificates for those customers will likely involve manual intervention with the customer because they would have to add in the domain control validation records that we talked about earlier. But then. Sorry, go ahead. I was just going to say, but after we issue backup certificates for all Universal SSL certs, which make up about 76% of the certificates in our pipeline. After that, we want to issue backup certs for advanced certificates and for our SSL for SaaS customers. First, ask providers that manage TLS certificates for their millions of customers. We also want to issue backup certificates for those and then also for our customers who upload their own certificates to Cloudflare. If they would like. We would also one day would like to issue backup certificates for them in case they ever forget to renew the certificate that they upload or some issues come up that they also have a backup that's ready to go. That makes sense. And so in order to have a backup and it sounds like a great sort of ordering of these from most valued and most people and then sort of transitioning over time to getting everything covered. It sounds like we're going to have to have multiple CA partners cert with our partners, the people performing that validation you mentioned. Can you talk a little bit about like how do we kind of think about that? Do we have multiple today or are we adding any how does how do we actually get to the point where we have a good diversity of that ecosystem? Sure. So today, the primary certificate authorities that we use are digital cert. And let's encrypt the backup certificate authority that we're going that we are currently using is tactical. We are currently working on adding more cars into our pipeline. So for example, Google is one of these certificate authorities. And so we want to get to a point where we onboard one more Acme compliant CA. Acme just means that they have an automatic certificate issuance integration. And so we want to get to a point where our customers where we can essentially load balance between different backup cars, our customers can have multiple backup certificates. And so that way no one CA has too much impact on our customers if an issue ever comes up. Yeah, and that's a really interesting I know in talking to customers previously, sometimes they work with a particular CA, so some kind of work with digital cert already or let's encrypt or one of the other ones and they have a preference or an affinity for that. I know we've we've let people sort of choose that from a primary perspective. Do you anticipate we'll let people choose who they'd like as a as a backup perspective as well? For sure. One of the things that we want to add to backup certificates is configurability. So allowing our customers to choose their backup certificate authority, where just right now starting with set to go. But like I said, we would like in the future to give our customers the option to choose and in the future also have multiple backup certificates. So if there ever is an issue with one of our backup cars, there's also a backup ready to go. That's that's great. And I know as someone who was a previous a Cloudflare customer before I became a Cloudflare employee, it was great to not have to worry about that stuff. Dealing with that is not fun and rather turn it over to somebody else to sort of manage that process, somebody that specializes in it. So great to see that. On behalf of our customers, you mentioned something called Acme. Just want to spend a second on that. If you think back to when we were early, early days, integrating to the different cars felt like everyone had their own sort of specific API that we were calling and going through different flows. Some would sort of do the validation of issuance all at once, some would do validation as a separate step from issuance and so on. And the effort for the engineering team that you work with here and team is, is sort of, as you add, each CA seems to add or double or triple or whatever the effort as you add cars. How does Acme kind of help that process? Think we think we lost Dina here. So I'll answer that question myself. The thing about Acme is it's a standard. And so the idea is that CAs will implement to a standard. Let's encrypt. You really helped pioneer this. And they were one of the big early adopters and helping drive automated issuance. And so it's great to see certificate authorities adopting this model. It makes it makes it easy for companies like Cloudflare to integrate. And so if you're a certificate authority listening to this and you don't have an endpoint, we look forward to seeing you launch one and make it easier to work with Cloudflare. So with that, I think we'll wrap. I know we started to issue some of these and we're excited to hear your feedback. If you have any questions, please, please let us know. Dina is monitoring all that feedback and she's looking forward to incorporating that into the product plans in the roadmap. Looks like we got a question coming in. I will try to answer it to the best of my knowledge. And if Dina is able to rejoin us, she can probably add some additional context there. So one is as part of the backup certificate generation process, do you have plans to also generate and publish TLS slash DNS records? That's a great question. And let me explain what those records are. And so if you think about when you go to a website and you're going to NPS, you get that certificate back to your browser. Your browser is essentially chaining that back up to some sort of root certificate that ships with either the browser or the operating system. Mozilla operates a trust or there are other trust stores out there that Mozilla is used by multiple browsers, depending on the platform you're running on. That certificate of trust goes up. There's also another way to convey the validity of the certificate or the specific certificate that you're looking to do, and that's by publishing DNS records. And so this is something that has not really been adopted on the website largely for latency concerns. And so the nice thing about having it locally and chatting back to it is you can you can validate and prove the correct certificate without having to make any sort of DNS queries. But this has really been adopted quite a bit on the email side. And so if we're able to then take that certificate that we're issuing and automatically publish in DNS and manage it on behalf of our customers, that becomes much easier. And so that is absolutely something I know that the team is thinking about and wanting to do for those that do like TLS or in DNS records. And so if you're interested, go ahead and read more about that. The second question is, will you use both certificates randomly in production or will almost all certificates be from the primary CAA until a vulnerability is detected and then everything switches to the backup? My understanding is that we are going to have a order and so we'll have a primary certificate that will always be used and then a backup certificate that would get flipped if there was some sort of compromise or reason to use a different certificate. Not actually a bad idea from a feature request perspective to check resiliency, but from a predictability perspective, from a way to actually give some predictable responses. Some of our customers will actually make connections and sort of expect to see a particular certificate back. And so if we were kind of randomizing or bouncing between those might throw off some of their testing. Of course, some customers will actually also pin in a mobile application. Typically they'll say, I expect this particular certificate, hopefully not the the leaf certificate, the one that's that's sent back, but maybe something higher up in the chain and intermediate, for example, because you may be reissuing on renewal and you don't want to break your application, you don't want to lock people out of it. And so my expectation from chatting to the team is that we will use that primary and then go to the backup certificate as needed if a vulnerability is detected. Which of the backup? So the last question is. Any plans to have a Cloudflare car. So this is something that we've talked about over the years. And Dina comes back. I'll let her speak to kind of the current thinking here. It's it's an area where from a where we focus perspective, it doesn't necessarily make the most sense to do. We like having a diversity of partners to work with. Going and becoming a certificate authority and going through that process of getting validated and being added to a trust. Or it takes many years and there's not a whole lot of advantage for us doing it when we have great partners that we can work with and that have those APIs that they've adopted as such as Acme. And so our preference is to spend that engineering time and audit time elsewhere and, and building solutions to diversify across them. I don't think I'll rule that out. It may be something that we do in the future, but it doesn't make the most sense. If you think about how do we spend our engineering time, where do we prioritize? It's not something that our customers are asking for or asking about. And so I don't think that you'll see that from us anytime soon. We like to see as we work with and we'll continue to expand that ecosystem. So with that, I will I will wind down here. Thank you so much for joining. Hopefully you enjoyed the session. If you have any questions, please, please reach out with feedback. Dina is excited to hear your thoughts and ideas here. If you're an enterprise customer, you can give that to your customer success manager. We also have the Cloudflare community forums that you can join and post questions on. And we look forward to hearing your feedback there with that. Have a great day. We're betting on the technology for the future, not the technology for the past. So having a broad network, having global companies now running at full enterprise scale gives us great comfort. It's dead clear that no one is innovating in this space as fast as Cloudflare is. With the help of Cloudflare, we were able to add an extra layer of network security controlled by alliance, including WAF, DDoS Cloudflare users can also allow us to keep costs under control and caching and improve speed. Cloudflare has been an amazing partner in the privacy front. They've been willing to be extremely transparent about the data that they are collecting and why they're using it, and they've also been willing to throw those logs away. I think one of our favorite features of Cloudflare has been the worker technology. Our origins can go down and things will continue to operate perfectly. I think having that kind of a safety net provided by Cloudflare goes a long ways. We were able to leverage Cloudflare to save about $250,000 within about a day. The cost savings across the board is is measurable, it's dramatic, and it's something that actually dwarfs the yearly cost of our service. With Cloudflare, it's really amazing to partner with a vendor who's not just providing a great enterprise service, but also helping to move forward the security on the Internet. One of the things we didn't expect to happen is that the majority of traffic coming into our infrastructure would get faster response times, which is incredible. Like Zendesk just got 50% faster for all of those customers around the world because we migrated to Cloudflare. We chose Cloudflare over other existing technology vendors so we could provide a single standard for our global footprint and sharing world class capabilities and bot management and web application firewall. To protect our large public facing digital presence, we ended up building our own fleet of proxy servers such that we can easily lose one and then it wouldn't have a massive effect. But it was very hard to manage because we kept adding more and more machines as we grew. With Cloudflare, we were able to just scrap all of that because Cloudflare now sits in front and does all the work for us. Cloudflare helped us to improve the customer satisfaction. It removed the friction with our customer engagement, its very low maintenance and are very cost effective and are very easy to deploy and it improves the customer experiences big time. And Cloudflare is amazing. Culture is such a relief off it. It's very easy to use its first Cloudflare 20 plays, the first level of defense for us. Cloudflare has given us peace of mind. They've got our backs. Cloudflare has been fantastic. I would definitely recommend Cloudflare. Cloudflare is providing an incredible service to the world right now. Cloudflare has helped save lives through Project Fair Shot. We will forever be grateful for your participation in getting the vaccine to those who need it most in an elegant, efficient and ethical manner. Thank you. and,