🔒 Latest from Product and Engineering

All right, welcome to the latest from product and engineering and coffee. The team was just sharing our favorite notes on coffee. I'm Usman Muzaffar, Cloudflare's head of engineering. And I'm Jen Taylor, Cloudflare's head of product. And we are thrilled, as always, to welcome some of our colleagues, Nick Sullivan and Wes Evans are with us. Nick, why don't you introduce yourself? I am Nick Sullivan. I'm lead Cloudflare research. Hi, I'm Wesley and I am the PM for the Cloudflare research team. Excellent. And it's so great to see both of you. Nick, you are one of the Cloudflare old timer by any metric. And one of the people who helped explain to me when I got to Cloudflare, like, how does all this work together? Where's all this fit? So it's great fun to be able to talk to you about this. Jen, we had another fancy Cloudflare week this week. What was that all about? So this is actually one that I've been oddly super excited for us to do. And so this we're finishing up today, our Privacy and Compliance Week. And both Nick and Wesley contributed quite a bit to that. Wesley, what is Privacy and Compliance Week and why did we do it? Sure. Thanks, Jen. Privacy and Compliance Week, in a nutshell, is Cloudflare's recommitment to our core values around trust with our users and our customers at its core. And we're showing that trust in a lot of different ways. One is we've launched the data localization suite, which is this really amazing new assemblage of products that help our customers keep in control their data where they want to and need to. This is particularly important for our customers in the EU. But beyond just our initial product offerings, we've really taken a step forward and doing a lot of really amazing work in the protocol space for helping to build a better Internet. You know, we've launched things like oblivious DNS this week. We've really talked about some of the next generation protocols coming out like ECH and even some of the really bleeding edge work in authentication technology like opaque or even post quantum cryptography down the line for security. So Privacy and Compliance Week has been a really tour de force in saying Cloudflare is here to help build a better Internet. And we're doing that at the root with privacy and compliance at the core for trust, for not just our own customers, not just the users that end up touching Cloudflare, but really the whole Internet. That's really great. I mean, the challenge of just even the first bullet there, right, like the data localization part of the trick here is the Internet is global. Like it doesn't know about boundaries. Right. So like and Cloudflare is trying to make sure that we meet our users where they are. So we've got data centers all over the world. And yet now we've got this entirely digital artifact, like the Internet and how we process traffic needs to pay attention to these very important rules. And so like Wes and Nick, either of you, what are some of the things we had to pay attention to as we started to think about data localization? You mentioned GDPR and the EU. Just briefly, what are some of the other things you have to pay attention to as you think about data localization? Sure. I mean, I can take the next chime in too. It's funny that you mentioned GDPR because, you know, it's interesting when we talk about data localization, there's actually nothing in GDPR that says data localization. It can be construed as having that. But what it really comes down to is a series of local and international court cases in the EU and other places in Europe and a growing majority of the world that have said, you know, we really care about our users' data. We care about where it's stored. And so when we think about this, we have to think about where our pops are located, like you said, Usman. But we also think about two really important things, where we're decrypting traffic and where we're storing cryptographic keys. So, you know, we launched two specific products around those GeoKey Manager for where we store cryptographic keys and then regional services for where we actually do that traffic decryption. Additionally to that, you know, we really have to think about metadata and where we're storing our logs. So edge log delivery was also a big part of this. And then to sort of encompass all of it, we've been doing a lot of work over the last couple of years with our serverless platform workers and durable objects, particularly, which we launched during birthday week earlier this year. Even though it's an unlimited beta, we've already added more functionality to it with jurisdiction restrictions to durable objects so that we can, you know, let's say you wanted to build an authentication system, for instance, and you only wanted to store user data in the EU. It's super easy to add a jurisdiction restriction to a durable object to achieve that goal. So you can be thinking from the very beginning about how you want to be doing data localization and data sovereignty, however that manifests itself for you as a business, regardless of your geography. Well, and that's the thing I really came to appreciate about the way that the team approached this challenge, right? I mean, if I'm operating a web application and I'm servicing Brazil and I'm servicing Germany and I'm servicing India and I'm servicing Japan, you know, I'm dealing with a global audience, but I'm dealing with different regulations in each one of those segments that are constantly changing. And so one of the things I really appreciated about the way that the team approached the challenge here was really thinking about how do we put the controls in the visibility into the hands of the customers and to give them the opportunity to set them as they need it, and to have more fidelity and fine-grained control over kind of specific functions and specific regions, while at the same time giving that the same kind of unified control plane that's really been kind of core to our offering. No, exactly. I feel like, Nick, this is one of the first projects that I needed your help with, like it's a good three or four years ago now, the GeoKey Manager and some of the challenges there. Like, just a little refresher for the audience, like what were we trying to solve and what was not obvious, like what was working and yet what was, we still wanted to give, like Jen said, this other knob to our users to give them the ability to say this is where I want my keys to live. So what did we have to solve technically to give them that ability? Yeah, this is a fun story. This is one of the earlier challenges we had to solve in the days of Cloudflare, six or so years ago. Well, we had a bunch of customers who were in the financial services industry, and some of the regulations that they have limit where cryptographic keys can be stored for any third-party service that they use. And so when they came to us and they're like, Cloudflare, we're really excited to use all your services. We think your caching, WAF, acceleration, all of these things are fantastic. We need them for our banks' websites, but our regulations and our industry don't allow us to share our keys with you. What can you do? And so this was one of the first times that- I mean, like as an engineer, it's kind of like saying, yeah, I need you to put a lock here. But by the way, you also need to be able to open it, but I'm not going to give you the key. So what can you do? It's just like it almost sounds like an unreasonable request right off the bat. But go ahead. Let's hear the rest of the story. What are we doing? And at that time, everybody was using this one tool for the job, OpenSSL. So every different service had OpenSSL. It was this nice library that allowed you to do TLS, enable SSL-powered sites. And the way OpenSSL was built was that the keys were there with the servers, and you couldn't separate the keys from the servers. So some smart folks on the engineering team looked at how SSL TLS worked down in the inner guts and saw, hey, there's a place where we can actually - We only need the key in one place. And can we extract that? Can we put it somewhere else? And so we built something called Keyless SSL, where we kind of stopped OpenSSL in its tracks right at the point where it was going to use the key. And then we connect out to a server that's held somewhere else. And we trialed this with some pretty large banks several years ago. And so they could run a key server in their infrastructure and keep control of their keys and fulfill all these regulatory requirements. And we were able to do everything that we wanted to do to help their site be fast and secure. It was kind of this light bulb moment when we discovered that we could do this. And we built that out. And over the years, we discovered that banks weren't the only customers that would want to do this. There's other customers who were just concerned about where our data centers were around the world or which data centers had the highest level of security. And they wanted some configurability about this. And so we actually repurposed Keyless SSL as a technology and built Geo Key Manager, which effectively rather turns Cloudflare into both the key server and the edge server. So we would be running key servers for any one of our customers all around the world in the locations that they were comfortable keeping their keys. And still be able to provide the entire suite of services worldwide. And now this is part of this great suite of localization services. It's so great to see because it was like building on top of technology that was really just keyless. Then like, wait, it's keyless, but we're running the key servers. But we're doing that to give them the ability. And then like fast forward two years later in a world where privacy and localization is so important. Like, hey, we've got the building block for what we need to be able to deliver. So it's really great. You know, one of the other things that you mentioned there, Nick, is like getting in the guts. And that's something I think is uniquely Cloudflare. It's like we're getting into the guts of some of this stuff. One of the things we announced last week or this week, sorry, Wes, is Oblivious DNS. And so let's talk about that for a second. What problem is it that we're trying to solve? I mean, two years ago we announced 1.1.1 and that had privacy in the title of the blog post. So what wasn't quite private enough? And how does Oblivious DNS make things even more private? That's a great question. I think, you know, to take your first part of that, right, you know, we launched 1.1.1. And what was amazing about 1.1 was that we launched the ability to do DOH with it. And then we took that a step further, right? We said, you know, we want you to really trust Cloudflare. So we did this massive audit of how we were using the data inside 1.1.1 to prove that we weren't violating any privacy concerns and that we were a trusted stakeholder, right? Right, right. There's one step beyond that, which is to say, okay, you can trust Cloudflare because we're doing audits. The next is let's go back to the core fundamental principles of the protocols of the Internet. Like you said, it's like get into the guts of it. And let's find a way to bring that trust to the protocol level. So Oblivious DNS is exactly that. So it's DOH, but we basically put a proxy in between the eyeball going to the DNS server. And what this allows us to do is remove the client IP address before it hits the DOH and at the DNS resolver level. So what's great about this is you get all the speed and performance of 1.1 .1. And you get all the privacy benefits of not actually sharing your client IP information with Cloudflare or potentially any other DNS resolver that chooses to run it. Cloudflare is one of the first people to actually launch DOH as a standard. And we've done it in partnership with a lot of great people. You know, we helped co -author the standard with Apple and Fastly. And then for proxy partners, we're launching with Equinix, PCCW, and Surf in the Netherlands. So we're really excited about this ecosystem. And we only see it getting bigger as browser manufacturers start coming online with this and as we continue to expand the proxy networks. I mean, even in the, like, what, three days since we launched it on Tuesday, we've seen a number of proxies pop up in the wild that weren't officially part of our launch network. So there's a lot of excitement and enthusiasm for the standard out here. And we think it really is one of the next core fundamental building blocks of DNF. That's so great. And so let's just make sure that we're clear. Like, so there's so much interesting technology. So when you say DOH, you're pronouncing D-O-H out loud, right? And so that's D-N-S-O-H, which is a different way of how, so normally DNS does not use this web technology. And so part of what we're doing here is also evangelizing that. And that makes it easy for browsers to just, you know, regardless of whatever network setting, like when the browser needs to find out what the IP address is, for example, .com, it is going to just go directly to Cloudflare Resolver. But the problem is, like, we like to use the analogy, it's the phone book of the Internet. It's almost like caller ID is still on, right? Like the people who are running the phone book can still see. Wait, I know, that's Wes's computer that's trying to look up that website. That's kind of interesting. That means Wes is interested in that website. And so that's what we're trying to obfuscate. So that necessarily means that stuff on your side has to be masking that on its way of making the request. And so doesn't this make this a performance problem in here, Nick? How do we address some of the, like, if we put more layers in here, is this going to slow everything down? How do we, I mean, the clock is ticking here. Users are impatient. Yeah, performance is key, especially on the client side. And when we launched DNS over HTTPS on 1.1.1.1, it was such a fast and performant way of doing DNS because our servers for our regular service were designed to be very close to all the people and the users in the world who browse the Internet. And so Firefox saw this and saw that it used DNS over HTTPS, which is an encrypted protocol, which is easy to integrate into a browser, and reached out to us about a potential partnership. And this is how the Firefox Trusted Resolver program started. And the Firefox folks really were concerned because you can't have a browser and have it be slower than the competition because then you'll switch back. There are so many different options for browsers, right? And so, as a trial before actually launching this for all their customers in the US, all the users of Firefox, what they did was they compared connecting over regular DNS versus DNS over HTTPS to Cloudflare. And surprisingly, what they found is that we were faster. And it kind of blows people's minds because DNS is unencrypted, it doesn't have any additional round trips, it doesn't use TCP. It's supposed to have all the things that make it fast to begin with. It's supposed to be really fast, right? And all cacheable. But it turns out that the average DNS server that a user in the US is using is just not that performant. It's either slow to connect to, or the cache isn't warm, or there's other kind of network effects to make it slower. So even with this additional encryption layer, Firefox found that it was faster. And so they enabled it earlier this year for all of their US customers. So if you're using Firefox, you're using Cloudflare's DNS over HTTPS, which has some great privacy benefits. It's awesome. I think one of the taglines, Jen, you like to say is, you know, privacy, security, and performance, pick any three. Totally. It's a great Cloud First story. It goes back to our DNA of like, we don't believe in false dichotomy. Well, the thing that I also love about some of the work that this team is doing is they're taking kind of old concepts, things that we've sort of been like, don't worry about it, that's plenty secure, you know, and really kind of opening them up and giving them a good hard look and thinking about and iterating and envisioning what the future could be. And you guys actually made a big announcement this week around serverless passwords. Like, can we just step back? Like, what is a password? Like, how does it work? And like, what is wrong with it that would mean that we would need a different way of doing them? Sure. Well, a password is a way for a user to prove that who they are, who they say they are when connecting to a website. And it's, they say with authentication, it's something you know, or something you have, or something you are. And the password is something you know, it's something that's short, easy to remember. There are some complexity requirements now where they have to have special characters and whatnot. And you can, you know, store it in a password manager, but it's effectively the what you know. There are new authentication mechanisms coming into play involving who you are, which is like I can prove I can log in with Facebook and that proves that I am who I say I am. And then there's also what you have, which you may be seeing this all the time. This is how you log in with modern phones is you use biometrics, you use your fingerprint or you use your face. And that shows who you are to the device and you can log in. And so as much as these new techniques are something that you have, another thing that you have is, is say a YubiKey, like a little plug where you have a have a chip and you touch it to log in. So these things are coming, coming around. There are ways that you can prove you are who you say you are to a website, but passwords are still going to be around for a long time. There's no way to really completely eliminate it. Not everyone has the latest smartphone. There are computers from 20, 30 years ago still connected to the Internet. So passwords are going to stay with us for a long time. And under the hood, again, this is another let's get into the guts of it. The way that password authentication works online is that you connect to a Web site and you create an encrypted connection with HTTPS and then you just send the password to the server. And this server then can be really complicated and multiple layers of infrastructure. There can be databases. There can be, as we know, building a Web site. There's hundreds of different services that live behind every single Web service that you interact with on a daily basis. And the password has to kind of ping pong around and find its way to the part of the ecosystem that checks it on the infrastructure. And this has led to a number of accidental times where these intermediate pieces of software have logged the password somewhere, put it into a plain text database, and those things have been leaked. And then everyone has to change their password. This has happened even to the largest, most security conscious companies in the world. It happened to Google. It happened to Facebook within the last two years. And so this idea of having to send your password to the server is something that Opaque, which is our announcement that we made this week, reimagines. Really, you don't have to actually send your password to a server. All the server really needs to know is that you can prove that you know the password. And that's actually something different. The proof of knowledge versus telling somebody something is actually very different. So Opaque comes up with something. It uses technology that we sometimes call zero knowledge proofs, which allows you to use some kind of advanced cryptography that, again, we've developed in previous projects here at Cloudflare. Privacy Pass is something we launched several years ago. It does effectively this. It says, OK, can I prove that I've solved a CAPTCHA? And it does so without actually providing information about yourself. So what Opaque does is takes the ideas from Privacy Pass of zero knowledge and applies it to the world of passwords, where when you log in with an Opaque server, instead of sending your password, you send a proof that you know the password. So some mathematical computation on that password that the server can then verify. And the great part about this is it doesn't leave any opportunity for the server to accidentally log this password. There's no loads of pipes and backend infrastructure that has to go through. And so we launched a demo of Opaque, which is an emerging standard at the IETF. And you can you can play around with it. And we think that this is going to be something that helps secure passwords. Although we see the future of authentication being a lot, lots of biometrics and security keys. And there's a great technology that ties us all together called WebAuthn, which we're very strong supporters of. But for the use cases where passwords still exist, we're really excited about Opaque as a potential way to make it more secure. And, you know, all this, like if we if we bubble up from the guts again. So, Wes, back to you, a question around like, so how what how does this stuff get from out of the lab, out of the kitchen? The, you know, the cloud for the research kitchen. And, you know, when when does this start to actually make the broader Internet with the billions of eyeballs connecting to it every second better? Like what what in your like how how how does some of the technology we just talked about, Opaque, oblivious DNS, you know, ECH, all this all this kind of stuff. How does this how does this make its way to the general public? Well, it's a great question. And I am lucky enough to be able to stand on the shoulders of giants and being able to answer because we don't have to solve the question. Bell Labs did this for us. Xerox PARC did this for us. Right. You know, they really proved out the concept that cloud for research is following, which is make sure that you exist inside of an ecosystem that has a really amazing company and user base that you can help build inside of. Right. You know, we don't have to take these technologies and then go out and find a use case for them. We know what the problems are. Right. We know authentication is a problem. We know encryption is a problem. We know that someday there's going to be a quantum computer that has quantum supremacy. Right. So let's spend time thinking about quantum computing. Let's make sure that we know our use cases. You know, being able to work inside an amazing company like Cloudflare where we can find those use cases is the first and hardest thing to do in some things like this. The second is making sure that we're actually solving real problems and working with real solutions. So Opaque is a great example of we know there's a bleeding edge problem around authentication. We know we can use Cloudflare's existing services infrastructure to actually ship demos and prototype concepts and work with our peers at the IETF and other places to really solve it out. And then we can take it a step further. Let's look at Oblivious DNS. We actually have people on the team that have helped write the spec for Oblivious DNS, like Chris Wood, that have worked with our partners at Apple and other places to say, hey, we know the spec is really good. Okay, let's go deploy it now. You know, we spent the last three years working on it. We know what the problems are. We've solved them. We have the respect and trust of all of our peers in the broader industry to say, hey, this is important. Let's go do it. And then we have the mass and scale of Cloudflare behind us as a company to be able to say, hey, we think this is a really important thing to do. Let's all go do it together. I think that's a really important thing. There's no magic formula that I can sit down and say, I've checked off these seven boxes. I can ship this now and put this in the hands of a different team. It all comes down to relationships, making sure we've got technical maturity, and making sure that we're solving real user problems. Just because we're doing research doesn't mean we're not solving problems. We're just solving problems on a different time scale. Yeah, that's great. Yeah, it's really cool to see this stuff come out and then actually solve customer problems and move there. One of the things that you mentioned, Wes, is quantum supremacy. One of the things junior engineers ask me is, how do you get up to speed? And I was like, honestly, if I've learned one skill over 20 years in engineering, it's the confidence to ask questions and just say, I have no idea what that means. Nick, what the heck is quantum supremacy? Even the term is kind of intimidating. And so why do we care about it? What are we doing about it? What does it mean to have post-quantum cryptography? We've got five minutes left. Five minutes? Yeah, I can boil it down. Let's boil down the math that makes my eyes bleed, please. We've got five minutes and 300 seconds. I know you can do it. No, it sounds like science fiction. But if you go back to the very first computers, the first modern style computers in the 1940s, there were these giant machines in these rooms with reels of paper that were kind of spinning really fast. And they had mechanical outcomes that happened. And they could compute, you know, simple mathematics. And then then you had vacuum tubes and then you had microchips. And all this thing happened over a span of really people's lifetimes. There are people who who were adults back when the computer was first invented. So technology moves quickly and things that seem like science fiction now are not going to be science fiction in the future. And one of these things is is quantum computing. This is an idea that is not so old, but it really is. It's kind of like the leap from Newtonian physics to Einsteinian physics, where everyone thinks the world works in one way and all computers are built with ones and zeros. And they can do operations of one light, one light, switch them off, switch them on. There are certain gates that you can you can compute things with. And then people think, yes, that's all of computing. But actually, no. If you go down to the smallest thing, one bit, that doesn't actually represent the real truth of nature. So under in quantum physics, every atom, every molecule, every well, anything subatomic can can exist in multiple states at once. And and this is what quantum computing leverages. Rather than having a bit, you have something called a quantum bit. So it can be a one, it can be a zero or it can be a cloud of possibilities. And it turns out that you can actually make a computer that operates on these quantum bits. And they've they've been building these computers over the last 40 years. There was just this year about that. And this is where the term quantum supremacy comes into play. So the idea is that, yes, you have these computers who can do things with quantum bits. But are they actually better than existing computers in any way at all? And the term quantum supremacy was it's sort of a threshold in which you can have a quantum computer and you can find a problem that it can solve better than any classical computer. OK, got it. So it's like it's actually better than what I've got today. Yeah, that's right. And so that that's that's the threshold that was passed. So Google's Google's quantum AI lab came up with with the computer and they computed a really obscure kind of statistical stamp sampling problem in a way. They simulated they simulated a probability cloud in a way that a classical computer could never do. And and then they said, rubber stamp, we have passed quantum supremacy. But it turns out that, you know, there's another thing that quantum computers are really good at that has nothing to do with statistics and sampling. And that is it really comes out of left field for people who aren't prepared for it. But quantum computers are really good at one specific problem, which is breaking cryptography. Breaking security. And not just breaking it. Honestly, what we should really do is dedicate a Cloudflare TV segment to the rest of this story, because every time you start to explain this to me, I get all excited because I want you to. But we are we are desperately short on time and I don't want I don't want to. But in a nutshell, is it accurate to say it's not that Cloudflare is building a quantum computer, but we're thinking about the algorithms that will be defensive against a kind of a console. Someone suddenly does have a computer which can suddenly start cracking SSL codes. We will be able to like like Wes just alluded to roll out to technology at scale that will defend against that. That's right. And if you want to deep dive on this, there's a segment that airs every several weeks called Master of Computer Science where I interview Scott Aronson, who's one of the top experts in the field way more knowledgeable than me about this and a great explainer so I recommend that one. But, um, yeah, so what so what we're trying to do is, you know, our current cryptography can be completely broken all of our security can be broken by quantum computers. We're trying to find new algorithms and NIST, which is a standards body in the US is leading this process. And we're, we're along for the ride, and it's a really exciting interesting time for cryptographers. I gotta tell you, it's always an exciting and interesting time and I get the opportunity to chat with the two of you. I just, you know, I am in awe of the work that is coming out of the research team. The way that you guys are looking around corners and sort of driving driving standards and driving the broader community to really think and innovate and in particular this week really thinking about that in the eyes of security and privacy so Thank you so much for the work you're doing. Thanks for this amazing week. As always, Usman, time has flied. Flown. Flown. Yeah. Both of you guys for coming again. I love chatting with you and we'll have you back on again soon. Yeah. Awesome. Thank you all. Bye. Bye. Bye.