Web3 Security

Hi, Derek. How are you today? I'm good. Hello, Wesley. And hello, everyone. Welcome to a special segment on Cloudflare TV in celebration of Security Awareness Month. And just in general, Wesley and I will be talking about Web3, security around it, and how Cloudflare is paying a big part of it. We'll get into all the details here. But first, we can actually introduce ourselves. Wesley, you want to go first? Absolutely. So hi, everybody. It's nice to meet you. My name is Wesley Evans. I lead product for Cloudflare's research team and all of our strategic blockchain efforts. So broadly speaking, I handle post-quantum cryptography, next -generation Airnet standards, everything to do with the Web3 space. It's a really fun day job. I get to toy around with the next generation of the underlying future of the Internet. Awesome. And my name is Derek Chamorro. I am one of the heads of security here at Cloudflare. I focus a lot on research and development on our infrastructure platforms and everything that Wesley and his team are building kind of rides on the hardware and the network that we kind of help build. So again, I like to play around with a lot of stuff. So we get to have a lot of fun and implement a lot of fun concepts. Yeah. So we've been talking about this for a while now. Wesley, I think I've known you for like over a year now, and we kind of got in on this blockchain discussion a while ago. And I'm glad that we're finally actually taking the time to be able to do this because I feel like we've been talking about this a lot, but we can actually, you know, actually really talk about this and let everybody kind of know what we're doing. Exactly. You know, it's not like Cloudflare has only been talking about this for the last year or two. I mean, we've been working in this space. And before Cloudflare Research was called Cloudflare Research, it was Cloudflare Crypto. I mean, we've been doing fundamental cryptography well since the early days of the company. And then we've been working specifically on Ethereum and IPFS since about 2017. So we've actually been involved in the broader blockchain space for quite a long period of time. We were doing crypto before it was cool. Yeah, exactly. Exactly. And so I wanted to set this conversation kind of like be free flowing and whatnot. So we're just pinging each other questions. But I think the first is I want to ask you in your interpretation, what is Web3? That's a great question. And to really answer like what is Web3, I think we need to go back in time a little bit and figure out what was Web1 and what it was Web2 and what might be Web3, right? Because Web3 is still a relatively nebulous term. There are a lot of different components of it. And the most simple way to think about it is go all the way back in time to the early 90s, the late 80s, past the days of ARPANET, right? Like we finally got to the point where we have ISPs. Some houses have the notion of a dial-up modem. And we're really getting to the idea of the World Wide Web, like this first conglomeration of lots of ASNs playing together to be able to create a global fabric that people could actually put stuff on, right? And it was this idea of sort of unidirectional communication. You could sit down. You could write some HTTP code. You might be able to write some CSS. And you go to put up a web page, right? You know, a really static piece of content in many ways, right? It was hosted on a single server. You'd put up an IP address. You'd get the DNS for it. You'd hook it up to a conventional host name. And you'd be able to find it in your browser, right? And that's how you got the notion of a blog. That's how you got the notion of a web page. That's how a lot of even some very early day e-commerce started, where you basically have a catalog online, right? You call up John Deere and say, hey, I saw this tractor on your website. And I want to, you know, buy it and have it shipped to my house. But you'd still have to do the whole transaction over the phone, right? Because it was really very, very static, like geocities, ISP-run web pages. It wasn't until the dot-com bubble, and even a little bit after that, we really started to get into this web 2.0 paradigm, right? We moved from a read-only world to a read-write world. We have the notion of chat really starts becoming present, right? The idea of holding stateful content, the idea of being able to dynamically edit something. And web 2.0 really came into vogue in the mid, let's call it aughts to early 2010s with this idea of mobile, right? The idea that you have a mobile application, and suddenly the Internet shifts its location in the physical world from your big hulking desktop computer or your relatively bulky MacBook at the time, or whatever other laptop you were using, to your phone. And suddenly, your world gets a lot bigger and a lot smaller. You have access to a huge amount of information in a much smaller form factor, and you're using it all the time. So where maybe web one was destination viewing, and you go to it and do a thing. It's like, I'm going to go surf the net. Web 2.0 is really this transition to the Internet's part of my daily life. I'm using it for email. I'm using it for my texting, because most of my text messages are on iMessage. We can think of iMessage as web 2.0. That's not a conventional text protocol that's going out over the airwaves. It's a data transit system that's encrypted now. That's web 2 .0, right? This underlying fabric. The problem with web 2.0 that I think many people in web 3.0 would say, is that this explosion of interactivity, this explosion of apps create an explosion of data. You put so much of yourself out there, and so much of your own personal content, that we sort of fell into the original sin of the Internet, which is advertising. And more than just having advertising content, we made it super easy for people to sacrifice a little bit or a lot of bit of privacy to have highly targeted, highly specified ads that would profile you and offer you content without any potential rumination back towards yourself. And so, web 3.0 really gets this idea of read, write, own. That's sort of what I like to think of it as. You have this third evolution of, okay, we went to a world that had highly centralized, but very fast and efficient data centers that could translate content, that could modify, that could do all sorts of stuff, but you don't own it. To a world in which we have a highly distributed system, still is performant, still is fast, but across more networks, across more machines, and critically, that we understand the underlying value of who has what data. And we tokenize that data, and we put that data on a blockchain, sort of an immutable ledger of transactions. So, not only do we know what happened, but we have a history of what happened, who owned what, and why someone did something could be traced back. And that's really, really important. I think we're going to talk about that a lot, but I want to sort of leave that there and throw questions at me for it, because it's a lot of- Absolutely. I think you brought up some really good points, because we've gone from this static notion of a web to this responsive notion of a web where we live nowadays, and the future is taking a greater ownership. And I feel like kind of that leads into this conversation of, this is kind of like Cloudflare's story of how we've helped the web evolve. And so, with going with that, what are Cloudflare's contributions to this Web 3.0 concept and evolution? Sure, totally. So, when we think about Web 3.0, I like to say that oftentimes when you see this conversation in the media, or you see this conversation even in industry, it's put as an either or, right? Web 2.0 or Web 3.0, the old way versus the new way. I like to say is that we've just had more tools. Web 3.0 gives us new tools in the toolkit to do things that we couldn't do before. And that's really actually what we're starting to see. If the early days of Web 3.0 were around this idea of decentralization and creating applications like Uniswap, SushiSwap, other types of decentralized exchanges, places like Dapper Labs with CryptoKitties and other types of play to earn games, these were small experiments that have gotten really big. But at their core, they basically are an HTTP application still sitting on the Internet that have changed two fundamental protocols. The first is they shifted their storage from a highly centralized AWS S3 bucket or something in GCP to a much more distributed file system like the Interplanetary File System, otherwise known as IPFS. So, we offer an IPFS gateway. It's a mechanism for retrieving content in a very conventional manner and serving it up to the browser just like the browser would expect to have it. But the file system is totally different than what's conventionally stored. The second is this sort of compute layer, right? So, storage to serve up a static web page and then the compute layer to actually do something with it. Lots of people are changing out their compute system for Ethereum or other types of virtual machines that are blockchain-based. So, we see a lot of people doing Polygon now. We see a lot of people doing Ethereum. We've been in the Ethereum space for quite a long time and we have a massive Ethereum gateway. And what's cool about our Ethereum gateway is that it's edge-based. So, our goal with all of our services at Cloudflare is how do we run them as close to our users as possible? We have the entire planet within 50 milliseconds of a Cloudflare data center. That's astounding. That's faster than the blink of an eye. And by getting this infrastructure for compute and storage as close to our users as possible, it makes Web3 really, really fast. Because what users have come to expect, if we look at the innovation curve, your traditional user expects to have an Internet that's fast, responsive, and most of all, secure, right? And we're going to get to the security piece of here in a second. What's been challenging for Web3 over the last five years is that we can do secure, but we weren't able to do fast and we weren't able to do responsive. What we're really building with the Ethereum gateway and the IPFS gateway is getting that fast and responsive back into the network. But more so, it's not just building discrete one-off products. We're baking the idea of how Web3 works into all of Cloudflare services. So from Cloudflare R2, which now works with IPFS Atlas, which is our new intelligent indexing layer. R2 is also our object storage system. Cloudflare Pages, so that you can use Cloudflare Pages as the backbone for building any type of static website you want. But instead of using KV, you use IPFS. We want to change the idea that you have to either build a Web2 website or build a Web3 website to you're going to build something that does something cool. And we're going to give you the tools to build it however you want. And you can come to Cloudflare to build whatever you want, not just a Web2 website or a quote -unquote Web3 website. So the idea is we're kind of making it transparent to the end user. At the end of the day, it could be built on IPFS or it could be built on KV or whatever, but that transition is primarily going to end up being transparent. Exactly. We want to have a transparent system that developers can build what they want to build on it inside of their constraints, not the system's constraints. And that their end users at the end of the day still get what they expect. Rapid times on load, really, really secure performance with very, very high reliability. Love it. Love it. Love it. We mentioned the secure thing, though, and I think this is really what's so fascinating to me. We've spent decades thinking about Internet security, not just at Cloudflare, but the ITF level and other bodies really developing a robust standards-based framework for how we think about security, how we do vulnerability assessments, how we do zero-day. Talk about the idea of the types of security concerns when it comes to a blockchain. This seems like a whole different world for the industry. Yeah, absolutely. And we didn't want to make this as a primer of what blockchain is because there's plenty of documentation on it. But blockchain, especially when it comes to security, we have to think of some simple components. We think about the hash functions for integrity verification. We think of cryptography, primarily asymmetric cryptography, because that kind of verifies ownership. This concept of a network or peer-to-peer network, which is distributed and decentralized. And then consensus mechanisms around this election process of who has the right to be able to write a block. And then the validation process as well. But when we group it together, we think about user, node, and network. And so when we look at the user, blockchain uses public key cryptography to protect the authenticity, integrity of transactions and blocks. But the security of these keys is dependent on the security of the user's private keys. You create this private and public key pair. The public key is actually used to authenticate what the private key has been able to validate and sign. But anyone with that private key can decrypt messages and generate valid digital signatures on the user's behalf. So failing to protect those private keys is one of the most common security errors that users make. And then we have a variety of different solutions in order to be able to mitigate that. So we primarily look at private key protection as being the main primer for user security. So we look at hardware wallets. Trezor, different types of hardware-based wallets have been popular in days where you can actually have this concept of cold storage. You take your keys offline. In order to be able to authenticate a transaction or validate a transaction, you actually plug this hardware wallet into your computer. These are the physical devices that store private keys and perform a lot of these typical cryptographic operations to prevent the keys from actually being leaked. We have very expensive hardware security modules. We use them internally. They're quite expensive, but they're essentially just these large hardware wallets that are built into computer chips with these physical type of protections. And then we have this concept of paper wallets. You can write a private key on a piece of paper and physically secure it, put it in a safe. Or I would say, give it to your grandma because she always finds a way to being able to protect that. Don't write your passphrase down on a piece of paper. Don't write your passphrase down on a piece of paper unless you know your grandma is going to be able to keep it. I know enough friends that have done that accidentally from 2012, when they had a lot of Bitcoin, and find their passphrase. And they're very, very upset about that. Oh, yeah. Watch one of the last Joe Grand videos on hacking hardware wallets for somebody who left a passphrase loaded on a phone that they couldn't unlock. That's actually a really good video. So user security, malware. I mean, anybody that's susceptible to malware gets infected with a piece of malware on their computer. Again, they write their private key somewhere, or they have it stored somewhere locally, then that malware might be able to retrieve it. And the concept of lack of app updating. If you don't update your wallet, or if you don't update your crypto software, then the software is dependent on the machines that it actually runs. So there's patches that are constantly set out. There's updates for vulnerabilities that are there. So if you're using a vulnerable form of software, you're more susceptible to the concept of key leakage. Then we look at the node as a whole. There are nodes that are responsible for validation, kind of suffer the same problems that computers have, or user computers have. There's a concept of shared vulnerabilities. The nodes are users. And if this node software is not updated, then it can be susceptible to that as well. And it's not just the software too, it's the infrastructure that node runs on. Because if you don't control the infrastructure yourself, or trust the vendor that's running the node, right? Absolutely. Absolutely. The concept of having insecure APIs. The poor configs. Blockchain runs on this concept of peer-to -peer network on top of the Internet, or on top of another internal network. So if the underlying network infrastructure used by this blockchain is not built to be redundant and built to be resilient, then you're obviously going to have failures within your blockchain itself. So this concept of lack of bandwidth is a possibility, physical attacks on the blockchain, or lack of proper segmentation can lead to problems that might arise within your network. So those are some of the bigger security concerns. And noting the features and noting the services that we offer, you know, how is Cloudflare contributing to a more secure Web3 in that aspect? Yeah, I mean, I think that's a really interesting question. It's, you know, I love to talk about this one too, particularly, like, let's just start with the individual person, right? Because we talk about the individual behaviors users can do. And that ties so well into the Zero Trust story that Cloudflare is helping to build, right? This idea of, you know, in the good old days of corporate networks, you'd build a castle in a moat with a VPN. And suddenly, if you could get over that drawbridge, you'd have access to everything. I think we have a new paradigm emerging for that, but in the Web3 space, but in a very different way. The idea that you have the idea of DAOs, right? And distributed autonomous organizations that are groups of people that organically forming together to build software, right? It's like taking open source on steroids, you're gathering together on Discord. And sometimes these projects go from zero to managing 100 million, a billion, $10 billion of assets under management, but yet still don't have a common identity verification mechanism, right? They don't have an IDP system. So like, how do you deal with that? And what I love about how some of our customers have been using like Cloudflare Zero Trust products is like, you can put, we should also explain what Zero Trust is. This idea that you don't have to trust every single thing a user does, that you should give least permission to access to resources. And you should do that at the Internet level instead of at the application level. Yeah. And I love the fact that you actually brought up the castle and moat concept is because that's kind of like how Zero Trust has been presented. Previously, it's like, as soon as you get into the castle, you have access to everything. And by default, that's how a lot of organizations have developed it. You're inside your network. You can access whatever you want, but we're living in this world where your network is not necessarily in a physical location or single location. It's distributed all over the place. You're using cloud services, you're using software services that are hosted by different providers. So this idea that your data is always in one location is not that anymore. You have this lateral movement, sometimes it's side to side. So should you trust that network now that you have distributed all over the place, you have to kind of put the series of checks and balances in place in order to be able to validate that user is still that user and it's still authorized to be able to access that even if it's laterally or located right next to that data, right? Exactly. And I think for web three native companies that have come out in the last three or four years, there's no centralized idea of corporate network. There's no concept of having a box somewhere, right? And so what we see with customers using Zero Trust and even organizations and people that may not trust each other enough to even establish a common IDP amongst each other, is that you can deploy our Zero Trust solutions and use your own permission to system of like, we're going to allow these 20 Gmail addresses, or we're going to use these specific social identifiers in our system to create and manage resource access, which is just a really profound concept because the amount of groups that have had a breach in a system or a social engineering attack, I mean, that's one of the biggest threats to the web three space in general is not just, okay, we're finding a vulnerability in the code or we're finding a vulnerability in the consensus mechanism. It's, I accidentally trusted the wrong person or I clicked on the wrong link. Yeah. Yeah. Yeah. No, I completely agree. And it's the, the irony is that, you know, I feel like during the, this concept of this inception of what blockchain was meant to be, it's like, oh, there's a lot of anonymity associated with that. But as we start getting into more like, you know, decentralized financial applications, you have to kind of like know who your creditor is and you have to have some form of trust built into that. And so now we're getting to this point where we want to be able to leverage this technology, but how we do it in an identifiable way that we know that we can validate transactions that are actually accurate, you know, eliminate the idea of, you know, of fraudulent transactions. So you have to implement this kind of layer identity in order to be able to kind of have that level of security of knowing that this transaction is actually valid. Totally. I mean, it's interesting that we go there because it comes up, it brings up a trust and safety perspective because you hit the nail on the head, right? There's this idea that blockchains are inherently anonymous and private. And that might have been true 10 years ago when no one knew how to do transaction tracing or do KYC analysis or KYT analysis. Nowadays, the IRS, the FBI, they love blockchains. They're great. You can see the entire transaction history. And once you can figure out who that wallet is associated with, which we've gotten very good at, you have a whole transaction history. So I think there really is a, there's an old narrative, right? About how blockchain technology and cryptocurrencies have been fundamentally utilized as mechanisms for the dark web and mechanisms for doing illicit transactions. When in reality, as we see this industry really start shifting into a world of regulated finance, it's actually a much more transparent institution and potentially a much more secure institution. I mean, we were talking before the show started about how JP Morgan was doing work with Polygon earlier this week and did the first DeFi transaction in Southeast Asia. I mean, JP Morgan's not a small bank. Not at all. And it's huge for the industry because technically, banking industry has kind of been so against moving towards blockchain. But now that we have these concepts of these new technologies that allow them to be able to kind of put identities behind these transactions in a secure fashion, we're not necessarily giving up the original primitives of what blockchain was established for. If not, we're just adding on to them to allow it to modernize traditional banking systems that have not wanted to modernize in that sense, right? Right. And as we start putting significant amounts of assets behind this, I mean, there was a case by, I believe, BCG. Let me find it in front of me here. It's, what's the stat on it? It's the Boston Consulting Group believes that over $13 trillion worth of assets could be tokenized by 2030 of illiquid assets. So we're not talking about even cryptocurrencies that are like highly liquid, right? We're talking about things like oil and gas. We're talking about things like art and real estate could be tokenized and put on blockchains. I mean, that's a massive industry shift. That is huge. That is huge. All right. So how are we protecting the user? How is Cloudflare protecting the user? Yeah. So I think one aspect, right, is we talked about Zero Trust. Zero Trust gives us this framework that allows us to do identity and known resource control, right? Which is important because decentralized applications, blockchains all rely on code. And so having a known understanding of who's interacting with your development environment for being able to access GitHub, for being able to push changes, for being able to access repos is really important, right? That's just bar one. And it also gives you the capability to do things like two-factor authentication to prevent one random engineer Vlad accidentally clicks on a phishing link and suddenly exposes your entire GitHub repo to the Internet. That's bad, right? 100%. Second thing, I think, is we get to node operations, right? So I love talking about node operations, particularly because running your own node is hard. It's not like running GoEthereum. It's just like hunky-dory process where you sit down, you download it, and you get it working. It's complicated. Keeping these systems functioning and operating correctly is really hard. And so this is where I think we've seen the industry really shift in the last five years to manage node operations with us, with groups like Inferior, Alchemy, or others that have shifted your node operation into a managed state, right? So you don't come to us to run a node. You come to us to get APIs. We run the nodes. What's great about having a group like Cloudflare do that is threefold. One, you get the reliability of our network, right? You get the reliability of our engineers who spend every waking moment with a 24-7 on-call rotation supporting, keeping that functional. The second is around patching, right? Now, when we see issues and we get CVE announcements, we patch our network and infrastructure exceptionally fast. I mean, many times we're actually the ones to detect zero-day errors before anyone else. And so because we can see these things, we can patch our networks and then patch our nodes faster. So it reduces your vulnerability surface. And then critically, what's nice about the way our node operations are set up is that they're inside extremely robust physical security parameters. And we sort of talked about that at the beginning. If you can get access to the node itself, you can do weird things to it. And so you have confidence with going with someone that has a node inside of a server, inside of a rack, that's inside of a hardened cage behind four or many more layers of defense. It's hard to get physical access to that, right? And that's really where node security, I think, is so important because you want to preserve that consensus mechanism and you want to preserve every way of keeping fairness in there. And you don't want to have any doubts about it. Yeah. And we can also test that too. It's like we've worked over the last few years as far as hardening our server platform in order to be able to ensure that nobody can... If somebody were to break into one of our data centers and pull out a server, we can say with great confidence, they can't point anything off that because we encrypt all three states of traffic. We have detection mechanisms in place in order to be able to determine what something is actually opening up a server chassis or whatnot. So we're pretty confident knowing that our servers are pretty resilient. And so any of the information that's actually stored on there is either going to be secured because it's powered up or in a non -powered upstate. People are not going to be able to remove anything from that. And getting back to the user, we made an announcement a few weeks ago that we recently made a partnership with Yubico. They're known as hardware keys primarily for multi-factor authentication or passwordless authentication, but people have used them before in order to be able to do secure storage. So this concept of creating these public-private key pair that's primarily used for authenticating a lot of blockchain transactions, that's something that people could leverage as well, right? Absolutely. And if you're not using a hardware key, stop what you're doing after this call, go get one. I will say I was a skeptic of this. I remember when I started working at Snapchat a long time ago and they said we were going to roll out hardware keys and I moaned about it. And it's sort of gotten to this place where I have a bunch of them now and I keep them on me all the time. And I feel a lot better about it because I know I can't be attacked and I know that my data can't be breached. And it's not just my own corporate stuff. I secure as many of my own personal accounts as I possibly can with hardware keys. It just gives me that level of confidence that even if I was to make a mistake, right? Because I'm not infallible, I can click on the wrong link. Spirit phishing attacks are really good nowadays. And we thwarted one not that long ago, what we call smishing attack. It was an SMS case phishing attack because of the fact that we had this Zero Trust infrastructure in place as well as the use of hardware keys. So anybody that would to click on a link, they were not really going to get anywhere because they still have to go through this authenticated phase where you have to physically have this key in place in order to be able to press on it and get that one-time passphrase in order to be able to access our network. So it saved us so many times. We're seeing the industry really skate to this too. I mean, we're talking about Yubico right now, right? And the way that that works. But we're seeing hardware keys get baked into everything else. I mean, often the most, the transaction I do most often with the two-factor nowadays that involves a hardware token doesn't actually involve a physical key anymore. It uses my iPhone and uses Face ID. We're seeing the idea of privacy attestation tokens for passwordless systems become really well -developed by our team and Apple and others. We're seeing the concept of how you can use facial recognition and facial technology on Android, iPhone, others to create the same types of cryptographic transactions that YubiKey provides. And YubiKey will still be a huge part of that mixture. But for the average everyday user, you walk into your phone using Face ID or Android's version of it, this is going to be just as easy. Absolutely. Absolutely. Yeah. I mean, you have a FIPS certified chipset in there that allows you to kind of do that one-time passphrase that's required for making a lot of the transactions and making them a lot easier. So little do we know that we have some kind of features that are baked into the phones that we own. We have a couple of minutes left. Well, you want to talk about the network because I think this is really important. Oh, yeah. Yeah, absolutely. And more specifically, we talk about key management. We talk about cryptographic keys a lot. Talk about how Cloudflare thinks about that at a network scale, right? Because we might, in many ways, we are in the DDoS business, but we're really in the security business. And we're often in the key management business because of TLS. Yeah. Oh, man. We are making some huge changes. And we have some internal projects that we're going to be talking about here shortly on kind of how we're revolutionizing the key management space. So you go to some providers and they have this concept of centralization of keys or custodial services of keys, but it doesn't really work well for Cloudflare. I mean, we're located all over the place, 275 cities, 100 plus countries. The way our network is built is meant to be kind of very portable, but be all over the place. So we're in the process of building something that actually meets that kind of level of scale, but at the same time, also still keeps that data as close to the user as possible. So a lot of our future service is going to be powered with this technology. I feel like our partnership is only going to get more stronger too, because you're going to be reliant on some of these key management practices that we are building in-house as well. And if anything, it's going to power some new services that I feel like your team is kind of like in the works, so kind of baking on. Oh, yeah. We're really excited about the HSMs that we're rolling out right now. I think that as you look toward the future of what... I wrote a blog post about four or five months ago, talking about how Cloudflare is going to be building an exceptionally robust Ethereum validator network as a test bed to help accelerate the Ethereum network, help provide additional security to Ethereum, because in proof of stake, having as many validator nodes as possible helps defend the entire network against attacks. We really take that quite seriously as a security company. But we're really thinking quite actively about what it means to hold keys, what those represent, what assets represent as they're associated with keys, and how to do that in a really, really secure way. And I think HSMs are a really important part of that mixture. So we have 60 seconds left. What are your final thoughts? My final thoughts, we should make this a series, man, because we have so many... We've talked about so many things and so many different concepts, and I want to get more into the crypto side of it too, because we have zero-knowledge proofs, we have PQC or post-quantum crypto that we can definitely start talking about. So I feel like there's so many new things that we should be talking about. And we can bring all our friends along too, because it's not like it's just you and me here at Cloudflare talking about this. I mean, our research team at Cloudflare and all of our academic collaborators and industry collaborators, I mean, we're on the vanguard of post -quantum cryptography. I wrote about that a couple of weeks ago, how we deployed post-quantum crypto for Cloudflare tunnels, how we're working on that for client side. So there's a lot of really cool stuff, particularly with zero -knowledge proofs and zero-knowledge proof middle boxes, which I'm very, very excited about coming down the pipe for the possible future of how to do privacy preserving Zero Trust solutions. Derek, it's been an absolute pleasure. Thank you. You're doing an awesome job. All right. Cheers, man. Bye.