🎂 Defending Against Phishing with Security Keys & Implementing FIDO2

Thanks for tuning into Cloudflare TV. I'm Michael Keane. I'm on the Zero Trust team, joined by Evan Johnson, our director of Security Engineering, and David Harnett, our senior director and chief of staff on the product team. And today we're talking about 502 and security keys, hardware, keys as one of the most secure forms of multifactor authentication, and specifically how that helps us defend against phishing and one of the most resistant forms of MFA that that you can pick. And we're lucky enough at Cloudflare to require security keys today on every single login for every employee to every resource that we have. And one of the reasons that's kind of timely and extra topical these days is spent a lot of coverage in various security news outlets and everything of a series of very related social engineering attacks that Cloudflare back in July was lucky enough to block one of the targeted attacks on our company, thanks to the security key requirement. And unfortunately other companies have not been so lucky when they're using other forms of MFA like Time-Based One-Time Passcodes and just other types of MFA that have lower levels of assurance and can ultimately be bypassed or stolen as well. So it's been a long path to get to this point of requiring security keys on every login. Evan, you wrote a blog post as part of Birthday Week of how we got there, and it's been definitely a bit of a journey. Can you can you help share with the audience a little bit about back in 2018 when we were, I think, considering security keys for the first time and prioritizing that project, what was kind of going on through the security team's mind to say, All right, it's time to embark on this on this mission. Yeah. Happy to share. So back in 2018, it wasn't just security keys that we wanted to do. We were really thinking of overhauling our identity and access management systems and tooling and everything. And a big part of that was security keys. We wanted a phishing resistant multifactor off, and previously we were using time based HTTP. So like the Google Authenticator Apps and the Office apps that a lot of people are familiar with. There's also the push apps that also kind of have the same problem where they're not phishing resistant because it doesn't matter if I log in with the password or an attacker logs in with a password, the push goes still to the user's phone and can be tricked into into illegitimately approving a push. So back in 2018, we really wanted to overhaul all of our IAM with to centralize our identity and access management tooling on a single identity provider using Cloudflare access to enforce a lot of our role based access control and authorization and at the same time start to. Move from. The old kind of multifactor auth systems to security keys and security keys we knew were a bet and thing. an initiative that would pay off over time. Back in 2018, the support wasn't very good. I don't think Safari even supported web often on mobile at the time and over the last four years. That was four years ago. We've been security key only for about the last year and a half. Support has gotten really good, a bunch of different there's been a bunch of improvements in the ecosystem and now's a great time for everybody to kind of lean into the web end and security keys, not just the companies at the bleeding edge, but that's kind of what we were thinking back in 2018 and the workstream that I was a part of. Yeah. And so you mentioned like the MFA fatigue issue with push notifications we've set how time based, One time passcodes can be stolen. We know with SIM swapping that SMS MFA is not good either. So we know what's kind of bad about the lower assurance MFA. But what is so good about 502 and about security keys that makes this phishing resistant? Yeah. So all of those were designed for all those kind of other second factor methods were designed as an extra layer in case your password was stolen or in case your password was leaked in some way. They weren't designed to prevent an active attacker who's trying to phish you. And so Fido two was really designed with humans making mistakes in mind. And the main way that it accounts for that is that when you get prompted for your second factor, there is a there's kind of a fancy protocol. It's not too fancy. It's pretty simple and it's documented in an RFC that you can go read but there's a back and forth between your browser and the and the security key that takes into account the URL of the page you're on. And then on top of that, it bolts in like a cryptographically secure challenge response protocol. So if I go and I touch the button on a phishing page, the challenge response will be different than if I touch the security key on the legitimate site I'm trying to log into. So because of that, there's no way unless the site gets completely taken over, in which case you've got way bigger problems on your hand that you can kind of impersonate the website that people are trying to log into. It's it's kind of hard to explain with words and no visuals and everything, but there are a ton of great blog posts and visuals that outline how Fido two works. And the main thing is that accounts for that URL and the host that you're you're trying to log into as part of the challenge response protocol. Yeah. And so you mentioned back in 2018 as this was all getting planned out, we're centralizing our iam we're starting our own Zero Trust journey with Cloudflare access, our own XNA service. I think we've heard with, with our own customers too. When you are embarking on a Zero Trust journey, it's not about just trying to roll out absolutely everything overnight. It's usually starting with your highest risk users or your highest risk applications, or just you have to start somewhere. So with security, Keys did in our own XNA implementation, did we do we have a particular prioritized starting point with one of our high risk assets? Yeah, we definitely did for security keys but also zero trust we have a zero trust roadmap dot org zero trust roadmap. If you Google that, you'll see like a general. roadmap for zero trust that takes into account some of the low hanging fruit that I definitely recommend getting started on. If you're looking at zero trust, but then for security keys what we did was we really identified actually I think in 2020 we talk about in this blog that got released, there is a high profile company that was compromised due to a phishing attack that the attacker got in, got to an internal admin page that and from there, the entire site kind of was hacked. Around that time we said we're going to roll out security keys. Actually, that same week we said we're going to roll out security keys just for this one application that serves a similar but not quite as powerful purpose in Cloudflare. And that we so it was selective enforcement. That was kind of the first thing that we went out to do, is we gave everybody keys. And then from there we wanted to selectively enforce one site that people had to use their keys on, and that was one to reflect the sensitivity of that site and then to give some give our employees a training ground for them to learn how to use their keys, learn, get familiar with them, and have somewhere where they have to use it, because this is more of a human behavior problem than a technical problem. At that point, you're just retraining people how they're used to working, and people have to get in the groove of using these new tools and technologies. And it's not hard. It's just people need a little push and and education. And at some point you have to introduce requirement. And that was a really easy onramp to do that, if that makes sense. I'm not sure if that was completely coherent. Oh yeah. And it definitely makes sense to start enforcing it on some of those highest risk assets, especially because I think some of these other organizations that have not been so fortunate in recent times that are getting that are falling victim to these attacks. And they're the attackers bypassing MFA. They're not just getting that internal access. And it's not just their own companies data that's at risk, but that of their customers, too. We're seeing several examples of customer data also getting leaked. So the stakes definitely are high there. So then as time goes on, we required it on that one particular app, both for the human behavior aspect and just for the sensitivity of that app. Then I think you said in your blog, February 2021 is when all of a sudden it was time to really amp things up and start requiring it everywhere. So what was kind of going on that made y'all make that call? Yeah. So back in February 2021, our employees started getting kind of scary phone calls from somebody from a social engineer, and we were really worried about that. And we had done a ton of groundwork to kind of put us in a position to require security keys we had. We were requiring them on a few internal sites and we were about to do a bigger rollout, but not to 100%. And at that time, because of this, the severity of and we were worried that our our employees would become a victim of this. We decided, let's flip the switch and roll this out everywhere and enforce. the... Enforce security keys for all of our internal apps. And luckily at that time, it was mostly just flipping a switch for us and making sure that everybody at the company was aware and everybody was bought in that there may be a disruption like some people may be forced to learn about those keys pretty quickly. But one we had really top down kind of support for that kind of disruption, if there was any, that it was important to protect the company and our customers. And then to we had done a lot of the legwork and the hard work already distributing the keys. A lot of people were using their keys. It was really just the kind of last stand people who were using totp still and hadn't adjusted. And so we gave them the push they needed and forced it a little bit under duress. But we were prepared, which is good. Now, Tony, you have a cool graph from around that time of when we started requiring it, where all of the hard keys is the main method being used and totp or the time based. One time passcodes go basically to zero. It was it was close to zero, but not exactly zero, I think because a little bit of nuances with some recovery methods. Can you speak to that a bit of what was going on there? Yeah. So if you look really close at that graph in the blogpost, you'll see Web often is way up here. And then the line for TOTP says it's like one or two. You can tell it's not zero, but it's very close to zero. And so the graph, it's a little bit mislabeled. It should say web athan and other not web often and totp, but the what's happening and why that totp or other line isn't at zero is any time somebody completely locked out of their account or they lose their keys or they've typed in their password ten times in a row and they get their account gets locked down because they have the wrong password for whatever reason. They just woke up that morning and can't remember their password then. Then we have an alternate recovery method which is offline and doesn't involve doesn't involve like a username password too, if a combination. So that's why that line isn't quite at zero and but it's very close and we've got a pretty large workforce. So that's happening pretty much every day or at least a couple of times a week. But it's certainly like by and large, everybody when they're logging in with a password. And a second factor, it's a security key. The second factor is. Yeah. And I think one could technically avoid that, right? If you have multiple keys given to each employee. And here at Cloudflare, we have one that we just it's a nice little tiny one that just stays in our laptop at all times. And we have another one that we use mainly for mobile, but I think we could then use that one for the recovery method too, right? Yes. So we give every employee two security keys. It's really important that one, if you're going to support Web, often as a SaaS provider, support multiple keys, you need to support more than one key and then to every employee needs to have a backup in case they lose their key. Actually, funny story today we were talking about this in our security team chat and somebody Rory, who is in the London office of Cloudflare mentioned that he was giving a presentation about security keys to new employees and he kind of picked his up and held it in front of the camera. And when he did that, he dropped it in a cup of kombucha he had. And so that security key is now toast probably. I mean, maybe he can it'll always be sticky, at least, but he has a backup just in case. And so you need more than one because life happens. That's great. And so the security keys by themselves and then giving every employee multiple. It's not just about the security keys. We also need software to help us roll them out. And I think identity providers provide great support for a lot of SaaS apps, but we use Cloudflare Access, our XNA service. I like to think of Zero Trust Network Access or XNA as kind of this aggregation layer across all types of sources, whether it's SaaS, self-hosted, XBMC, whatever. It kind of helps you reach the longer tail of resources and even put MFA in front of other apps that otherwise would be hard to put in front of like some super old legacy stuff. So what are some of the harder to reach types of resources at Cloudflare that we've been able to put security keys in front of? Thanks to say. Yeah. S.h. for sure non HTTP protocols is a tough one because your regular identity where proxies and that work over HTTP need support for other protocols to otherwise this is all for not if your infrastructure layer isn't enforcing security keys to then that layer is vulnerable to attacks. But we solved that. We solved that problem throughout this with Cloudflare DX and our tunnels product. So we use tunnels over to also enforce security keys for our non HTTP protocols. But then the other one is internal sites. If you're trying to centralize your access control, it's really hard on internal sites unless you go through like you could have hundreds of internal sites and make them all speak. Saml So you need some layer in front of it. And for us that layer is our identity aware proxy Cloudflare Zero Trust product and And. We enforce security keys in that as well as a bunch of roll based access control stuff. But otherwise, if we didn't have that, we would be making every site speak SAML or ORTH or some complex thing instead of just validating a JWT. So it's if you have a bunch of internal sites, you need some type of product like that. So it's always cool to hear our own story of Cloudflare of how we rolled these out. I'm sure it was a lot of work, but I'm sure when things like the recent attacks happen that the team is glad that y'all prioritized that work when you did. And I'm sure it's going to continue to pay dividends. For sure. let's say David, switch over to your blog from this morning. Speaking of what we've enjoyed sharing our own story about how we've rolled out security keys and sharing best practices and advice with other organizations. We announced a new collaboration with Yubico this morning who provides the Uber keys or the security hardware keys that we use here at Cloudflare and a special offer for current customers. So can you explain how that offer works? Absolutely. And hi, Michael. Hi, Evan. Good to be on this broadcast with you. And hi to anybody who's out there watching. Great to hear the story, Evan, of what we went through ourselves. So what is the offer today? So we use yubikeys and we've been working with Yubico, the maker of yubikeys, or quite a while now. And today we just launched an offer. It's actually two offers for any Cloudflare customer. So the first offer is that anyone who is a customer of ours, anybody at all of any of our products can get Yubico keys for a really low price, as low as $10 a key, and they can just order them straight from our dashboard. So if you're a customer, you'll have our product dashboard, You can log in there, you can see this offer and you can get yubikeys sent straight out to you at a really good price. That's the first offer. The second offer is if you're a larger organization that has 500 employees or more, you can get 50% off the first year of a subscription service that Yubico offers. It's called Yubi Enterprise Subscription, and that's got unlimited amount of keys that you can get with that offer and you can sign up on our dashboard for either offer. You'll get an email from Yubico and then you can decide that you want the keys offer or you want the subscription offer. So that's what we're doing today. The other part of this is our support organizations are both trained across Keys and Cloudflare zero Trust, so that if you're a customer who needs help, you can call the Yubico support org, you can call ARS, you can chat with us, you can send us email and we will help you set up yubikeys with our Zero Trust service. The whole thing here about making this easy is that, and Evan has been mentioning this a few times. We have a Zero Trust service Cloudflare Zero Trust that has multiple products within their all integrated easy to use and a key is just one part of that, but it integrates really well and very easily so. So our customers now can get zero trust. I can get zero trust with keys. So that's what's happened today. We've already had thousands of customers sign up. I'm watching us right now during this during this broadcast. So it seems to be going well and we're really excited to see what happens going forward. Very cool. And, you know, Evan talked about why security keys are so resistant to phishing compared to lower assurance methods of MFA out there. So if everyone if we know that security keys are so good and that we should be using them, I guess I'm talking with Yubico and other customers about this. What are the challenges that organizations have in embarking on a journey to roll out security keys? What is holding folks back? Yeah, there are a number of things and we have lots of conversations with our customers and in talking to you because they're having lots of conversations with their customers and the problems or the issues fall into a few areas. The first one is why do I need a key? Why do I need a security key? People hear that you can have a really nice Uber key and you can touch the key to show that you're a human. And then it's it's phishing resistant, but they don't know. How do I attach that to an application? How do I attach that to a zero trust service if I just buy a key? What do I do with it? How do I set it up? So setting it up with our zero trust service is something that we're making really easy. So Zero Trust, of course, is about not trusting anything or anyone. Every time you, you access an application. So every time you access, you have to give us signals. And those signals are things like your location, your password, your key, whether you have a key, whether your, your device has malware on it, these are all signals and we go yes, yes, yes, yes, grant access. So key is one of those. And the first the first thing is, okay, why do I need a key? And if I have a key, what do I do with this? And the second thing is for smaller companies, whether and organizations, whether you're a small NGO, whether you have a blog, whether you're a small business, it can be expensive to buy security keys for everybody in the company. And it's not just one, as Evan was saying, you don't want to drop it into your kombucha or I'm in Seattle into my Starbucks coffee because you may run into problems. So you're going to need to. And Yubico recommends at least two. We have we have to I know I have actually three here as an employee of Cloudflare. So you need multiple. It can get expensive. So our special offer for $10 a key is way lower than what you could get otherwise. And, and it was great that Yubico partnered with us to give our customers this offer. So that's the second thing. The third thing is really about your employees getting use to using them. And Evan spoke about this. You don't want to have 43 apps and then all of a sudden you log in the next day and they're all locked down. You don't know what this key is and why would I use it? So, so, so rolling it out carefully with app by app and maybe employee group by employee group testing it in the IT department first testing with different types of apps, different types of geographies, etc., etc. is really important. So again, Evan's blog and other blogs that we've written and also our web pages that we now have available now that we've launched this partnership, will help customers through that. We also have our debt developer documentation. So our support organizations have come together. They've also updated all of our developer docs related to setting up keys to show the journey and make it easy. And really we want people to learn from what we've been through. And of course there's lots of other customers that are going through the same process and we don't want them to fall into the same into issues as they set up. So that's really, those are really the reasons why it may seem very obvious to implement keys, but why there are some obstacles and why everybody in the whole world is not using them right now. We're hoping to fix that and we're hoping at least our millions of customers will now have access to keys and know how to set them up really easily. Nice. And it almost sounds silly to even mention, but it seems like even just the logistics of getting these keys to everyone, especially in a remote environment these days, I think a few years ago we handed out all these keys in person at a full company in-person meeting and there's kind of everybody setting them up right then and there, which is cool and efficient. But I think part of this offer can even help with that, too, right? Just the pure logistics of shipping them to everyone. Yeah, absolutely. So the first offer where you can get keys for this special $10 price they'll ship them to the to the account on file so that'll generally be an administrator it department who will then distribute them to employees. But the subscription offer you can have a subscription offer with Yubico where they work with you to obviously securely take your employees information and then they send them out to employees. These were, of course, in a world where you can't just say, okay, at the next all hands meeting, we're going to hand them out to everybody in the office. You have to be able to get them to California and to Texas and to Washington. I think that's where the three of us are. Or to London or anywhere in the world. You have to be able to get them there safely. If an employee has a problem, what do you do if you've sent out a yubikey that doesn't have the right interface, what do you do? And Yubico has a subscription service that helps with all of this. Also helps with. So I have these keys. They work. I want to upgrade to a to a newer key. How do I do that? They help with the whole thing. So certainly this offer is primarily about making things straightforward and easy and affordable. I hope it helps lower the barrier to entry here even further. I love how we're sharing our own story of how we've done this at Cloudflare and now with this cool collaboration and these offers, I hope other organizations will embark on this journey with us and eventually get to their own place where they also are requiring security keys at all their other resources. So for those watching, if you are interested in the offer and you're not one of the thousands that already have claimed it, go ahead. I think in the description of this TV segment, we put the link to either go learn more about phishing resistant MFA and how you can selectively enforce strong authentication using a service like Cloudflare Access or XNA Service. You can go learn more about that or you can claim the offer directly within the dashboard. And if you're more familiar, maybe with our application services and haven't yet tried out our Zero Trust product can also sign up for a free account of that for up to 50 users. So I think that's about all we had. Thanks again, Evan and David, and we'll go ahead and cut to commercial, but thanks for tuning in and enjoy the rest of Birthday Week. Q2 customers love our ability to innovate quickly and deliver what was traditionally very static old school banking applications into more modern technologies and integrations in the marketplace. Our customers are banks, credit unions and fintech clients. We really focus on providing end to end solutions for the account holders throughout the course of their financial lives. Our availability is super important to our customers here too. To even one minute of downtime can have an economic impact. So we specifically chose Cloudflare for their Magic Transit Solution because it offered a way for us to displace legacy vendors and the layer three and layer four space, but also extend layer seven services to some of our cloud native products and more traditional infrastructure. I think one of the things that separates Magic Transit from some of the legacy solutions that we had leveraged in the past is the ability to manage policy from a single place. What I love about Cloudflare for Q2 is it allows us to get ten times the coverage as we previously could with legacy technologies. I think one of the many benefits of Cloudflare is just how quickly the solution allows us to scale and deliver solutions across multiple platforms. My favorite thing about Cloudflare is that they keep developing solutions and products. They keep providing solutions. They keep investing in technology. They keep making the Internet safe. Security has always been looked at as a friction point, but I feel like with Cloudflare it doesn't need to be. You can deliver innovation quickly, but also have those innovative solutions be secure.