Cloudflare for Teams: Our Story: Episode 1

Okay, welcome everybody to the first episode of a new series called Cloudflare for Teams, Our Story.

This is a new series where we will talk about the story that Cloudflare, our own company, has gone through with Cloudflare for Teams.

Today I have two great guests.

I've got Evan Johnson, who's the head of product security for Cloudflare, and I've got Joe Sullivan, our chief security officer.

Before I hand over to them to introduce themselves, I just want to do a quick overview of what is Cloudflare for Teams.

So Cloudflare for Teams is our new platform for user security, so for all the teams in an organization.

And what we offer is we offer secure access to any application and the Internet from any device anywhere.

And we have two major products. The first one is Cloudflare Access, which provides secure access to any of your applications anywhere in the world, to any of your workers, remote workers from any device.

And then we have Cloudflare Gateway, which is our secure web gateway product.

Together, they make up for Cloudflare for Teams.

So with that, Evan, our head of product security, just hand over to you to introduce yourself, and then we'll go to Joe.

Yeah, I'm Evan. I work on our security team and kind of at the intersection of software engineering and security, making sure that all of the things that our company, that all of our engineering teams at the company build are secure.

And it's a lot of fun, especially at a company like Cloudflare, building things like Teams and 1.1 and all of these different products that I'm sure you've seen if you're watching Cloudflare TV.

Awesome.

Joe? Yeah, hi. My name is Joe Sullivan, and my title here is chief security officer.

That means I get up every day thinking about how do we make sure that our customers' data flowing through our systems is secure.

I joined in summer of 2018, just a little over two years ago, and it's been an exciting journey so far, focused on really building out a world-class security organization, helping us get a bunch of security certifications, making sure that Evan and his team have the resources to support all of our product development and all the other fun things that security teams get to do.

Awesome. Two great jobs. Great working with you.

And with that, I'll just introduce myself really quickly. So I'm David Harnett, and I joined Cloudflare in January of this year with the acquisition of a company called S2 Systems, where I was CEO of that company.

And we do remote browser isolation, which is going to be part of Cloudflare for Teams gateway product.

So that's me. So let's get into the questions. Let me start off with Joe. Can you tell us, before we go into the story of our company with Cloudflare for Teams, can you tell us a little bit about your story, how you got to Cloudflare for Teams, what you did, or how you got to Cloudflare and what you did before you joined the company?

Sure. I came into security probably not the traditional way. I actually started working in Internet security when I was a federal prosecutor.

I think it was 1998, I was in the US Attorney's Office in Las Vegas.

And one of the senior managers in the office said, Joe, we'd like you to start doing the high tech cases.

You're one of the few prosecutors who has a computer on his desk.

And so here's a special extra computer, and we're going to set you up with a bunch of training.

And I loved it out of the gate. So I started doing more and more tech cases.

And at the end of 1999, I moved back to Northern California. And we started a high tech unit in the US Attorney's Office here.

And I just was 100 % on high tech cases, doing every kind of Internet crime you can imagine.

Not doing the crime, but prosecuting the crime.

And then after doing that for a while, I was recruited over to eBay.

eBay at the time was kind of one of the hottest e-commerce sites in 2002.

And spent a few years there working on what we called trust and safety.

Believe it or not, back in the early days of the Internet, you would order something on eBay, and then you would put cash in an envelope and mail it and hope that the goods would show up.

And shockingly, it worked.

But over time, the bad guys found the Internet too, and we had to adapt and really get better at fighting fraud and other abuse.

Then I went to Facebook.

I was the second CSO at Facebook. Was there from when we were smaller than MySpace to over a billion users, built out their security organization.

Then I went to Uber, built out their security organization over two and a half years.

And then the most recent stop was I joined Cloudflare in 2018, and I've been working on security here.

The thing I've loved about the different places I've worked is just the breadth of security issues we have to deal with.

A place like Cloudflare, if there's a security issue on the Internet, it's our problem.

And I actually love that. Wow. What an amazing background. And now you're helping the Internet be more secure, which is awesome.

Evan, can you beat that one? No, a lot less crime in my background.

So I started working at a startup called LastPass, which was a password manager prior to working at Cloudflare.

And I was a software engineer there.

And I really studied security in school and was really interested in security and was breaking a lot of systems and stuff legally, of course, on the side to learn about security.

But then I started software engineering a lot more and working at LastPass as a software engineer.

And when I made the jump over to Cloudflare, I actually thought I was joining a product and engineering team.

And little did I know, I was joining a security team. And I had never been a part of a security team before coming to Cloudflare.

But a lot of people had told me what I was doing was AppSec.

And so I just kept breaking things here and helping engineers fix them.

And that's basically the entire story. So software engineering, breaking stuff, fixing it, and repeating.

Awesome. Great to break things, especially if you're going to fix them afterwards.

Yeah. I love LastPass.

I'm actually running it on my machine here. It's amazing when you have something like that, that you realize that you've got like 215 passwords that otherwise were not secure before you started using a password manager like that.

Yeah. Great product. Great product. Cool. Well, it's a privilege to be speaking with both of you with such amazing personal stories about getting to Cloudflare and what you've done before.

So let me start asking about our story by talking about user security and team security in Cloudflare.

So before we kind of go into our story of how we've used teams, talk to me a little bit, maybe Joe, since we started with you first, about how do you look at security for users and for teams, some of the issues you're dealing with, the pain points you're dealing with.

Let's just start there.

Yeah. Like at the highest, simplest level, when I think about protecting a company and the assets of a company, I think that there are two ways that the company can get attacked.

It's going to be directly through our code.

We're a company that has a lot of code facing the Internet, or it's going to be through our people.

And so I try and think about how we frame up our defenses in both of those areas.

From a code standpoint, we've got the product security team.

We've got a bug bounty program. We hire external pen testers. We have a software development lifecycle.

We have a bunch of things that are built in to really make sure that that code that's facing the Internet is secure.

But then you take the other side, our employees.

More and more of the larger compromises in recent years have been compromises of an individual at the company, and then abusing the access that an employee has.

I mean, for all of us, especially now with our remote workforces in the world of COVID-19, every company has employees scattered around who need access to the internal systems of the company.

And so the fundamental question is, is the person who's accessing our systems right now, the employee that we trust, that we vetted, that we put through a background check, that we interviewed, or is it somebody else?

And if it's the employee, are they doing the things that they should be doing?

Are they doing something that they're not?

And so what we've seen in security is that more and more of the attacks on the code are getting harder and harder, and more and more of the successful attacks are on the people side.

And so when I think about securing the company, I'm thinking about both of those.

And like most companies right now, I'm probably spending more on securing the authorized access.

No question about it, if you get 100 CISOs in a room together and you pull them on their budget for 2020, they're spending more on identity and access management than anything else right now.

And part of the reason is because of COVID, but even more importantly is the fact that we're all operating in these hybrid environments.

The world has evolved away from that, okay, the whole company's behind a VPN and a single network, and you're all operating from a few offices, and so your connections aren't even going through the open Internet.

That's not the world anymore.

We're all dealing with authentications to SaaS apps and on-prem apps and our own data centers and someone else's data centers, and you just got to pull together a comprehensive strategy.

And so when I got to Cloudflare, I spent my first month in 2018 thinking about what are the priorities for us as a security team?

And we picked two things. One of the two was identity and access management. And in hindsight, we haven't changed a thing.

That's still our top priority. Awesome.

I can see with the next question, how Cloudflare for Teams fits right with all of those issues that you're facing.

But Evan, same question to you, the problems that you are seeing and the issues you're trying to address with user security and team security at Cloudflare.

Yeah. We have at Cloudflare so many internal applications, and every tech company does.

You build internal applications for the people working at the company to be able to operate, onboard new customers, you name it.

And there's so many different engineering teams and so many different things happening that it's tough to really centralize your authentication and authorization systems in an internal environment, more so than SaaS.

Because in SaaS, you purchase a SaaS product and it may integrate directly with your identity provider, but that's not really the case with internal applications.

So the big value to me with Cloudflare access specifically as part of Cloudflare for Teams is being able to take all of these internal applications that don't know who I am necessarily, or maybe 30 different applications made by 30 different engineering teams, put one shield in front of it, which is Cloudflare access and unify the authentication and authorization across all of our internal applications.

So it's a really powerful security tool compared to the legacy VPN that we always talk about in the sense that a VPN doesn't know who I am.

It does know who I am, but the IP packet going across the wire doesn't say Evan at Cloudflare.com.

It just has my source destination IP address.

And so Cloudflare access lets us put one unified shield up in front of our internal applications and then tie requests at an individual HTTP request level to a specific employee, which is just super powerful.

And so that's kind of the big value that I saw in Cloudflare access early on when we were making this big IAM push.

That's great. You're both talking about something that we now refer to as Zero Trust and providing application authentication, application by application, whether it's an internal app or extending that to SaaS apps.

So this is an industry issue that's going on right now, but also a big opportunity to transform away from the order that a network had once been to a Zero Trust model.

So I think the next thing I'm going to ask you, just so that we understand where we are in these questions, I'm going to ask you, where are we now on the journey?

Tell us a little bit about the journey of access, which you both mentioned, but also gateway, which we use in Cloudflare in our offices.

So I'd love to hear where are we?

And then what I'll ask after that is what still needs to be done and what's your vision for the future?

So on the story of, tell us a little bit about what we've done with Cloudflare for Teams with access and gateway.

I'll just throw it out to both of you since you work so closely together and you can both respond.

Yeah, let me start. The journey for us really started in 2018. And that summer of 2018, I think Evan and I and a few other people on the security team, we decided that identity and access management was going to be our top priority.

And we looked at how things were set up at the company. And a lot of the most sensitive things were only accessed through VPN.

And we looked at the employee experience with that.

And one thing that everybody realizes is that VPN can be very painful.

I was reminded of that. I had a customer call this morning and we were interacting with a financial institution.

And so there were a bunch of people from Cloudflare on the call and a bunch of people from our customer and none of them were on video and we were all on video.

The reason they were not on video was because they had been instructed to stay off video because it would overtax their VPN.

And so they as a company during this time of remote work can't see each other, can't have the same kind of connectivity with each other.

But for us, we have access in place.

And so as a result, we're not worried about overtaxing a VPN. An amazing story is, in fact, about a month into the COVID-19 lockdown, we actually shut off VPN for 750 of our employees, more than half the company.

We took away their VPN access.

As everybody else was trying to figure out how to get their VPN to work and handle the workload of their company, we were busy turning it off.

And that's because of the vision we set forth in 2018 of how do we get to a place where we can have good usability and good employee experience, but also good security.

And so a little short story. In the fall of 2018, we had a company meeting where the whole company was going to get together and talk about strategy.

And we as a security team had the courage to ask our CEO, could we have a half hour slot in front of the whole company?

No other team was asking this. And we said, we're kind of brand new.

We want to present something to the company about how we want to go forward and the employee experience.

And our CEO said, sure. And so we started brainstorming about how we were going to talk about identity and access management.

We had this vision.

And so we decided that we wanted to smash VPNs on stage in front of the whole company.

And so we started, so we thought we would go on eBay and buy a bunch of old VPNs and then bring them up on stage and smash them.

But when we started talking to the venue, they said, you can't really smash old technology equipment like that.

There's workplace issues, there's chemicals, et cetera. And so we were like, what are we going to do?

And we came up with the idea of commissioning a pinata of a VPN instead.

And so we got this giant pinata of a VPN custom made in the Mission District in San Francisco by a lovely family that makes pinatas for birthday parties and the like.

And then we asked our CEO to step up on stage and smash the VPN in front of the whole company while blindfolded.

And didn't know what he was going to say, didn't know how it was going to go in front of the whole company.

But suffice to say, the VPN got smashed actively and vigorously by more than one person because so many people hate VPNs.

And everybody wanted to smash the VPN and everybody got to enjoy the candy that came out of our VPN on that day.

And it set us on that journey in front of the whole company of like, let's create a better experience.

And it was us, the security team partnering with the access team that existed even back then.

And it was working on kind of the first generation of the product.

And then I would just say like that set us in a great path as a security team that like we have this mantra, use Cloudflare to secure Cloudflare.

And there's so many great examples of where we used access.

Evan could talk about how there was a time when we needed to issue short-term certificates to people who only needed limited access to production.

And so Evan's team partnered with the access team and we built short-term certificates at Cloudflare and then built them into the access product.

There was the other example of, you know, I said, where we shut off 750 people from VPN because of access.

And then like just in the last week, we've all been talking about how the, you know, what was in the news last week, the Twitter hack, and the idea that internal tools could be exposed to the Internet.

With access, we can have the granularity to say our support tools can only be accessed by employees using a hard key.

You know, we as a company, we use a hard key second factor for most sensitive environments and only want someone who uses a hard key to be able to access them.

With a tool like access, we can actually say, okay, it's okay to act like our identity and access provider doesn't give us the granularity to be able to pick which applications require a hard key and which don't, but access can give that to us.

So just so many stories of how like access for us is kind of like a, I don't know, it's like a Swiss army knife solution in the identity and access place to give us the control and security we need.

Awesome that I now know the story of the legendary VPN banana.

That's great. That's great. And Evan, same question that Joe, Joe had some really great stories there.

What else would you add to that about our journey to where we are now and how we've gotten here on Cloudflare teams?

Yeah, I, I think it really, I remember long ago when access was first being built and, and kind of tested by what's now product, our product strategy team, that there was only two websites behind access.

And one was, and it was called edge off at the time.

And one was Grafana, our metrics, and one was Jira, maybe one of our, one of our other internal tools.

That's pretty ubiquitous. And the, and from there, starting with just two applications, people kind of immediately saw the benefit of, of having, not having to get on the VPN to access these internal tools and hugely beneficial.

And Sam, who's the product manager for Cloudflare for teams.

He has this great story that we went and did a talk and he has this great story about how the paved roads in Texas and how, I don't know if any of this is true, but he's such a Texas, Texas guy that he has this great story about how we introduced these applications that people used to have to get on the VPN to access.

And then just like how, when Texas was trying to pave their roads, they just made one mile of the road paved so people could see the benefit of it.

We did the same thing with access.

We'd put one application on access, people would see the benefit of it.

And then five more people would ask for another mile of paved road near their house.

They would ask for the next five internal applications to get onboarded to access.

And so we eventually, we, we've gotten hundreds of applications that are internal behind access and it's just it, it's kind of the ubiquitous way that things are supposed to be set up now.

And it was kind of a monumental change that took a long, long time.

But I feel like we're there now where basically every web application internally is behind access.

And then we even have, like Joe mentioned, we, we have many of our production machines accessible through our short-lived certificates SSH feature, as well as through the Cloudflare D tunnel through protected by Cloudflare access.

And so it's, it's just really interesting to be able to tie the identity from your identity provider with all of the strong security benefits of a, of a security key and your multi-factor authentication that you enforce in one place all the way to your production environment so seamlessly.

And there's still more to do.

You, one of the things was what else do we have to do? One of your questions was what else do we have to do?

And we've, we're really working on making usage of that more, more common in production.

There's still some improvements to be kind of, to be made on our internal systems to make it so that it, it is the best, the paved road all the way to production.

So it is much easier and we're slowly getting there.

And, but today it's possible people do use it and really like it.

It's really nice to be able to not have to use the VPN to make it all the way to production.

If, if that's your use case. Yeah, that's great. It's great to hear.

Sorry, Joe. It's great to hear the, the analogy of the Swiss army knife, because the, the value that we can provide over time with access of not only the identity provider, which of course is integrating different identity providers.

Like we use, we use Google and we also use Okta.

Just being able to do that integration is great.

And then if you bring on contractors and you can use LinkedIn for the contractors or get help for the contractors, but then also being able to do a device posture check.

Of course, we've, we've done the Tanium deal. We also, and that's rolled out now to our customers.

We've got a Azure active directory.

We're doing more in our roadmap. So it really becomes that Swiss army knife of authentication that includes identity.

So it's really, it's really nice to hear that way.

But Joe, back to you, you were, you were going to extend the story there a bit.

Yeah. I wanted to touch on gateway a little bit as well, because we've you know, one of the nice things for us as a security team at a security company is we get to be the first customer for our products.

We get to dog food them so to speak, but we also get to give a lot of feedback and help the products get better.

And so, like I mentioned the short term certificate but turning to gateway short story last October as a security leader, I wanted to demonstrate to the company, the importance of identity and access.

And that second factor being a hard key.

So October cybersecurity awareness month, and a lot of companies do their kind of internal employee education during that month.

So I decided without telling my team that we were going to do a red team.

And we were going to have, and basically what a red team is, is you don't tell the security team and you hire some people outside the company to attack the company.

And for this particular red team, we hired some folks to send phishing emails targeting specific teams.

And our goal was to show the company the difference in security of different second factors of authentication.

We all know the challenges around SMS as a second factor, hopefully, and recognize that SMS is just not good enough to be a true second factor of authentication.

And so a lot of companies use soft tokens or approval buttons or things like that.

And don't go all the way to the hard key. Granted, getting hard keys to work in every context is hard.

You do need a product like access really.

But our company was on that journey of moving, we've gotten rid of the SMS second factor, but we still had a few other second factors.

And so we had the red team attack.

And the red team attack was, it highlighted some things that we were good at and highlighted things that we needed to improve at, which is what you want from a red team.

And in this case, one thing that the company was good at was reporting it.

So some of the employees who got the phishing emails reported them to security.

And my team responded like they should. And the cool thing is we had the gateway product in place and we were beta testing it.

So a lot of times when you get a phishing URL, one of the things you want to do is what we call black hole that URL or block it or just eliminate it so that none of your employees, if they click on it, can actually go to it.

And gateway gave my team the ability to quickly basically block that URL from any further employees accessing it from our machines.

And so that was a part of the incident response that I thought went really well.

Number one, our employees reported it to us.

And number two, our team was able to stop the bleeding.

Of course, not everybody reported it and one person clicked on it.

And that gave us the opportunity to show how their second factor could be stolen using evil engine X.

And so we were able to accomplish our goal with the red team of showing that like hard key second factor is really important and continuing to invest in identity and access management was really important.

But we were also able to highlight in front of the company the value of another part of the team's product.

That's great.

That's great. It's we've a minute to go. So Evan, you touched on some of the work still to be done.

So maybe we can take that up in a future episode if I'm lucky enough to get you back again.

But thank you very much to you, Evan, and to you, Joe.

I think we obviously have a long way to go, but we're getting there pretty quickly.

And it's great to have a security organization in the company that works so closely with the product team.

So as we're building, we're testing, we're all contributing and designing together.

So I just want to thank you both very much.

And thanks to everybody watching. And this concludes our first episode of Cloudflare for Teams, our story.

Thank you.