🔵 Keynote: How Roche is Adopting Zero Trust Principles

Welcome everybody to Cloudflare Connect. I'm John Graham-Cumming, Cloudflare's CTO, and I'm delighted to have with me for the next 30 minutes, Tim Ehrhart, who works for Roche as the global team head of security monitoring and incident response.

Did I get that right, Tim?

You did. Thank you. Does that abbreviate to good to who? I haven't even tried.

Where are you actually based physically? Where am I talking to you today from?

Basel, Switzerland, by the headquarters. All right.

Well, brilliant. Well, thank you for being with us. I'm a little bit further away from you.

I'm sitting here in Lisbon, but at least we're on the same continent.

I think people will probably be familiar with Roche, but it'd be good just to give us a little bit of a brief overview of the company.

Obviously, it's a company that's been around for a long time, what the mission is, and a little bit about your team as security monitoring incident response.

What does that actually involve?

Sure. Roche is a pharmaceutical and diagnostics company trying to bring life-saving medicines and information to patients around the world.

We've been around for over 120 years and we're based here in Basel, Switzerland.

My team is primarily focused on traditional incident response, trying to find the badness and deal with it.

We also cover vulnerability management, think of our team as the firefighters and the fire wardens, trying to find all the risks that we can before they get exploited and then putting out the fires whenever they happen.

Is this a large team of people?

How many people do that? The company size is not very big. It's only about 20 folks, although we're just a small slice of the larger security organization.

Tell me a little bit about your background. You've been doing this kind of work for a while.

Tell me a bit more about that. I noticed you started out as an analyst.

What sort of analytics were you doing? I was an intelligence analyst, starting off with the US Army for a few years.

I've changed roles a few times, but doing the intelligence analyst work for 13 years for the US, based in Germany the whole time.

That actually was a nice transition because you work a lot in technology.

It was a slow transition toward the cybersecurity space, but actually Roche was the first cybersecurity role I've had.

Interesting. Let's talk a little bit about the strategic objectives you have right now.

Obviously, we've been through or we're still going through this pandemic.

How has that strategy had to change over time and where are you taking things today?

A lot of what we have had to consider is round Zero Trust.

We're in a special challenge, of course, as a large company with a lot of legacy, a lot of old systems, and a lot of specialized systems for production and manufacturing.

We've always known that we've had to go in this route a bit, but the pandemic has changed priorities there and how we think about Zero Trust, but really around connectivity generally.

Supporting people in the pandemic has been a challenge to say the least, but we've overcome it.

Prior to March 2020, that kind of time frame, what was the setup of networking within the company?

Was it a VPN style of working or people in the office style?

Roche is pretty diverse. We have people all over the world, nearly every country, and we have offices in almost every country as well.

We have a physical presence, physical networks, and so most people are used to working in the office.

Now we have a large support force, etc., so we also have a lot of people who rarely, if ever, come into the office.

We've had a bit of diversity there.

We have some people who are very fixed in the office or our buildings, for example, people working in laboratories or manufacturing areas.

They're just people that can't do those jobs from their desk at home.

On the other side, we have IT folks and people in administrative roles that can be far more flexible.

But until the pandemic, and even to some extent today, VPN was the classic way to have remote access.

Physical access locally, VPN remote, with not a lot of variation until maybe 2019, 2018.

We've started to shift a little bit, but the pandemic really accelerated a lot of that change.

To go prior to the pandemic, because one of the things I've said to a lot of people is, for me, the pandemic wasn't a sea change.

It was a real accelerant to things that were already happening.

So it sounds like you were already thinking about what the network architecture, the access architecture needed to look like well before 2020.

Yeah, it's something that became painfully obvious in dealing with incidents over the years, and dealing with penetration tests and security audits.

You see the same things coming up again, and you start to really recognize how invaluable it is to just use something like a Zero Trust model.

Now, there's no easy button, there's no easy product to just make that happen, especially not at our scale.

But there are certainly things that we could do to go toward that direction.

And in fact, that's what started us on our journey was we purchased the Cloudflare WAF and some other pieces a couple of years ago.

And the idea there was, how do we make it easy to reach websites with consistent security and anywhere?

And that was a big sea change for us, was just getting that model out there.

Because prior to that, every website we would operate, every service we would operate, either was a cloud service, or we had on internal load balancers, and DMZs, and really traditional models.

And so in 2018, 2019, we started to shift that quite a bit.

But it was slow, it was certainly not easy to shift the tens of thousands of websites that we have.

And let's just talk a little bit about Zero Trust and just sort of define it, because this is one of those terms that I think is easily confused.

Because what is it you're not trusting? It's just such a strange thing, because in fact, you're trying to increase trust and trying to increase your access.

Tell me about what does it mean to you? I think that's a really good question, because if you talk to different people, you do get a different flavor of what they're thinking.

But we tend to think about it as, how many places can I have a choke point or something to identify the person, maybe the device, and what they're doing?

There are a lot of combinations of that. I mean, you can use legacy technologies and do some of that.

But the whole idea is not making an open trust assumption.

If you're internal to the network, or you're on the VPN, it doesn't inherently give you really a lot of trust, or maybe it does, unfortunately.

And so what we want to do is break away from that model where being plugged into the building just magically makes everything available to you, or being on the VPN makes it available to you.

And the obvious benefits are that you're far more resilient to all sorts of attacks.

Whether it's someone abusing the system, now there's an audit trail or some controls in place.

Or from an instant responder, it gives us a lot more options and gives us a lot more confidence.

And I think once we saw people in the pandemic working from home more and not being on the VPN, I was always happy to say, if there was a compromise or a problem on a laptop, and it happened inside the company, we've got a lot of controls, we've got a lot of things, but it happens.

But that attacker, that malware, that issue is inside this big flat network.

And there are all sorts of risks that come with that. I can mitigate a lot of them, but it's still challenging.

But if, on the other hand, you work at home, you don't have a VPN regularly, and you generally work through a web browser, even if you have the worst ransomware strain, you're not really going to impact our company.

And that's the huge change for us, is looking at the risks differently, where we used to look at, if you're inside the company network, if you're on the VPN, that's the safe place.

And now we say, actually, I would much rather my laptop of every employee to be working way outside of the company network, and only access what it needs through a secure mechanism.

It really shifts the way we think about security boundaries.

It's interesting, isn't it? Because I mean, I certainly know my career, and for a long time, there was a sense that you were inside the walls of the castle, and you were safe inside.

And then as the Internet got used for some things, I remember Hotmail coming along, I remember Salesforce coming along, there was this need to poke a hole in the wall, to get out to the Internet.

But the Internet was a scary place. There's all sorts of threats and worries on the Internet.

And I feel like there's a parallel with what's happened with cloud computing, where people were like, oh, I'm not going to, I'm much better in my data center with my cloud, my servers, and I control everything.

And suddenly, everything's in AWS and GCP, and Azure, people got comfortable with it.

And it feels like the next thing is, well, look, the network, we're all using the Internet all the time.

Isn't that what the new network looks like?

If you buy that argument, how do you control that new world?

It sounds like you sort of hinted at this through the end user, the end points, and all that kind of stuff.

But perhaps talk me through the thinking about that.

How do you think about securing this world where we're all on the Internet all the time?

Well, I think it's important to think, to break that model of thinking, castle, moat, internal and external.

But it changes how we prioritize what we do. In the past, as you said, you have this barrier, you have a firewall, you have maybe great web filtering, you have control of your DNS, you've got a lot of controls, but they're kind of rigid, and they're not super flexible, and they pose a lot of other problems.

And of course, it's all or nothing at that point. But what we're learning is that if we embrace this a little more, and we think differently, and I think your analogy with cloud services is great here, is the data is more important and where it's processed.

Those are the things that I care about.

So identity, data, and its processing unit, what device it's on at any given time.

It's not the network, it's not the transit, it's not usually the physical location.

And because we start thinking about that, even in our space and incident response, we care about now the security of the device much more than we used to.

And we've always cared about it, but we would often think about the network barrier.

And now we say, how do I really make that a secure device? How do I make sure things don't happen on it?

Or if they do, I can detect them. And if I can't detect them, or if I do detect them, how do I make sure it doesn't get worse?

And so all of this really comes together in a kind of a simpler model.

It's just a shift, I think, in mindset.

So we don't think, for example, in the cloud of, do we put it behind the firewall?

We usually think, okay, it's in the cloud somewhere, but by default, that database or that service isn't available.

So we can use an identity to access it over many different ways.

And we're shifting that to do that for all sorts of things.

And so today, we have a laptop, we can secure it well. And then we can say, in the cloud, your identity is what gives you access to that resource.

Or on our internal infrastructure, we use Cloudflare access very heavily.

And we can say, okay, we're going to give you access, but only this person, and only when they're in this country, or only when they're on a device.

And that just makes life a lot easier in the end.

We're more resilient, we get access.

It's a great experience, but it takes a really big mindset shift. Yeah, it's interesting.

So you mentioned the location of the user, which I think is a really interesting one.

Once you move away from the location being the office, you get this incredible granularity, right?

Which is, I'm sitting and talking to you on my corporate laptop.

I obviously use Cloudflare access all the time, that's how I do everything at work.

I have a hardware key for second factor authentication, because we've decided that's the standard of Cloudflare, I can't use an authenticator or something like that, I have to use a hardware key.

And also, I can be country restricted.

So if I suddenly pick this laptop up, fly to, I don't know, scary country, Cloudflare can have configured, no, he doesn't get access from that country.

And I think that's one of the great benefits of turning what was a physical situation, just like with the physical machine, the cloud transformation into, okay, well, now it's a software defined world.

Yeah, it's much more flexible than it used to be.

I think when we would use VPN, and we still do, of course, but you're kind of in and out, and you lose the sense of location when you do the VPN, for example, it's kind of restricting.

And now with all these more modern tools, as long as we are accepting of this new way of thinking, we can be much more flexible.

And so Cloudflare's example, if we want to change something in our access policy, we will click a button and change this globally.

If I want to block a country, or I want to add a security header, for us, it's totally shifted from dealing with, for example, an individual owner of an application or an individual component over a web proxy or another service to say, actually, I can control a lot in the sense of identity here in one place.

And so that's really changed a lot for us.

And it's just incredibly flexible how we can respond to something.

And also, you can do things like, presumably, you can incorporate device integrity as well.

So if I've modified the device and be like, no, I'm sorry, you've installed some software that was disallowed, therefore, your access is now dropped.

Yeah, again, it's an issue or a good example of flexibility. It's not yes or no for the whole device.

It comes down to the user and the device and the other contexts around it.

So there are options we didn't normally have in the past.

And so there's work behind that to understand device health and safety or these vocational boundaries or other restrictions.

But the flexibility is there, and it's much easier to implement today than it used to be.

I want to come back to something you said very early on, which is you actually started the cloud using our products with the WAF and those things.

You said you had tens of thousands of websites.

So just tell me about that. You're not alone in this. There are many companies that are in this situation where, in fact, you think maybe they have one, but in fact, they've got many, many.

So how is it you end up with tens of thousands?

Great question. I was shocked to learn that when I joined the company. And it's something that's not obvious.

But if you think about a company who has a lot of services, a lot of products, and then is really global, they kind of all multiply each other.

For every product, you probably have that deployed in different languages.

You have different divisions. You have different regions, different audiences.

In our case, we might have patient audiences or health care professional audiences or internal audiences.

So suddenly, your applications, at least your websites, multiply.

And just being a large company, of course, you have a lot of applications.

And I guess we also count a lot of those that some might not include because our internal applications are all being migrated to being accessible anywhere, which has been a slow transition.

But if you count them all together, yeah, we're easily over 10,000 applications.

It's interesting you say it's been a slow transition because I think one of the temptations is for vendors to be like, here's this incredible future.

This is what you should do. And then the reality is, actually, it's going to take us a long time to get there.

And what I'm always saying with our customers is it's definitely a journey where you're at some point in this thing.

You're like, OK, I want to do this first. I want to do this.

I know I'm going to do this critical application or whatever. You mentioned you still use a VPN for some things.

We are living through a massive change in the way in which we think about networking.

One of the things I think is interesting is that you and your detection role, presumably with the Zero Trust and even the WAFT deployed across many, many websites, you've got a different level of visibility into what's happening if you suspect something bad has happened.

Yeah, it changes a lot because in the past, especially internal applications, were always the hardest thing for us because once you've bridged that VPN or once you're in internal, there are authentication logs.

There are things available.

But ironically, the internal apps were some of the hardest things to really track what was going on, whereas a public app might have a well-defined firewall rule, well-defined logging, et cetera.

And by switching and by moving all of our internal apps into this kind of standardized web application firewall, along with access policies, now we have that same level of visibility and better on those internal applications.

So we see who did it, where they did it from, the context of those accesses.

And we have it very standardized, whether you're using an Nginx backend or IIS or Apache.

It doesn't matter. The technology standardized it. So for us, that makes life a lot easier to have one source of truth, at least for most of the web-based access.

Right. And are you pulling that data into your own seam or your own logging infrastructure so you can then use your own tooling on it?

Exactly.

In fact, we pull all that data in every few minutes, which is very convenient.

So it means we can write rules or detections or anything very consistently across the board.

It also means we have good retention. We know that we can look back a long time to make sure, did something happen in the past?

We have an audit trail for that.

And now it applies to internal stuff where we didn't often have that internal logging.

So certainly improved that. Yeah. So it's great to have that visibility that we just didn't have before.

Is your management coming to you with measures of like, I want to know this Zero Trust stuff really works for us.

How do you think about the success of it? Obviously you can see the sort of qualities you're getting out of it in terms of things, but do you have a way you measure it and go, actually, I can tell you why it's better.

So numbers are always hard.

This was always an exception and there's a lot of complexity, but we did track the progress during COVID and slightly before, starting at the beginning of January, 2020, we started looking at how many active users do we have on VPN and how many active users did we have in using cluster access-based internal applications that you didn't need a VPN for.

And it was interesting because of course, prior to COVID, we had lots of people, most people were working on-prem every day.

Maybe Fridays, you had a spike in VPN activity. And you only had a few applications at that time using this model.

So yeah, the numbers were pretty low.

And of course, as soon as you had the pandemics and everyone was forced to be at home, the VPN traffic spikes.

And we actually found some capacity limitations. When you've got 130,000 users across an infrastructure, you hit some practical limitations sometimes, whether it be licensing or bandwidth or whatever.

And then just even indirect ones, like just general performance slowing down because there are 20,000 people using one connection at once.

So as we dealt with that, we started to measure them and we saw these spikes in VPN activity, but it was also really helpful to see as we try to mitigate that, as we try to tell people, hey, there's another way, we rapidly try to onboard as many web applications as possible.

So we try to get on these applications that everybody would use.

So, you know, if you're working remotely, you can't badge into a time card.

You have to do that through the web. So let's get that on there. Let's make sure that everyone can record their hours every day, because otherwise you need VPN just to do that.

So we went after a lot of apps that were very, very common across the entire company.

And by doing that, suddenly people who might work primarily in the email or their collaboration tools or ServiceNow or something like that, if they only had to log in just for a couple of basic things, we were probably covering them with this access solution.

And that meant they didn't need VPN. And so at the same time that we onboarded these applications at the beginning of the pandemic, we also advertised saying, hey, you don't need to use VPN as much.

It's certainly a transition to get there.

And it's still tough to remove all the need because we have a lot of legacy.

But we saw the use of access just skyrocket and double and then triple.

And suddenly we're having tens of thousands of people every day just using this access layer.

And that's great. So I remember this because you were one of the accounts that I was keeping an eye on because you'd been trying out access.

And, you know, I had a little dashboard in case people would tell me.

And I actually quite well remember the point at which you just pointed the rocket straight upwards and were like, OK, engines are lit now.

We're going to onboard.

Because I remember there was a graph that went like, yeah, they're trying it out.

They've got a couple of hundred people using it. And it went like this. So that was obviously a big transition.

Tell me about that. Like, you know, from a sort of planning and implementation perspective, obviously, you were going through a planning process.

And suddenly, obviously, for all of us, the world got upended.

What was it like to do that actual change from, you know, from the sort of perspective of your team and the acceptance in the company?

So it was gentle for my team because we didn't have to technically do the implementation.

But my partner teams that were doing it certainly had a lot of work there because we weren't, I think, really prepared to do that sudden onboarding.

When we did our project to start this new model, we had planned to have a project, lay it out, do all the groundwork, all the stuff you have to do to document things, make sure the knowledge base articles are there and make a standard for it, which is fine.

But we were never staffed to onboard hundreds to thousands of applications suddenly.

And we didn't have a pressing need.

We were doing a pretty good job of just consistently adding a few applications, making life better for everyone.

But the pandemic just accelerated that.

Suddenly, we had a real pressing need to make things better for users today, not tomorrow, just to allow people to work.

So that was kind of one of my regrets, is we hadn't planned for the scale to really accommodate that scale.

And not every application needs to be done that way.

But it was tough to be suddenly put in that position.

So I wish we had planned a little bit ahead to scale up better.

But it was relatively simple because we had already worked out the hard stuff.

We worked out how to coordinate kind of the exception cases, et cetera. We knew what we were doing coming into it.

So scaling wasn't too bad. Right, right. And so you figured out the connectivity side of it.

And then presumably from the Cloudflare side, it was like, OK, add this thing, hook it up to this identity provider.

Off we go, and we'll run with it. If you, having gone through this, obviously you're continuing to go through it, this kind of transformation, what would you say to a peer about if they're thinking about Zero Trust, or maybe even the other solutions, specifically on the Zero Trust side of things, what should they be thinking about if they're going to make one of these transformations?

Not necessarily at the lightning speed that you did, but now having had that experience, there's a lot of learning you get from it.

I think acceptance had some challenges.

And I think one of the hard parts was even outside the security space, people thinking about an application being available anywhere actually really is scary for people.

They're used to, I have to VPN or be at the office, and that's the only way to get this, because that's the safe way to do it.

And to change that around and say, actually, the safe way now is that we want you to go through this special security filter.

We want your application to be on a place that only way you can access it is from a company-owned device, under this criteria, with a strong identity.

And doing that was really tough, because people just don't think of it that way.

So one of the ways that we helped to overcome was to onboard all the security applications.

So we had a few test applications early on, but the first thing we did was to try and onboard every security-relevant application first and eat our own dog food and say, if we trusted enough to put the most critical security applications here in this configuration and make them available on the Internet, even though they're not truly public, that gives people confidence that we really believe this is the right way, and we've got the security level to support this.

So I think setting that example is really helpful. And then people say, OK, this is something that is valuable.

The second piece is explain the benefits, because a lot of people don't want to change.

You've got an application that works fine today.

Why do I need to make it do something different tomorrow? And if you're the application owner, you're maybe not dealing with the user of that application.

You're not the one suffering like the user might be or inconvenienced.

So that's another challenge. And I think what we found by using a service like this was, say, we can take away a lot of problems.

And that was a message we always had is, we can do security with a tool like this and a solution and say, we'll manage your SSL certificates.

We'll manage some of those basic settings.

We'll handle security monitoring. We'll put security headers on every website you have.

We can put an access layer in there. And best of all, we can make this available anywhere to all of your users.

And that shifts it a little bit.

It's no longer, ah, security trying to do something and make my life painful. It's security's coming to make my life better.

And that helps move things. That's the thing that I find is really true with the Zero Trust solutions.

The day I deleted the VPN from my laptop was a happy day because it was one of those things where it was a pain authenticating to it.

It always felt really clunky. And then I was like, then I had to authenticate again to application.

It just felt like this really weird, antiquated thing.

And if I went from, I don't know, a coffee shop down the road to my home, the VPN went through this, I think it was churning, a sort of slow thing where it reconnected and tried to figure stuff out.

It just feels like a happier world for an end user as well.

And I actually deal with the same situation.

I don't have a VPN. I use a Chromebook every day. And that's not the norm, but because security came first and we put all of our applications there, the vast majority of the time, I don't need anything more than a web browser on a Chromebook and I can do my job completely.

Yeah. I'm presuming you can run the power wash on that Chromebook at any time.

And it's just like, yeah, we're back to the network computer ideas of the nineties.

There was a question that actually came from the audience.

Sometimes we get audience questions. Somebody asked the following, by giving control over VPN access, authentication, et cetera, aren't you giving Cloudflare the keys to the estate?

So the question came in the form of, did you get asked that by your management?

And how do you sort of talk, how do you think through that implication, I guess?

That's a good question. And that certainly came up in our risk review process.

And when we look at it, if you would have asked me 15 years ago, I would have been a little bit more hesitant, but today we face the reality is everything we do now is cloud.

We are pushing to a third party in every case.

The important thing is understanding the trust that we can put in those third parties.

And so with Cloudflare, for example, we did a good vendor review.

We understand the security models that Cloudflare does, and we understand the trade-offs that we make there.

But considering that we have so much benefit to this, and it's just very little risk that we see from Cloudflare directly is, yeah, well, there is some trust that has to be involved here.

But at the end of the day, it made sense for us to do this.

And there are certainly occasions where we can still do things on -prem or in our own way, but the vast majority of systems can go in this model.

Okay. So we're getting very close to the end of this.

I wanted to talk a little bit about, you were planning in 2018, 2019, pandemic hits, you go on the rocket ship.

We're going through the thing now. If you look forward sort of three to five years, what do you think the network and the application infrastructure looks like within your environment?

I think we're always going to have, or for a very long time, we'll still have a legacy.

And I think that's important to recognize is we're going to continue on this path of Zero Trust and our connectivity will certainly change.

But at the same time, we've got to keep in mind, we've got to support those applications that are key to the company.

And not everybody will work remotely. We certainly have laboratories and manufacturing space, and we've got to make sure that our solutions accommodate the whole broad range of solutions we have in the company.

So I think we'll continue down this path, but there's still a lot we've got to maintain from the legacy side.

Sounds absolutely great. Do you think you'll still have a VPN in three years?

Do you think that's always going to be a sort of backup technology that's in there that's helpful for some scenarios?

I think it's going to be the backup. I think we can move away from that being the norm though.

And we're already talking about even restructuring our VPN policies later this year, just because that shouldn't be the normal way to connect to most applications today.

So we'll have some real changes on how we use VPN.

And I think we'll slowly wean ourselves off of that need over the next few years.

Fair enough. Is there something, having made this move, you know, using our platform or using these kinds of technologies, is there something that has been a surprise, either pleasant or unpleasant about, you know, when you change to a new world?

Surprising.

The ease sometimes. There's complexity involved, but once you kind of get over the hurdle and you start to understand what's there, and you realize that you're not...

just how much is available when you concentrate this access layer and proxy in one place.

You know, for us, it's surprising how quick we can respond to things, how we can get into things.

I don't have to talk to 20 people now.

I really have a capability of saying, I can fix something on the fly. And that was surprising.

My favorite example is security headers. Trying to convince a thousand people to put security headers and content security policies there, that's really tough.

I can go to one place now and set a default policy and make that very consistent and just kind of raise the bar across many things at once.

And that was a surprising simplification.

Interesting. I'll make sure the team that works on our header transformation, we just actually announced a new way of doing header transformations today, just while we were talking it was released.

So more stuff going on there.

I guess on the WAF side of things as well, it's, you know, if there's a CVE comes out, you can just go into the WAF configuration and say, am I protected against that on every application?

Yes, I am. Okay, great. Again, flexibility.

If we had to do that on, you know, a thousand Nginx and Apache and IIS backends, it's a nightmare.

At least I can put reasonable mitigation in there pretty fast with this.

Right. Yeah. And you deal with this speed as well, right?

Because obviously hackers are already trying to exploit those things very, very quickly.

So the faster you can be rolling out the protection, the better, even if it's a somewhat simple one in the beginning, you'd be like, okay, we'll put that in place.

And that buys me the time or whatever to do a patch or upgrade my software.

Well, listen, we are into the last 15 seconds, Tim, thank you so much for talking about this.

You know, thank you for being a customer. It's been great, you know, having Rush as a customer and, you know, good luck with your future Zero Trust transformations.

Thanks, John. It's been a pleasure. Cheers.