🔒 Security Week Fireside Chat: Nils Puhlmann & Joe Sullivan

Hi, everyone. My name is Joe Sullivan. Welcome to Cloudflare TV. And I'm here today with a special guest.

We're going to each introduce ourselves in a minute. Our topic of the day is going to be account takeovers, something that we both have had plenty of opportunities to deal with in our career, and hopefully we'll have many more opportunities to deal with as we continue in our security careers.

Let me turn it over to you so my guest can introduce himself.

Yeah, thank you, Joe, for having me.

I'm Nils Puhlmann. I have run security for both consumer-facing and also business-facing companies.

Super excited to talk about account takeovers today, especially looking at this from two angles, B2B and B2C.

Awesome. Yeah, my background, I'm the chief security officer at Cloudflare right now, which is primarily focused on B2B, but we do have actually a lot of small businesses and consumers who use our products as well.

But my experience with account takeovers really started way back shortly after the Internet started.

I think account takeovers became a problem as soon as e -commerce started to grow.

I was at eBay in 2002, and I think we saw more back when they first started keeping track of how many phishing attacks were going against consumer-facing companies.

eBay was the most phished site where people were getting emails saying that it was eBay when it wasn't.

And then we acquired PayPal, and then PayPal was the most phished consumer-facing website for a number of years.

And then I went to Facebook, and Facebook became the most phished account takeover.

We've definitely seen account takeovers evolve, though, in this last 20-year period as I think it maybe initially started as a consumer -facing thing, but it's definitely become a challenge for all of us running identity and enterprises as well.

So, Nils, you mentioned that you've had to deal with it in both contexts.

Let's start out on the consumer side.

How did you first have to face the challenge on the consumer side? So, having run security at two gaming companies, both Electronic Arts and Zynga, obviously, the gaming world was notorious for account meddling, I would want to call it.

And I think actually what you mentioned, Joe, kind of made me think about this a little bit when you said it started with e-commerce.

I actually think there might have been a shift where cyber criminal activity shifted from attacking companies head on to realizing that there's actually a lot to be gained by going after the user.

And it was easy, right? It was easy, it was cheap. So, whenever you have a lot of users come together, you mentioned eBay, game companies definitely in the early days and even today, lots of users come together from all over the world and from kind of all parts of society.

So, I definitely had my fair share of having to deal with it.

There was obviously value there to get, there was sometimes just reputation gains that people wanted to get by getting into accounts.

And then, as you kind of hinted to all the way now to where I think companies that are more in the B2B space are really learning that ATO account takeovers are different in the B2B world, but they are increasing and they are also getting more and more sophisticated.

Right. And when I was thinking about this topic back in the early days, to me, account takeovers was very much associated with phishing.

And then I think it evolved and I started thinking it was about not just phishing, but then there was the next layer, which to me was very much automated guessing of passwords.

And we saw large -scale attacks where they would just try the basic passwords that we all know everyone uses against every email address that they could find.

And so, then companies had to respond with some basic rate limiting.

And then to me, like V3 of the attacks was probably going down this whole other path where the dark web started to get filled up with these lists of compromised accounts.

And so, you had large consumer platforms that were getting hacked and then their database of usernames and passwords were sitting on the Internet.

And we then learned that a large percentage of our customers use the same password on all their websites.

So, to me, those are three specific trends in terms of how the attackers were going after accounts to take them over.

Are those the main three that you think of or there are some others that should be on that list?

Yeah, I think you're right, Vivek. It was an evolutionary path and I think it was a path that also took into consideration costs.

Because if you think the early days, just spamming people was easy and cheap because there was hardly any good anti-phishing technology out there.

I mean, we remember the lawsuits against the spam kings that eventually went away, but it was easy.

You could run massive operations hardly for very low costs and be very effective.

And then we kind of had the blacklist, the spam blacklist, and we kind of countered that.

So, there was the next level then. And I feel like every time there was a huge wave of successful attacks, we countered it with some sort of mechanism, whether it was better anti -spam, anti-phishing technology, or we ended up with databases of stolen credentials that now were available and you could use.

But it was really always an arms race. And the attackers always evolved first.

It wasn't as predicting the next wave of a version four, it was actually as always responding with something that took years often to force the criminals to come up with a new way.

Right. It feels like lately, there are lots of different kind of plays on the word phishing, you know, vishing, spear phishing, etc., which brings me to kind of the transition to the kind of the corporate environments where we see the attacks happening.

Whereas on the consumer side, we're much more likely to see these automated large scale attacks where you're trying to counter a massive password guessing operation.

In the corporate side, what I see is much more targeted attacks where they will, initially they were targeting just based on passwords, but as our corporate defensives have gotten better and we've added better multi-factor authentication into our environments, we see much more targeted attacks.

Have you seen that evolution as well? And the level of sophistication go up?

Yeah, it's more targeted, but then also I think what might be actually to our, when I say our, kind of business environments, disadvantage is also the complexity of system infrastructures.

Let me give you an example.

There was a company that just, without naming names, just rolled out a feature that they had to pull back within a day that probably would have made things like this much, much easier.

Right. And we all rely on it. And it's like a new feature coming out in one day that impacts business environments globally in a tremendous way without really having thought through what the impact is.

I think we will probably see more sophistication we have seen on the business side, you know, the abuse of the insecurity of SMS messages as another factor, right?

SIM swapping, you name it. So I think the attacker that goes after structured business environments has to probably spend more time thinking and creating it and not massively just blasting you with things, which means they operate similarly when we have seen nation -states preparing their attacks, right?

Like the time you spend preparing is much bigger than you actually execute.

Right. Yeah.

It's interesting to think about how, what the attackers get out of the attack will, it will increase their willingness to take their time and be very targeted.

And so if you can, if you're taking over, you know, eBay seller accounts to list plasma TVs and have a high feedback rating, it's, there's still a lot more work to be done in actually listing the fake items for sale that, you know, that stuff that I was dealing with back in the day versus later on when I was at Uber, if you could fish and take over the account of a, of an Uber driver, then the risk was when Uber, you know, delivered the payment to the driver, they could just quickly move the money to a different ACH to a different account.

And so the, the, the opportunity rose for the attackers.

And then you take it to another level. When you talk about the corporate attacks, if they can get into one of our corporate environments and modify settings you know, for all of your customers or take over a large scale customer account that the opportunity, you know, like we saw with the Twitter attack from a few months ago, they social engineered, the customer support team got their login and then logged in and started using a bunch of different accounts for cryptocurrency motivations, which have you seen the rise of cryptocurrency related attacks impact your world from an ATO standpoint?

Not directly, but obviously there's a lot of, I think the crypto industry is getting a lot of heat right now, right?

We see lots of breaches. It's probably also an outcome of security is hard, security is expensive.

And, and, you know, it, it probably takes more to, unfortunately, it takes more to establish a mature, well-run security department in a company than it is necessarily to figure out the latest crypto technology.

So you have a lot of companies that, that are just not ready from a maturity point of view.

And that actually leads me to, to something you made me think about is we have also seen more, more ecosystem attacks trying to get, to get into accounts or into infrastructure.

So instead of hitting, you know, mature companies, like you mentioned Uber, straight out, you just go after supplier or vendor or somebody who is, in the ecosystem, there have been more still email-based or email-borne attacks against smaller companies, you know, like law offices, real estate companies that tend to have weaker protections, compromise them.

And then through that compromise, then actually get into like a bigger company and, and our detections and defense mechanisms normally do not take into account that there should be probably a different trust level based on the size or the maturity of, of our supplier network or our third parties.

It's mostly like a very binary thing, right?

I do business with you, therefore I trust you. And there's no, there's no sliding scale.

And I think the attackers have learned that.

So instead of going straight after, you know, the, the main ship in, in, in, in the Navy, they go after the supply ships and realize that they're very, very effective about, right?

Right. Yeah. There's this, again, in the security terminology list of account takeovers that I think the newest is this, what we call business email compromise.

And we've seen financial, the finance teams of a number of the large tech companies in Silicon Valley fall for this one and lose millions of dollars.

The employees of these companies have been trained really well on recognizing a phishing email that comes from a random source by looking at the headers and the email or, or things like that.

But in the business email compromise context, you're getting an email from someone you already do business with, and they're just asking you to change the ACH routing for your next payment on your regular billing.

But it's not, it's not the real user. It's not the real person you're used to dealing with you on the other side.

It's someone who's taken over their email account.

And so, you know, I've seen some security startups like aim specifically at that tiny little niche area of like targeting finance teams.

It's, which I think let's turn our conversation for a minute to, so from like, we've kind of laid out the landscape of the different types of account takeover attacks that we can see.

Let's talk about solutions. It seems like we all spend a lot of time on education, trying to educate the end user.

Is that a winning or losing strategy?

Interesting that you bring this up because I had a heated online discussion with somebody in our space that I value a lot, but we didn't agree on this completely, where his point of view, what that's, that's all theater.

We don't really need that anymore.

And, and I actually don't believe that completely now training and awareness alone won't, won't solve our problems.

But I mean, we have seen in, in the physical world that that is still necessary to this day, right?

Like you still need to have people be mindful of old saying of like, see something, say something.

It works actually. Right. Like it's, it's, it's in every other environment and I don't see how this would be different in, in, in a virtual cyber cyber environment.

But we also need, I think, you know, much more creative defenses, like what, what you mentioned earlier, like the V1, V2, V3, and then how we like shored up our defenses in a very synchronous way won't work anymore.

I think we'll have to think about these problems completely differently.

Let me give you an example. I was in a meeting the other day and I saw, and it was demonstrated how cyber criminals now train with the use of AI, their phishing methods based on publicly available emails from companies.

They get the tone, they get everything right.

And it's so good that all our existing defense mechanism would not detect it, even the most advanced.

So that leads me to believe that we need to think about the problem differently and be much more creative.

And one way could be, um, we probably also need to train our folks to, to, you know, perhaps write emails differently or like change the style or, or, or do things that increase the cost and the complexity.

And we haven't really done this, right? Like we, I think we, we rely on technology a little bit too much.

And instead of taking learnings from other worlds and say, you know, then we need to change how we operate.

Now that's, that's a big one, right? Because normally in security, we say, we should never ask our companies, um, too much how to change how we operate.

But, but, you know, if a risk goes up so much, you have to, you have to do whatever works.

Yep. Yeah. I I'm with you on the idea that training and educating employer, our employees in the corporate context or in our users, in the consumer context, um, there is value in that.

Uh, I don't buy into the mentality of some security people.

Like you were referencing who say, you know, all user, you know, they love to say things, all users are sheep and they need to be herded along and treated, you know, as not intelligent.

I disagree. Um, the way I think about it is, especially when it comes to these like targeted phishing campaigns, if I don't need every single person to report it to me, I just need one because as soon as we, we get, if 20 people in my company are getting targeted on a phishing attack, if one of them lets my team and I know, then we can go instantly remove the other 19 from inboxes before that was employees ever even saw it.

Um, and so like, it's, it's like you said, um, one vigilant employee that can help a whole group in those scenarios I've seen, uh, fairly frequently.

And a lot of the email providers give you a tool that if it automatically happens, if one person marks it as spam, then they'll pull it back with automation from the rest of the employees who received it.

Um, and I, and I think we can also do education in terms of things like, like if you get a request to change the routing of financials, that's the kind of thing that you should always validate by jumping on a video call with the person to talk to them.

Don't just accept the email. Uh, and so getting some basic checks and balances through education definitely can make a difference in these cases.

Um, I also think we need to be think bigger than just in the traditional confines of companies.

Um, like, you know, users are targeted in their private accounts.

Uh, executives specifically are targeted in all sorts of accounts that, that they tend to normally spread all over the Internet, uh, and don't really take care of.

Right. So I think cyber criminals have realized that, that going after the weakest link is actually very, very effective, cost-effective too.

We haven't done that yet as companies, we always like still while technologically we don't have, you know, the mode and the firewall and all of that anymore, but perimeter protection only, I think mentally we still think like that.

We still think like that, right.

We protect business email accounts. We actually probably should say, actually, no, anything that happens, even on your private Gmail on, on your, you know, something that that's out of the norm that happens on your personally own phone, right.

Or through SMS, we should know. I agree a hundred percent. So a good friend of mine who I used to work with, and he's now doing consulting and he goes, he's been getting called in to help companies respond to ransomware attacks.

And, you know, the typical ransomware attack is, you know, the, one of the employee makes a mistake, somehow opens the door to software being downloaded into the company environment, and then everything gets encrypted and the company has to pay to unlock it.

According to my friend, like the last five of those incidents that he responded to at five different companies, the, the entrance to the company was exactly the same in all five.

It was the company employee opened up their personal email on their work laptop.

You and I know we have really good email security tools that we can buy off the shelf.

And we have customization in our enterprise email products themselves that can block out a lot of risk.

But if an employee opens up a browser window and opens up their personal email account, we have no idea what they're going to click on.

We have no visibility into it. We might, you know, we might have a decent endpoint solution product running on that laptop, but if it's a, it's a, we have a new product for browser isolation here.

And, you know, usually when I think about browser isolation, I think about like my high risk employees, you know, it's, it's the, it's the teams that are always opening PDFs that are being sent to them.

That could be malicious. I worry about those teams. But I actually, I would like one of the ways I want to implement it is I want to say, if you're opening up any personal email on my company machine, I want it in an isolated browser because I fear this specific attack.

And this is probably why we will see more virtualization technologies being used in the future and security, right?

Like, like running things in some sort of container or isolating, you know, activities from each other.

I, I just have a hunch that we'll, we'll have more of that because that in a way beyond one of the only chances we have away from getting better about detection is reducing blast radius and, and, and attack surface, right?

And, and doing that transparently for a user is, is, is challenging, technically challenging, but I, I have a feeling that's where the future lies, right?

We, we have to operate a little bit like think about, think about running in an ER in a hospital, right?

You want that sick people come to you, but at the same time, you don't want better spreads, right?

How do you do it? So there's a protocol.

And, and if one would come in that is really, really sick or contagious, okay.

The blast radius is, is limited and it's appropriate to, to the health of the other beings coming in that type of thinking in security.

We have it, I think when you and I talk and we talk with others, but we haven't, we haven't seen enough of that implemented in technology, right?

So let me throw there. There've been definitely a lot of different solutions out there considered around these account takeover challenges.

I want to throw a couple of different ones at you and like see how your thoughts on them.

So we talked a little bit about user education.

We're starting to see more situations where platforms are forcing users into multi-factor authentication, good idea or bad idea.

I think in terms of a numbers game, good idea because you're reducing the number of potential compromise.

You don't eliminate them, but you make it exponentially harder for an attacker, you know, will he move to a site that doesn't use MFA?

Yes. Right. When you look at the consumer facing sites, you know, like say social media platforms, the percentage or even email providers, the percentage of users that have turned on multi-factor authentication when it's not mandatory is never north of 10%, right?

Why is that?

I think, listen, I think that's normal human behavior. If you go back, take the car industry.

I remember, you know, when the first numbers came out and people wouldn't wear seatbelts.

Actually, that's why the airbag was invented, right?

Because people wouldn't wear their seatbelts. The first airbags were actually enormous and huge because they were supposed to catch a person without wearing a seatbelt.

And then I think we built things in, warning lights and beeping lights.

And so you couldn't even start a car. I remember it was one car you couldn't even start it without, you know, and then people would plug the seatbelt in behind them.

But I think through building in more mechanisms, eventually we got people to do it.

Not because, and then there was a big campaign by the car makers showing people flying for windshields.

I think that also helped, but it's just human behavior that we want it easy.

We want it quick. And we don't want to do the extra step until the extra step becomes normal.

Okay. Today, you wouldn't think about putting on a seatbelt, right?

It's normal behavior, but that's the goal.

You have to make it normal. And that's hard. That's hard in the online world, right?

Yeah. I think we're seeing, particularly with financial accounts, they're actually, they must have bigger losses because they do force you to at least do some SMS verification or email verification, like as a second factor to authenticate into your accounts regularly.

So I think people are starting to get more used to multi-factor, but like, as you said, SMS is pretty unreliable at the end of the day in terms of it being a trusted second factor.

What about forcing people to create really long, unique passwords?

Does that cause more customer support pain than value?

Because every time I create another account, I personally use a password manager for everything because I do have a unique password for each thing and multi-factor authentication, but is the average user using a password manager and able to handle all these long passwords?

You know, I remember well when I said I have a password manager for my daughter and she looked at me, she said, daddy, this is way too complicated.

Like I have several devices. This is crazy. I'm not going to do this.

Right. And it hit me because we are used to it. You and I work in this space.

So of course we want to do it, right? It's still too difficult. The whole authentication piece, there have been so many things where, you know, the FIDO protocol and you name it.

I mean, lots of advances have been made, but we haven't solved a problem.

And I don't think we have actually put enough effort and money behind rolling out systems that do what I would call context-based authentication, meaning depending on what you're about to do, I'm increasing or decreasing the level of certainty I have.

Right. So if I'm a bank and you want to do a bank transfer, $10,000, yeah, I'm going to ask you for a lot of things.

Right. But if you're just going to come and check in and do something that you normally do, why?

Right. And especially if I could watch patterns like that, like, you know, for example, let's say somebody has the same browsing pattern on Facebook.

Right. And you let the person in the low threshold of certainty, but then they do what they normally do.

Why? It's fine. But if they suddenly do different things, now you can throw something additional at them and say, hey, we need to recertify you or whatever.

And some sites have done it, but it's still too difficult. It's so hard. You know what it would take to roll this out.

It would take several products and lots of engineering hours.

Right. Yeah. I think like I've lived through that being at eBay, then PayPal and Facebook and Instagram and everything at Facebook, then at Uber, we had to build our own custom kind of account takeover systems.

At each company, we use different terminology to educate the users.

We use different ways of deciding when to force authentication.

We did try and make it risk-based because we didn't want to create friction for the user.

But I had to have large engineering teams just focused on looking at risk in context and then deciding when to do a challenge.

And if you're a new startup, a young company, you've gone in and worked at young companies over and over again.

How do you get the company to invest in this?

Or do you think that we're going to see a trend toward more federated login? I think that's the solution.

We probably need more federated, but we also need probably some services that actually offer this as a SaaS service, where I, as a young startup can just plug in and they can also use the knowledge of having many customers like where these thresholds should be.

I think there's another angle that we haven't really looked at, but let's say there would be a service like this and several startups plug in.

The knowledge they could derive by seeing patterns across different startups, and perhaps even with the same user accounts is tremendous.

Because if a user account is taken over, guess what? Most likely the abuse won't just happen on one site, it will happen on many sites.

And we haven't really fused that together to say there was a suspicious login on site ABC from this user.

Okay, I now need to basically raise the bar on all these other sites where I know this user has an account, just to be proactive.

We don't have anything like that today and we need these services.

Yeah, the one place we do have it right now is where like here at Cloudflare, we do have the bot mitigation side and rate limiting and surfacing captures.

And so we can do that up to a certain point, but we're not focused on the actual management of the credential and the identity.

And so we do see large -scale adoption of our bot mitigation for that.

But to me, the question is, what comes next? If you are starting a company today, would you want to be in the business of managing a bunch of accounts and passwords for all of your customers?

I think you shouldn't be. I think somehow the feeling though is like number of users equals to some sort of value.

I don't know what that is.

That's from the early days of the Internet, right? I think a lot of VC decks were probably created with saying like, hey, I have 10 million accounts, right?

Under management or like 20 million users, right? You remember the early days of Facebook and when I was at Zynga, right?

There was always this competition, like how many users is it?

Which now these days, I feel it's actually secondary. We probably need to have a courage to say users in itself or a number of users does not directly create any value.

It's what we do on your platform. So why not outsource that whole piece?

Now that'll take a lot of courage, but I think eventually we should have that and then put it in the hands of somebody who actually does a good job.

That's probably why the adoption of the federated Google, Facebook, Apple logins has taken off because it's so much easier to implement, right?

You get an SDK, API, everything done, and you don't have to worry about this piece anymore.

But I've spoken with startups and say, now we are worried, but all of that stuff is now in control by Facebook or by Apple.

And I would probably say that might be a good thing until you get big enough where you can take it over.

Hey, you know what?

We've filled our whole half hour. That was fast. Thank you so much for joining me on air and Cloudflare TV talking about ATOs.

This has been great.

Thank you so much for having me, Joe.