SECURITY SPOTLIGHT - The Future of Application Security

Hello everyone. Welcome to our episode today on Security Spotlight on the topic of future of application security.

And what I'm really excited about is I have two of my partner in crimes today, two director of product at Cloudflare, Pat Donahue, who's looking at our application security from a VAT perspective and Sergi, who's with the background you can see that he's very focused on bot management.

Hey, both of you, welcome to the, welcome to the episode today.

Good to be here. Thanks for having us.

Yeah, it should be fun. This will be fun. It'll be a fantastic conversation.

To make it more fantastic. Let's start with something a little bit more personal.

From before we get into application security, what got you, let's start with you Pat, what got you into building products into application and things like that?

So I think it started when I was really young. My dad would bring home computers from work.

He worked in Massachusetts at some various computer companies, Apollo computers, one and some others.

And he would bring them home and I would kind of teach myself how to use them and teach myself the program.

And I studied computer science in college and economics as well.

And then, you know, tried my hand at being a software developer for a little bit.

And I found that I was, you know, mediocre at best.

And I was much better at talking to customers and helping understand their problems and then turning around and talking to engineering teams.

I also, from a security kind of product perspective, I spent some time as a penetration tester and it was really fun to kind of get in and, you know, get plopped down in a conference room and say, Hey, try to break into this network and write up your findings.

And that was really fun, but I thought it was, it was more fun to build something that, you know, people could use.

And I started to get into product management from that perspective. I co -founded a couple of companies and, you know, when you're an early stage company, you're very much doing everything.

And so, unfortunately for the companies, I was also doing the programming, but, you know, I kind of shifted into doing more product management as we grew and were able to hire some better developers.

So building products to help companies, people solve problems, and that led to Cloudflare.

And now you're here helping build a better Internet.

So that's, that's really cool. Yeah.

What's, what's your story? So it's kind of similar to past, but a little bit different.

So, I mean, I was a computer game dork growing up in the nineties. Computers were better than consoles.

They had better graphics. So I did that, but I didn't really know that computer science was, it was a thing.

So I went to college as a, similar to, as a business and econ major.

And I was like in the, on the finance track.

So going to do the bank internship and then going to get my MBA and all that stuff.

And then I, through the interview process with banks realized I didn't like this very much.

So I stumbled across a small email security startup.

So one of the first ones in the cloud, there's three at the time. Each one will argue that they were first.

I actually don't even know what the right answer was.

And I somehow impressed the people I talked to by asking, does mail get to you through an MX record?

And the only reason I knew that is I registered it out the domain at the time.

And I don't even really know why I did or what the domain was, but I just did it.

And so I was a generalist intern there, did like everything from Salesforce administration to like playing with Excel, you know, that kind of on my background there and started for whatever reason, joining sales calls as kind of an SE.

Just did that. Eventually moved into product management after a few companies later, I've been doing that ever since.

All security companies, I've never done anything that's not a security company, which is a little bit weird.

The person with roots in economics turned into a building products and security.

That's amazing. And I think one of the most fascinating parts of doing all of these conversations in security spotlight is just getting to know people's origin and their journey to where they are today.

So that's really great. Talking about journey, let's talk about the journey of applications and application security.

So let's start with applications. A lot has changed as you both have background in developing applications, but you have written code modestly said that that was not good code.

But really in all seriousness, how do you see, and that will start with you, how do you see this momentum of application development that has changed and what are some of the transformations that have happened?

Yeah. So I think a lot of, you know, what we've seen over the past 10 years has really accelerated over the past few years.

And so, you know, the, the kind of the really obvious trends are the, the more, you know, the faster development cycles, more agile.

You know, a call for, we like to get stuff in our customers' hands very, very early.

You know, we, we definitely follow the saying, if you're not embarrassed by your first release, you know, you're waiting too long.

And so I think there, there is that continued acceleration of, of, and recognition that, you know, the direction you may take on your product may be very different than when you envisioned.

And so, you know, get it in your customers' hands, iterate quickly.

You know, of course the continued move away from really large monolithic applications to microservices and kind of the architecture that supports that, you know, you see increasingly things running, you know, in Kubernetes and other, you know, container orchestration architectures, you see more stuff running at the edge.

I think in the last couple of years in particular, as edge computing platforms have matured, we see a lot of, you know, customers building stuff directly on workers today, which would historically run, you know, on their origin.

I think also, you know, the, the, the tendency to bring in third-party dependencies has continued to increase.

And so, you know, I think everyone's familiar with that, that, you know, left pad vulnerability with the, the, the node library, but, you know, I think across the, the development platforms you're seeing, you know, more and more dependencies brought in, including, you know, get run ultimately on, on, on your customers' environments.

And so, you know, sending third-party JavaScript to browsers that has continued to, to accelerate.

I think all of this is, you know, you see fewer people doing more things as well.

And so, you know, everyone's probably heard the term DevOps.

And so it's, it's, it's less, less the case that you have software developers kind of building and packaging in the old days, you know, you would send someone floppy disks or CDs.

And now it's, you know, really just kind of deployed some sort of continuous integration and deployment.

And, and that's really developer driven unless taking, you know, SRE teams or whatever the teams may be and having them involved in the, the actual code deployment.

And so these folks even in some cases are responsible for, you know, security as well.

Right. So the DevSecOps kind of role.

And so fewer people doing more things. And then lastly, I would say the, the, the focus on developer experience is something that has really sprung up in the last few years.

In particular, Stripe is a company that comes to mind that does a really terrific job here.

You're selling to developers on a lot of cases when you're building things.

And those are the technical evaluators and the decision makers and not necessarily the decision makers, but usually a strong influence on the decision maker.

And so if you can win the hearts and minds of developers today, you know, that's really advantageous.

And then even more recently is really the shift to, to remote work.

I think the application development, you know, companies have been forced to do things remotely where they may not have been ready for them.

But you know, COVID and the world we live in changed everything. And so that evolution is, is really forcing companies to figure out how to develop software where you're not in the same room or same building.

And, you know, I think that that is still in a feeling out period.

I'm really pleased with how, at least at Cloudflare, we've kind of continued to keep our productivity quite high, but you know, I think that that is definitely a challenge in building applications and secure applications.

Yeah, totally. A new, new normal and all this business technological pressures and on top of that environmental pressures now.

A lot of it also is changing from a bot's perspective, right? So what are your thoughts then application development?

Yeah, I mean, you see a lot of automated traffic, right?

So whether that's, that's good or, you know, a good, helpful bot or something that you've actually built to monitor your own systems or subscribe to, to monitor your own systems.

And then you also see obviously the kind of the malicious automated traffic that is doing something to your site that you, you probably don't want.

A lot of that is, is kind of driven by this, this API first development mentality, right?

So you, you build a bunch of APIs and then you build on top of those the way that you expect your users to interact with that software.

And that works really, really great.

You have a lot of open source software, lots of tools and frameworks that help you build that and put it together.

And it really opens up the, the kind of engineering stack to more than what was traditionally just a front end developer.

I'm a little bit older than Pat. So I remember when, you know, you, you built a thing and then you handed it off to another team that would actually expose it to the world.

And, and now it's your, your, the full stack developer, you, you start from the backend and then you kind of build it and get it all the way out there.

I think the other kind of tangential interesting thing there is a mobile first development.

You kind of get there with APIs.

This was a huge last year. Everyone I was talking to last year, it was, we are building a, our new thing and it is, we will build it for a mobile app first.

And then we will figure out the desktop experience from, from there. I think that's slightly less so in the last five or six months as we spend less time away from our, the computer I'm staring at right now.

But I think as we, as we do see a little bit of a pullback from, from that type of development, we still have to think about that the majority of the Internet and a lot for a lot of users, the mobile device is their primary screen in the primary, in the primary screen that they want to interact with, with someone's software.

So I still think as, as that pulled back a little bit, it will normalize into, into still the primary development method or the primary development target.

So do you think like this, the mobile first API first, are they getting more mainstream?

Because I remember like five years back, I was hearing a lot of companies taking that, talking about it.

And, but it was mostly like the top echelon of companies, like SAS companies talking about it.

Do you think that this is becoming more mainstream to where everyone is thinking that way or majority is thinking that way?

Yeah.

I would say that it's pretty unlikely that if you're, you're building a new thing starting today, that you're, you're not building it with an API framework and a number of different ways that you interact with that framework, right.

Whether that's your mobile app or your, your web front end, but it's still the same API backend.

Cloudflare dashboard is this way. We have an API that both our customers hit from the, from their, any API tool they build or terraform or the, the web UI is built on top of that.

So I think it's, it's pretty common. I think there is that area in between where you have the old app and you have to make the decision to build a new one.

And, and, and that's difficult, but yeah, definitely anything brand new is, is going that way.

Yeah, definitely. I've been at companies in the past where taking paths, comment about going from monolithic to microservices kind of thing.

Going from not thinking more API first, mobile first to going to that is definitely a journey I see a lot.

Great. Thanks for sharing your thoughts on the application development and how it has changed.

There are few people or maybe a group of people that are sitting on the sidelines, always looking at these changes that are happening in the application development and maliciously thinking how they can take a malevolent opportunity out of all the changes that are happening.

Right. And so from, and both of you are, you know, leading product teams, you're very, very curious about how that threat landscape is changing.

Share your thoughts around how you view the threat landscape changing in the application security space.

Sure. I'll, I'll start out here. I think, you know, actually I was a Cloudflare customer before I was a Cloudflare employee.

And I remember having, you know, one or two services exposed where everything was sort of routed through and, you know, our, the company was relatively small.

We had 40 folks and another one, you know, about 10 folks.

And so the, that surface area was largely due to the way that we were operating and how we were running this big, you know, Java enterprise application that was very centralized and running and, you know, core core data centers versus really distributed.

This was a lot of the stuff predated, you know, any of the cloud providers like Google cloud, AWS, et cetera.

So I think that the surface area is, is, you know, there's more to cover today, but, but the vulnerabilities that people are trying to exploit are largely the same.

You know, to be honest, I think in the first half of the year, we looked at what are the top four things that our WAF is, is blocking and it's the usual suspects, command injection, you know, SQL injection, file inclusion, a lot of spoofed, you know, fake kind of search engine traffic to try to get around various rules and things that Sergei and his bots team protect against.

I think the other thing that, that we saw is it seems like people have a lot of time on their hands.

The, the DDoS attacks that we saw in second quarter, Omer and my team just published a nice blog on this with, with someone on your team talking about, you know, the trends that we're seeing.

And so the, the attacks have doubled since the first quarter, I think largely, you know, people have a lot more time on their hands, maybe are a bit bored.

And then the, you know, it's, it's that constant kind of arbitrage of cost to launch an attack versus cost to defend an attack.

And so I think what, what attackers are sitting there doing in a lot of cases, they're not able to have any sort of material impact on the applications in some cases.

But if, you know, if you're getting charged for attack traffic, you know, there, you can still cause someone a problem by, by sending a whole lot their way, you know, while you're sitting at home, you know, sequestering yourself.

And so the, the, the, the size as well of the attacks that we've seen have really spiked.

And so, you know, we saw a recent one that was over, you know, 700 million packs per second, I think was the number.

And those are really the more hard ones, you know, we're at the scale where, you know, a large kind of volumetric attack, it's just, we're scaled up so much, but the, you know, the packets per second are the ones that get more exciting for the team.

And I think the, you know, with everyone remote, there is that, there is that kind of fog of war, if you will, of how do you detect and respond to, to attacks, at least very recently.

And so, you know, a colleague of Sergio and mine just, he got, he got sent this really targeted spearfishing attack where someone was trying to kind of play on that, you know, that lack of information and cohesion by being in the same office and, you know, trying to take advantage of that.

And so that's something that, you know, we're seeing change in this landscape.

And then lastly, I think just the continuation around stolen credentials and breaches, you know, you see Troy Hunt, who, who built his, you know, having been point service on Azure and Cloudflare, you know, he's, he's sending me an email far more regularly than I would like, letting me know that, you know, an email or account of mine has been in a breach.

And so we're seeing more of this and I think more attempted utilization of it.

And so, you know, you've got folks trying to, attackers trying to credential stuff from these databases and that's a trend that we see growing as well.

So I think Sergio, you're probably seeing similar stuff on the bot side as well, I would imagine.

Yeah, it's actually been really kind of unexpected.

So we saw, you know, the general gradual trend over, and we'll be writing a blog on this as well, of just traffic increasing over, over time.

But in the last four or five months, we've actually seen a, a markedly different and stronger increase of automated traffic over, over human traffic.

So human traffic is actually sort of plateaued. While most of the growth in, in overall traffic has been bots and not, and not, you know, the, again, the good bots, the crawlers that are helping your website, but the actual malicious ones.

Credential stuffing is something we see that's probably the most common issue that we help our customers with.

The first one that they tend to see that they have a problem for.

And that is, is both real credentials as in, you know, stuff that they've stolen elsewhere and want to see that works on, on another side.

But we've also just seen something like that, just buy stolen credentials. Not even buy.

I mean, they're just available. Right. So yeah, a lot of it is just out there and they can, they can pull it down.

And, and, and I think the purchase ones are probably a little more effective, but we've also seen a lot of just credential stuffing of random credentials.

Not ones that aren't on the, the, the, the known exposed list or empty credentials for that matter.

And, and, you know, that could be a bot testing its framework elsewhere to see, make sure it's, it's working and evading detection.

Or it could be somebody just kind of messing around, but we, we definitely see quite a bit of that.

We're seeing a lot of scraping.

I think that is one of the lesser known issues in the bot world is just pulling content down and costing customers resources.

So serving up content to not your customer, you know, somewhere between, you know, 50 and 70% of your traffic could be to an automated system.

That's just stealing your content to either use elsewhere or post elsewhere or something along those lines.

We, more interesting is we've seen kind of a rise of cottage industries, and this is, this is not recent.

This is probably more along the lines of the last couple of years where you have automated systems and really good tools that are meant to do a very, very specific thing.

So not, you know, scrape content from anywhere or log into any, any login screen.

But actually, I want to buy this set of sneakers from this site and I want to buy as many of them as I can.

So not even an industry like, you know, sneakers, but actually the specific site within that industry.

So someone sells this tool. They sell it to a small amount of folks. They actually artificially suppress the, the, the supply.

So they get one higher price for the tool.

But two, it's harder to detect because it's only on one site. And then it doesn't matter to the buyer because even if they're charging, let's say, you know, four or $500 for this tool, if they managed to buy three pairs of sneakers for it, they kind of paid for themselves and the ability to, to resell that.

And then, you know, you get to keep one. So that's great too. Another cottage industry we've seen that's kind of related is residential IPs as a service.

So as you find more and more systems that are trying to use a global threat intelligence and from IPs and attacks, you want a relatively clean IP or at least one that is not clearly bad.

So you can buy access to someone's machine, run your software there and, and, and connect to some site.

And, you know, whether that machine is willingly giving you access or as part of a somewhat unclear, but that that's definitely something we've seen.

It is kind of interesting. Last thing I'll talk about is a lot of times bot attacks are accidental DDoS.

So they're not volumetric in the sense that, that Omer on Patrick's team is, is, is looking to stop.

But more of, you know, if you're logging in or you're buying something, most of the things that a bot may do actually need to go to origin and, and generate some CPU usage.

So there is a higher likelihood that they will knock you down based off of just, you know, not volume, not volumetric DDoS volume, but just a decent amount of volume at a given point in time.

Yeah. And that's, I mean, this, we could talk about the application security threat landscape for an hour or even more because there's so many different vectors, right?

Let's switch gears to talking about how do we as IT professionals and security professionals secure against these things.

And I can go back to my time.

I used to be a security engineer a few years back, and a lot of it used to be kind of a bolt on approach.

I, you know, I want to protect against application security.

I buy a WAP box and, you know, I keep bolting on and bolting on different hardware boxes.

My question to both of you, because you work at cloud and you're building products that are more cloud delivered.

And the environment is changing to a cloud environment.

Do you think that the bolt on approach works today?

And if not, what's the alternate and why does it work better? So, it's not that it doesn't work.

I think, well, so first of all, you rarely start off with a homogeneous cloud environment, unless you're starting your company today, or, you know, in the last few years, or you're kind of transitioning from one thing to another.

So, let's kind of use the example of a legacy data center engineering environment moving into a cloud platform.

First problem is that it's not your Internet stack anymore, right? So, the traffic is not going through the series of boxes, the bolt on boxes that you described, on its way out to the Internet and back in.

So, that's problem A.

So, what do you do? A lot of times, you're either going to absorb the functionality of the cloud provider.

So, the providers know this. So, they have cloud specific versions of the boxes that you have, their own version.

It's definitely different than what you're used to.

It's not your rule set, it's not your policies.

But sometimes, that may be good enough. It really kind of depends on what you're building there, right?

As that environment evolves, where you want to kind of have more heterogeneity to your security posture, you want to kind of make sure that what you had on prem exists in the cloud.

A lot of companies start off with a extend the perimeter to cover the cloud.

So, this lets you use your tool, but now you're losing a lot of the benefits of the cloud provider that you're using, you're backhauling traffic back to your data center.

And you get some folks happy with that approach, some folks unhappy with that approach.

But it is definitely not an ideal state.

Another thing we've seen customers talk about doing is augmenting their existing environment with some sort of virtual appliance that the legacy hardware provider gives you.

So, a virtual box in the cloud.

I hear that more than I would like, but yeah, that is a reality. It's awkward, it's clunky, but it's a way to kind of get close to what you had.

And then we have the customers that are moving to us, the ones we talk to more often, which is something like Cloudflare extends your perimeter to the edge so you can put your backends and your applications wherever you want.

But you have a single edge where your security folks are still in charge and in control of the security posture.

They can write the necessary policies, they can see all of the traffic in one place.

And that gives you kind of the best of both worlds. Yeah. Yeah. I think I like to talk a lot about, when I talk to customers about pushing your security posture to the edge and putting it in front of whatever may be behind it.

A lot of the companies that we talk to are in a really hybrid deployment model, or some of the larger companies are just starting their move to the cloud.

And so to the extent that they can put Cloudflare in front and terminate TLS and apply a uniform security posture there, makes it a lot easier to migrate from on-premise equipment to the cloud.

I think one of the things that I see a lot of, and I think it's kind of an anti-pattern is what Sergi touched on, which is taking your infrastructure as it exists today and trying to forklift it to a cloud provider, and not trying to simplify.

And so I talked earlier about fewer people doing more things.

And if you've got separate teams that are trying to manage and learn different systems, if you have a Cisco router and a Palo Alto firewall and some other third-party intrusion prevention system and so on and so forth.

When I was in college, I used to manage a NOC.

And so I would be physically racking and stacking and patching these things together and trying to learn the different systems.

And it was mind-numbing to try to go through them all.

And I think I like to try to talk to companies and advise them to maybe there is something you have to forklift, but don't stop there.

Try to think about how can you consolidate down, ideally to Cloudflare, but if not somewhere else, and have fewer systems to manage.

And I think if you have simplicity, complexity is something that will drive security vulnerabilities.

And I think to the extent you can simplify that, it's all the better.

Yeah. Actually, one last point is there's also a performance hit there as well in some cases.

And so Sergi talked about trying to route stuff through centralized choke points.

Everything that we build as a company, and we talk about this all the time, and we have companies pitching us on partnering and something we always ask them.

There needs to be, at worst, no performance hit. If ideally we can improve performance through a security solution, but if you start getting into routing things through centralized choke points, or trying to stack proxies together from different providers, besides potential interop problems, you also have latency hits there.

So if you can consolidate, then you're going to get a lot of performance advantages in addition to a simplified security posture.

Great. So we just have 30 minutes for today's conversation, but I'll finish this because we both are in products.

Finish it with a lightning round, which I've not done at Security Spotlight before.

But 30 seconds, pick one product or feature that your team is releasing, and talk to us about that.

What makes you excited about it? So Pat, we'll start with you.

30 seconds, your time starts now. Sure. So do this real quick.

So we're consolidating everything down into a web application firewall. It's kind of the control plane for layer 7 security.

So we're adding all sorts of tools in there, combining threat intelligence with more static rules.

And as part of that, we're really focused on API security, discovery and security.

And so you're going to see a lot from us coming soon here, more positive security models, schema validation, mutual TLS authentication, all sorts of stuff.

And I think we're going to partner with Sergi and team on some really cool machine learning to detect anomalous traffic and block it.

You did really well on time. That's awesome.

That was right on spot, 30 seconds. All right, Sergi, your time starts now.

Sure. So we've been piggybacking off of Patrick's web application firewall with our analytics for since the beginning of bot management.

Well, both of you, thank you for joining.

I think there's so much more to talk that I'll have to invite both of you again to one of these episodes, chat a little bit more, but thank you for taking the time.

Thank you for sharing the insights. And thank you everyone for watching until next time when we come with another exciting episode of Security Spotlight.

Until then, goodbye, take care. Thanks guys. See you later.