Teamwork for Internet security: Rapid response & compliance

Hello everyone and welcome to This Week in Net. It's the May the 24th, 2024 edition and we're in sunny Lisbon, but we're going to travel to San Francisco again for the second week in a row to talk about security in a different perspective.

First, we're going to talk about how to put teams together to work on a short notice in a major task.

And then we're going to talk about compliance in the eyes of security.

So let's hear from two of our guests. Hello everyone and welcome to This Week in Net.

So we're in person in our San Francisco office.

So it changes a bit the virtual background. No need for that. And we're here with Ranee Bray, our Chief of Staff from our security team.

Hello, Ranee, welcome.

Hi, good to see you. Thanks for having me. I'm always asking this because in virtual backgrounds in video calls is important here.

Not that much, but why not?

Where are you usually based? Seattle, Washington. So just a quick flight north from San Francisco.

We are here specifically to talk about Code Red.

For those who don't know, what was Code Red specifically? Yeah, so Cloudflare had a security incident on Thanksgiving Day that we wrote a great blog about, lots of great information.

I believe Grant, who I report to the Chief Security Officer, did a session with you.

Yeah, about that. Code Red. So that incident on Thanksgiving Day, Code Red was essentially what we kicked off following the incident to strengthen our security posture, hardening our infrastructure, just really make sure we were prepared and mitigating any impacts.

So it was a 30-day, very concentrated effort of the entire company coming together to ensure that we weren't going to have further impact or a subsequent incident.

What I find interesting in Code Red and the way the company got together to resolve something, brings some awareness on how companies work, how missions work, how there's a problem, there's something we want to deal with or resolve totally, and we should resolve it.

So all hands on deck, if possible. How was to put that into action and try to get results quick?

Yeah, it was amazing. And I think it's that security of culture that we have at Cloudflare ingrained in us, and very mission-driven organization as well.

But what really, I say I will attribute the success to of all hands on deck, everybody get this work done in 30 days, top priority, is that we had that support from the very top of leadership.

So leadership said, while this was a very limited incident, we take it very seriously.

So this is number one priority.

The chief security officer has full reign. If we ask you to do something as part of Code Red, you will prioritize, you'll get it done.

At any team, different teams.

Yeah, finance, communications, HR, right? It wasn't just IT, infrastructure, security.

Everybody across the company. It takes a lot of different teams to pull something like this together in rapid succession.

So having that leadership from the top down and having it announced to the whole company, everybody was very much aware of what was going on.

If you got a ping, you had to go and jump in. Yes, yes, yes.

And the organization really did that. It was great. In terms of steps, what were the main steps on bringing more security, even in all situations for this?

Yeah. So attackers like to catch us when we're not around.

So Thanksgiving Day was intentional, I'm sure.

By Sunday, our CSO already had a plan. You know your priorities as a security organization.

It's, you know, there's threat plus the vulnerability, plus the likelihood of something happened.

Those three things added together give you your priority.

Grant already knew, and now he already knew kind of, you know, where we might need some areas we need to remediate a little bit.

So knowing those, and then now we have a likelihood of, well, maybe the likelihood has increased now, right?

We have a sophisticated threat actor targeting us.

So he already had a plan for us within like 48 hours of like, this is where we need to focus on.

So by Sunday, after Thanksgiving, I already had the plan from Grant, high level.

Monday morning, reached out to create work streams, owners.

By the end of the first week, we already had teams stood up, which was amazing.

We obviously, you know, continuously improved the structure over the 30 days, but yeah, Monday following the incident, we already had a plan.

We already had work streams, owners for each work stream.

We brought in our technical program managers across IT, engineering, infrastructure, and security to all collaborate and work together.

And in terms of lessons, were there lessons learned during this period of that work?

Yeah, yeah. I think the number one, before you even get to an incident, is your strategy of how you manage your information technology in general matters, right?

And by that, what I mean is like, things cannot be so manual.

Click ops needs to go away. Everything needs to be scalable, automated, so that when you have an incident, your resiliency is fast because speed matters.

As far as the execution piece of getting the work done, I think there was four big lessons learned that we, I don't know, maybe lessons learned is the wrong word, but four pillars of success, I guess I would say.

One was that top-down leadership, security culture from the top down.

Everybody was very motivated, right?

This is the most important thing. Most important thing. Leadership sets the tone.

Everybody's on board. So you have to have the support of the entire organization.

Second, you have to have a CISO who has the plan, right? You have to have a leadership team that understands the incident in context and where should we focus our priorities on.

30 days is not a long time, right? So where do we focus on that's relevant to what we're experiencing right now?

What are those areas that we need to go back and harden?

Just do some double checking of assurance and making sure we're really validating that we're good in case they come back again.

The third one I would say is you have to empower your teams. And that was really something we do well at Cloudflare.

Empower the decisions to be made by each Workstream pillar owner because they know best.

They know their area.

They're the SMEs. Empower them to make decisions. If you wait in a period like this for a week or two for leadership to make a decision, you've already lost time.

There's a trust level there. They're the experts. So trust them. Yeah. The last one is the framework though.

You want to empower people, but leadership needs to know what you're doing.

Everybody across those pillars needed to know what each one was doing.

So we weren't overlapping. We weren't, right? We were truly prioritizing and remaining prioritized.

So you have to stand up a lightweight framework so that the people who are empowered also feel safe and they feel like their leadership knows what they're doing.

So a very lightweight framework, not heavily governed, right?

But clarity, a framework of daily standups. We ended up standing up office hours so anybody in the company could call in to office hours and say, hey, I was asked to do this as part of Code Red.

I really don't know what I'm doing.

So really make everything accessible and give leadership the right purview into what is happening so that they can affirm that you're on the right track.

But you didn't wait for them, if that makes sense. So you're empowered, you're moving forward, but we're still seeing a big picture view to make sure we're still on the right track.

So we had weekly prioritization meetings with leadership, daily standups with the localized leaders.

And we also continuously, as part of that framework, got feedback.

There's an ongoing incident investigation with a third party we brought in CrowdStrike.

We would get daily updates from them as well, should we be reprioritizing something else based on what they have found through their investigation.

So I think that framework is the last pillar to make sure you actually accomplish what you set out to accomplish.

There's this interesting part there, which is in this situation in specific, the attacker tried to do things, but he was not, the person was not, we don't know, it could be she, was not particularly successful.

So no data of customers were leaked, nothing like that.

But it moved along the company. What were the, was there a sense of achievement after those days and how the, having a deadline, having put all of the teams there with a deadline, made things move forward more quickly?

Yeah, the teams were very motivated, but yes, very limited. The deadline was important.

Yeah, yeah. The deadline was important, right? I think Matthew said it best.

You can't, you can't have a code red go on and on, otherwise it's not really an emergency, right?

Exactly, yeah. So the 30 days was very helpful because it gave teams not only a clear deadline, but a sense of this is going to come to an end and I'll go back to, you know, because it was a lot of work.

Teams really put in the extra, the extra effort. But yeah, we had very limited impact.

This was all truly proactive, right? Proactive, make sure nothing else happens or try to assure that nothing else happens.

But the deadline helped.

And I think the teams closed out feeling very prideful, as they should have, because it was an amazing, it's one most active questions we receive from customers when they want to talk to us about code red is, how did you do so much in 30 days?

So our employees should really be prideful in that, that other organizations are even shocked that we were able to get so much done in 30 days.

And that customers are connecting with us, not because of a specific product, but how did we do something quickly?

That's interesting. They're peers. It's a strong sense of community and security.

Before we end things, I want to ask you, we're at RSA at this moment, it's happening.

Any interesting trends that you saw here specifically?

Yeah, I don't know a trend, but I did attend a very valuable session that, a tabletop exercise at the executive level, very important, because when you're in this kind of crisis mode, you have to know who's on first, right?

So we got to go through it firsthand live, but you cannot, I guess, recommend enough with a value of going through tabletop exercises early, even up to the executive level.

Like I said earlier, like every organization within the company we reached out to, PR, communications, HR, finance.

So the more that everybody understands who's making decisions, when is the right time to communicate it, right?

How do we evaluate customer impact?

How do we let the customers know? How do we let our people know when they can talk about it?

The more the organization has that ingrained into their processes, they know who to go to.

So I think, yeah, I would say that's the best session I attended at RSA this week was just tabletop exercises or, because as we've seen this past year, there's more and more cyber incidents.

So you got to be prepared.

Thank you so much, Renee. This was great. Thanks for having me.

And good, good work on Code Red. Thank you. And that's a wrap. My name is Ling Wu.

My position is Senior Director of Governance, Risk, and Compliance, and we center the security organization reporting to Grant Barzikas, our CISO.

I joined in June 2019. So we've been here for almost five years now. I have three teams, but one of our primary missions is really just to allow our customers to send their most sensitive information through our products and services while we adhere to those security regulations, frameworks, standards, and requirements.

Compliance is really important for the Internet and especially for businesses because of the type of sensitive information that you're sending through the Internet in general, whether it be passwords, patient healthcare information, credit card numbers, sensitive government information, whatever it is, there needs to be some level of requirements that have to be put in place.

How do you transfer that information?

How do you how do you retain that information? How do you store that information?

I think it's all very, very important. You want that information to also be accessed in a limited manner.

And so that's when governance as well as standards and frameworks begin to arise is because there generally needs to be that for data in general.

And so one of the things that I think has been really, really difficult for a lot of companies is just every country, every region is coming up with their own standards and regulatory requirements when it comes to transfer of information.

And so some of the challenges that we see here at Cloudflare is how do we keep up with that?

How do we maintain it?

How do we allow, since we're in the forefront of the Internet, how do we continue to allow people to use us in that way?

And what we're seeing evolve is everyone wants their information to be in -country.

Everyone wants their information to be in-person, managed by in-person, in -country persons.

And that becomes really, really difficult, especially when we're a really global company.

We have presence all around the world.

And the use case for why people use us, why companies want to use us, is because we're everywhere.

We have the ability to use us in front of our companies or customers' products and services to make things faster, to secure things.

And if you centralize everything within a country or within a region, that kind of defeats the purpose of what we do.

So we've been working hard on our ends, teaming up with the public policy team, speaking to regulators, speaking to standard bodies to provide our use case and our experience of why some of these standards or some of these requirements and upcoming laws don't work for us due to the nature of what we provide to the Internet.

Right now, we have been looking into a number of public sector compliance standards.

And we've been looking into the Australian public sector with IRAP.

We've been looking into the Japanese public sector with ISMAP.

And of course, here in the US, we've been looking at FedRAMPi as well.

And so all of those require data localization, all of those.

Luckily, none of them right now require in-country persons.

But as you progress to the various levels of these accreditation paths, you could potentially enter.

I only want Australian persons to manage the infrastructure within Australia.

I only want US persons to manage the US infrastructure as well.

We've been trying. We've had a number of conversations with regulators saying, why can't there be reciprocity?

Why can't you use the same authorization package, the same compliance package that we provide in the US to Australia, or US to Canada, or US to Japan?

I don't think there is enough communication between the agencies, honestly, and sometimes even internal within country.

I think a couple of years ago, I spoke to a UK agency, and they asked us the same questions as another UK agency.

And I was like, why can't you guys just transfer our responses from what we provided six months ago to today?

And they were like, we'll try.

We'll talk internally. But we still need you to do this.

I'm hoping that at some point there will be reciprocity. It makes it easy for not only Cloudflare, but many other cloud service providers in the industry.

I think the only thing that has surprised me from an evolution perspective is just more laws and it's becoming a lot more strict, I want to say.

So not only are there data localization requirements, but sometimes there are more granular security requirements that we would have to adhere to.

And they're very country specific. And I kind of hope that there could be some unified version of that to make everyone's lives a little bit easier.

But yeah, I think it's just people, a lot of standard bodies and regulators are just coming up with their own version of what security should be like for a cloud environment.

What would I predict? That this just becomes more and more important.

I think a lot of what we're seeing in the world right now, especially even here in the US, that there are more security incident notification requirements.

We have to adhere to so many. And so if we ever have an incident that arises within the security team, we work directly with our privacy counseling.

We have to figure out, do we have to notify the specific regulators for each country?

And they all have different notification paths. Some require a specific token that needs to be entered.

Some requires an email. There's no one place that you can just say, hey, I have a security incident.

Please notify all our regulators.

We have to do that actually manually. And we can only see that it's growing.

What for me would be a better Internet for the future? I have a four -year-old and I have parents who are aging.

And so for me, what would be really great is to be able to basically have a score or a grade for anything within the Internet.

Because my parents don't know if something is trustworthy or not. They don't know whether it's a real site for them to enter their information.

And my daughter is four years old right now.

But when she's older, I'd like her to understand that things are real or unreal and have that grade.

It should be a free thing that we provide to everything, to websites, applications, to whatever you are doing within the Internet.

There should be a score of some sort. In our blog recently, we had a few different blog posts.

One is East African Internet connectivity, again impacted by submarine cable cuts.

So that was after the March problems that occur in West Africa affecting 13 countries.

There was a problem with submarine cables again.

Then we had a blog post about expanding regional services configuration flexibility for customers.

So this brings an increased set of defined regions to help you meet your specific requirements for being able to control where your traffic is handled.

And we also had a technical perspective on running Fortran code on Cloudera workers.

So we'll discuss a bit more of that in one of the upcoming weeks, but we have an update in terms of blog there.

We also have an update in our developers platform. This is related to AI Gateway.

AI Gateway is an AI ops platform that provides speed, reliability, and observability for your AI applications and is now generally available.

In this case, we have Craig Dennis, our developer editor speaking a little bit more about it.

Hello, we are going to talk about AI Gateway and I want to talk about why I love AI Gateway.

I just ended up needing to use it. The new open GPT 4.0 came out and I had some questions about it, right?

Because I wanted to build an app and I built an app using it because I'm always exploring different models.

And one of the things that I wanted to know is I wanted to know the logs that were coming through.

I needed to like write this little app, what sort of what's going through here.

And I'm also interested in analytics.

What is going on in my app? What is happening? So what are the analytics that are happening?

And of course, you know, the big one, dollar signs, how much money is this going to cost me?

And if people are using the same sort of query, and this is a lot, there's a lot of this going on.

I'm doing a really good work here and I'm costing me a lot of money.

Can I actually cash some of this stuff?

What sort of caching would I have available? And then also, if somebody comes and they want to start using it a whole bunch, is it going to bring my app down?

Am I going to end up owing a lot of money? And so what if I wanted to add a little bit of rate limiting in it?

So like, you know, they say that these are just one API call away, these AI apps that we do that like, but there's a lot of stuff that we might want to know.

In effect, I'm here to tell you that there are some solutions to this.

Like how many lines of code do you think this is?

How many lines of code do you think it would take to write all of this? All right, I'm gonna clean that up.

All right, let's take a look at some code. So I wrote some code.

Here's my GPT 4.0 code. It's not nothing too advanced. It's a little I've just run a thing through a command line here.

And I'm going to pass it a prompt.

And you can see I'm going here to this GPT 4.0. I've passing in my OpenAI API key.

So one of the questions that I wanted to know is how many lines of code do you think it would take to add real time logging and analytics just alone, we're not talking about caching or rate limiting anything like that.

Let's see what comes back.

I love how fast this API is, right? It's coming back super fast, scrolling through some stuff.

It was giving me some code. That was very nice of it.

Awesome. So we've got we've got a really nice response here that's coming back.

And let's see if does it say does it give an estimate of time here? I didn't see 200 to 100 lines of code.

What if I told you you could do with a one line of code?

Check this out. All right, we are going to go and we are going to go into the AI gateway here.

And I've got this Craig demo here. And I'm going to click this API figure.

So here's my line of code that I need. I'm going to grab this, this API endpoint, you see other providers in here.

So I've gone and I've made a gateway.

So now I have a gateway that's all mine. And what I'm going to do is I'm going to run all of my requests through that.

So what we could do then is we can see, you know, for here's my open AI one, I'm going to specifically grab this, I'm going to go back to my code.

And I am going to jump in here. And I'm going to do here's my one line base URL equals, and we're going to paste in that thing, I'm going to run it, I'm going to run this query again, going to get a different thing, right?

Because it does something different each time. But it's the same question, which is interesting, right?

Because what really should happen is it shouldn't have to do that again.

So so about 330 to 100 lines. So watch, here's caching ready, I've turned on my gateway, I turned an option on for caching and check this out.

Boom, it is cached now. And it's captured in the Cloudflare network, which is very fast, as you are aware.

Alright, so back to the analytics. Let's go back.

I have been testing this thing a little bit here. You don't need me in there.

That's for sure. Let's get down here. Alright, so you can see that that came back cached, right?

And it was successful. I've got real time logging of the things that went through.

And I can take a look at my analytics. And you can see over time, you can see that I just blew a penny, y'all.

I blew a penny on this demo.

I hope that you enjoyed it. And you can see that I've been doing some cache responses here.

You can see how many requests I made, how many tokens I sent. And again, one line of code.

And that's a wrap for this week.