Okta compromise (and HAR sanitizer for all) special edition

Hello everyone and welcome to This Week in Net. It's a special edition related to the Okta Compromise and also a file sanitizer tool that we put together and is now available.

We're recording this on October 31st, 2023, so it's Halloween. I'm João Tomé, based in Lisbon, Portugal, and with me I have Sourov Zaman, Incident Response Manager.

So happy Halloween, Sourov. Hey, happy Halloween. How's it going? Yeah, so Sourov Zaman, Manager of Incident Response globally over here at Cloudflare.

Been here for some time.

Excited. This is one of my first times I'm doing one of these. So it's Halloween, we get to share some spooky stories, right?

True. Securely related.

Yeah, this spooky story will have a good ending, so it'll be great. Exactly. So in this case, you're at Cloudflare for how many years?

I'm actually making two years.

November, mid-November. Oh, so you joined at around the same time as me. Yeah, and surprisingly, I joined when Lock4j was happening.

My first day out of training was Lock4j.

And I remember my manager at the time. That was a major vulnerability that still has reflection till this day, right?

Yeah, most definitely.

I mean, a lot of the things that we're facing now, and how we know how to approach an issue and how it comes up, right, is because of some of the work that we did over there, right?

And we also see like, Lock4j still exists, right? Or exploitation of Lock4j still exists.

There's people are still exploiting it. We still hear horror stories about it, right?

But for us at Cloudflare, right? So interestingly, I guess it was like my first day, they pushed me into the fire.

I'm like, I don't know even what Cloudflare does and how we're situated, but we'll figure it out, right?

But a lot of the learnings that we took, for me, at least, especially for me, right, that was like my onboarding by fire.

So a lot of the learnings I took from there allowed me to navigate through Cloudflare and also helped me even for this incident, right?

Like I knew who to reach out to. I knew kind of like what our capabilities were, how to execute them and things like that.

That's good.

Before we go there, I want to ask you because you're based in New York, right?

In the US? Yeah, I'm in New York City. I'm based in Lisbon, Portugal, but in Portugal, Halloween, now it's big, but when I was growing up, it wasn't.

So do you have a Halloween story that you want to share with the audience?

I have a few, right?

I think I don't have my trick or treat basket with me, but yeah, sure. So I've consistently celebrated Halloween up until like the age of, you know, you turn into a teenager, you dress up, but you don't really want to do the actual going to house to house.

But some of the fondest memories I have, I grew up in Queens in New York City and I grew up like in this apartment complex.

And I remember this one particular Halloween, I was probably like eight or nine.

I still remember it. All the kids in the apartment complex, you know, we're like, Oh, we should go to Halloween together.

And we all decided to dress up the exact same thing. So I don't know if you're familiar with the white Power Ranger.

So we all dressed up as the white Power Ranger.

And this one kid dressed up as the gold Power Ranger. I'm just like, why are you different?

Why are you different from us? But it's just like, I sometimes sit around and think of that like situation.

I'm like, Oh, this is, you know, growing up, New York City is different now, right?

It's like, it's not the same, those sort of like smaller communities and things like that.

They're here, but it's slowly fading away.

So I, you know, as I'm growing up, I come to wonder like when I have children, and if I choose to live here, will they have that same experience?

I don't know. But some of the fondest memories, you know, it was great.

A lot of candy. I would get sick every day. Like the day after Halloween, I would just be sick.

I have a lot of cousins, and they all have children. So we're taking like a group of maybe 12 children to go trick or treating.

So everyone's kind of dressing up as I don't know what what's popular anymore.

So all these kids are essentially dressing up as YouTube stars.

So that'll be interesting. Here, the fashion is Wednesday Adams from the show.

And also the kids, the boys are more on the skeleton side.

Oh, well. Let's dig right into the Okta Compromise. For those who don't know, how can we summarize what is the this Okta Compromise?

Okta, of course, is a security company that provides services for a lot of companies, including Cloudflare.

And what happened? What can we? So let's let's let's take a one step right back before we get into it.

Right. So kind of like you, as you mentioned, Okta is a security company, but like it's their security company.

It's also a little bit more crucial as to what type of security they provide.

Okta is an identity security provider.

So what they do for us and countless other customers is sort of manage the way that we access our internal applications, our external applications, whatever the case is.

Right. So think of it as like you put your applications behind Okta and Okta manages all the identities, like who's accessing what, what permissions and how.

Right. So you can imagine. Right. Like we put some critical applications back there.

Right. Because it's supposed to be safe. We have a trusted relationship with them.

Right. They have attested to it. They have said that, you know, they've shown evidence that like they were complying with all the standards out there as as many companies are.

Right. If time and time again, our request have showed that like the sort of validity of these standards and how it's in their environment and all the security controls that they have.

Right. So for us, the way that we use Okta.

Right. So majority of it is like our employee applications that are sitting behind Okta.

Right. When we need to access it, we will log into Okta and then that will sort of help us navigate into whatever application we need to.

So if I need to access Gmail or Google, I first authenticate to Okta. We have hard tokens of vital hard tokens.

We put in our credentials. We validate with multi-factor with the hard token, takes us through Okta, right.

Authenticate and then takes us to takes me to Google.

Right. So that's kind of in the realm of how we use them.

Although the the hardware keys are not necessarily from Okta. Right. We have this.

Exactly. They're not from Okta. We we implement them. Right. Exactly.

That's our second factor authentication. So now what happened? Right.

You can you can sort of just surmise that, like, all right, if your applications are behind Okta, you probably have Okta administrators and things like that.

Right. So as I mentioned, we have a trusted relationship with Okta. So to troubleshoot various things, we are engaging their support teams.

Right. And the way that we engage the support teams is generally like we submit a ticket on their end.

It goes into their ticket management system. I won't say what they use, but they go into the ticket management system.

Right. And that ticket management system, a customer support lead or a support engineer or something like that.

We'll go through a ticket and figure out what happens. Right. In our case, an IT admin was trying to figure out an issue within Okta.

Right. Because we're consistently trying to make it more fine tuned.

We're consistently trying to make it automated.

Right. Consistently trying to make it more safe. So we do assessments.

Right. And as part of the assessment, generally, what the Okta support team will ask of us is to upload a HAR file.

Right. So will our IT admin grab the HAR file, upload it to Okta?

Right. Can you explain what those HAR files are, just for those who don't know?

Yeah. So a HAR file is generally used to, it's from your browser.

Right. So a HAR file is generally used to troubleshoot issues that you may have within your browser.

Right. You can troubleshoot sort of like sessions.

You can troubleshoot products. You can do whatever, all that stuff. Right.

What the HAR file contains, though, is actually quite important. Right. The HAR file, inside of the HAR file, you may actually have a valid session.

So a session that I already had signed into Okta, my browser captured it.

Right. So that export of that file has my session in there.

Right. But many times that's used to like, oh, your browser is not working.

It's not connected to Okta correctly. Right.

Let's look at what's happening. Send me this HAR file. Right. Let me look through all the settings and configurations.

Give you the status, what is happening there.

Exactly. Exactly. So figuring out, you know, configuration and status and settings.

Right. And that's what it's generally used for. So that happened in terms of from our end to their end.

Right. Right. From our end, we export it out.

We put it into our support ticket. Right. Remember, I mentioned that they have the session information in there.

Right. So we built an established relationship with them, like very trusted.

So we trusted that like anything we gave them is secure.

But what happened is a threat actor or an attacker took that HAR file from our IT admin, from their customer support system.

Right. And they he was able to extract a session using that session from their end, access our Okta instance.

Right. So accessing our Okta instance as an admin will give you access to applications we have there.

We'll give you rights to, you know, if you want it, you can create a new account.

You can disable an account. You can add yourself as give yourself rights to another account.

Right. So imagine like it's like, you know, if you live in an apartment building, right.

Okta is like your super.

Right. So, yes, I have my apartment building. I secure it, whatever. But the super has like the key.

So a superintendent, they have the keys. In cases of emergencies, they can come in, they can fix things and they can do whatever they need to.

Right. So. And they can bypass like certain certain things, like I may have, you know, certain locks in it and things like that.

Right. But my building requires that we allow our super to come in when they need to and they will let us know.

Right. So Okta acting as the super can come into our accounts using that mechanism.

Right. So that access. But in terms of making like customers at ease, no customer data, not that information was passed.

Right. So for this incident, no customer data was impacted.

Actually, we we saw the bad guy come in. We saw him come in.

We were watching them. Right. We saw them sort of move to another account. We saw exactly the path that they took.

So what were they touching in our environment?

What were they doing? What were they targeting? We saw all of that. Right.

And we were tracking them. Right. And then we stopped them. So we stopped them before they can actually have access to any of our production systems, any of our critical information, any any of those things.

Right. So not only do we stop them.

Right. So, you know, call for products such as access, such as zero trust and all that stuff.

Right. Because we have we talk for our own products and because we enable them.

Right. We have Zero Trust session logs. Right. We can see number of bytes going out, number of bytes coming in.

We can see east and west traffic.

So we're kind of tracking this director. We're tracking the identities of it.

We're tracking on the network side. We're tracking on the endpoint side. And all that correlates into like our SIM.

Right. So we're watching this director move.

We're seeing like what things that they're touching. Right. So we're able to stop.

We stop them and start in stopping them. Right. We were able to ensure that we know kind of what tactics they're using.

We know the attack path that they're going down.

Right. And we know kind of like what they what they were trying to target and like what they could do.

So, you know, with that type of visibility.

Right. Now, when we contained it, we were actually pretty certain that like, OK, now this guy's out.

They're not coming back in. Right. Or they're not. So that's quite important.

Right. Of course. Of course. And there was in the process also, like, how did this happen?

Right. This is this seems related to Okta in terms of the problem, the vulnerability.

So this was on Wednesday, October the 18th, that we our security incident respond team saw that discovered that vulnerability, that attack.

Yeah. And you know what? I can I can sort of paint the picture.

And I'm like, right. So on October 16th, I mentioned that our IT admin uploaded a hard file or create a support ticket.

So on October 16th, our IT admin was working with Okta and they uploaded this file to troubleshoot.

According to Okta, October 16th is also the same day that a bad guy accessed our, you know, their portal and then pulled down that hard file.

So they pulled down that support ticket, all the company information.

So essentially what this tells me is that a bad guy was in their environment, essentially watching, just waiting.

Right. For those that hard file to arrive. Yeah. Or anything. Because they were watching the entire customer support system.

Right. And they were waiting for anything.

Right. An RFI. It could have been a ticket. It could have been an email address, whatever it was.

Right. But it's very opportunistic. Right. Of how they got to this is very opportunistic.

But it was not actually targeting Okta. Absolutely. Many companies were impacted by this in a sense.

Yeah. One password. Right. They essentially reported similar things beyond trust.

Right. They were probably one of the first people to report it because their issue goes back to October 2nd.

And what's interesting is that at that time when they reported to Okta, they were like, oh, we're looking into it.

We don't have any evidence of whether or not we're compromised, essentially.

Right. I mean, if I'm misquoting, please, whoever on the Internet, but this is I'm going to go according to their blog.

Right. So the same sort of like TTP, the same tactics.

Right. So essentially what they're saying is that someone accessed our admin credentials or admin hard file from their customer support system.

Same thing. Right. Like we know the exact date based on the timestamp within that file of when sort of we uploaded it.

And according to Okta, we know the exact date that the bad guy went into that case file and pulled down that hard file.

Right. And I'm telling you, it's in the same day and within.

Yeah. On 16th. So the attacker held on to that for a little bit. Right. We didn't see the first activity coming into Cloudflare from Okta until October 18th.

And this was kind of like cross -functional how we identified it. Right. So on one hand, we had an alert.

An IT admin disabled MFA for another IT admin. Right.

That's the alert that we see. So it's in a broader sense, it's like an MFA reset alert.

Right. And so we receive that in tandem. Right. We were we were getting alerts that, you know, one of our sort of our password management system was locked up.

Like, oh, what's happening here? Right. We start receiving those alerts. They're not immediately interconnected.

Right. However, because we saw that kind of in the in the span of the time frame that was happening and the accounts are being targeted.

Right. We have sort of like logic that can tie that together. Right.

So these are two separate nerds that got tied together. And then we're like, OK, these are related events.

And that's happening in the span of minutes. Right.

So these related events are like, OK, what is this telling us? So from immediately from there, we're able to identify which account was being used.

Right.

What accounts are being targeted? Kind of what they did in that targeted account and that targeted application.

Right. So that's when we kind of turn it up.

We engage our entire search team. We came in. Right. We were reviewing Octologs.

We were reviewing sort of like our the logs within our SIM, which is like all the security technologies.

We were reviewing our ADR alerts come to what's going on.

What's interesting is like, no, we like I said, we found the attack path fairly quickly.

We saw them going to different places. But one thing on that day, at least, what perplexed our mind is like, this doesn't make sense.

Like, yeah, OK, fine.

We're contained. We did. We know that they didn't touch anything. But how they get in in the first place, because.

If you compromise just an account and try to sign in, right, our MFA would block you.

Right. So a hard token sort of this is standard across the board, right?

A hard token helps you at authentication. So technically, if you if you just found credential and, you know, the path of our like, you know, oh, Cloudflare, Google, I'm going to access this.

And you just go into the browser.

You can't do it without touching the hard token. So what does that tell me?

That tells me that a session had to be stolen. If a session is stolen, how did the session get stolen?

If it got stolen from our end. Generally, what you kind of see is that for you to steal a session from like my computer.

Right. I guess one of the tactics that you could do is like send me a phishing email.

Right. And then you drop a malware.

Right. A malware drops. And then the malware goes into my browser, takes out the hard file.

Right. And then sends that hard file, sends it out to the attacker.

But all that activity is still happening on my host. And there's still a residue of that.

Right. We should be able to catch. We were perplexed.

I was like, I don't see any host activity. What is happening here? Right.

It was weird, different than usual. So it had to be something on Octus. And it was, as we saw.

Yeah, exactly. So I remember having this conversation and I was having it with our higher-ups.

And he's like, so what did you guys look at? I was like, I'm giving you with high confidence and certainty that everything that I looked at on a host level.

We looked at on a host level. Right. All the technology that we have.

We did forensics. We essentially did full-blown forensics on machines.

And I'm like, I am with high certainty. Right. Because we use a methodology that has time and time again has proven results.

And this is the same methodology that, you know, my background is from DFIR consulting.

This is a similar methodology that we have used for countless other incidents.

Like I've been on APT incidents where Iran is attacking them.

China's attacking them. Methodology is sound.

So I'm like, I'm very confident that we did. We looked at everything that we could.

Then why don't you find anything? And this seems like this is either an identity-based attack or it's all based on the network.

There is nothing on the host.

And I remember saying, I think ACTA is compromised. And the person that I sold to is like, you mean like ACTA -ACTA, like the act of the company?

I'm like, yeah, I think the act of the company is compromised.

Right. So this was on the 18th night or 18th, like for me, because I'm in New York City and the rest of the teams are either in California or in other places or Texas.

So for me, it was like dinnertime.

And I remember saying that and I'm like, we need to talk to ACTA. So we had our, we raised a support ticket.

We had our contacts reach out to them. The next morning, early in the morning on the 19th, we had our first call with ACTA.

We're in that call and we're talking to a support engineer.

I was like, what the hell?

What's happening? The CISO comes into the call and he's like, hey, we have to tell you something.

We were compromised and they accessed your account from ours. And I'm like, oh my God, could that make sense?

Right. Because I'm here racking my brain.

I'm like, I don't understand. So what happened? Exactly. That explains, right?

That explains it. Yeah. It's a bit of detective work in a sense, which is interesting, especially when things don't work well.

Like we were not compromised. We mitigated attacks.

We spoke before and you were telling me the other day that it's not a matter of companies being the target of attacks.

Attacks, vulnerabilities, they will happen.

It's a matter of the protections you have in place for those vulnerabilities or attacks could be mitigated.

So you won't be as impacted as you would be if you hadn't the protections in place.

And in our case, you mentioned Zero Trust in terms of finding and seeing the path of an attacker.

But also very important, the hardware keys, like a second layer.

And we at Kaufler, everyone knows that has to put his finger on the hardware key every now and again to enter certain applications.

So that played a central role here, right? Yeah. I mean, so let's touch on that for a little bit.

So I'm going to work backwards. So the hardware keys, yes.

So with the hardware keys, we can say that, okay, this is not just credential stealing.

You have to steal a session. That's the thing about hardware keys.

They protect you at authentication. But then if somehow you're able to grab a session, generally to grab a session for a user, you have to do a man in the middle attack and grab that session.

And then your other controls also come into place.

So whatever kind of EDRs you have in place, whatever sort of endpoint controls or network controls that you have in place, that's where it comes in place.

But essentially, you would have to compromise Kaufler first, technically, to get a session for a user.

Or you would have to compromise their systems while the hardware keys are in there.

So after you authenticate with the hardware keys. So it gives us that initial layer.

It gives us that protection. So we know that, okay, it's not just a stolen credential sign in.

They've popped a session.

Now, we start developing hypotheses around that. How did you pop the session?

You can't sign into Kaufler stuff without a Kaufler sanctioned device and without the hardware token and all that stuff.

So if you popped a device that belongs to Kaufler, we also have other things that give us visibility onto it.

Whether that's our security technologies, whether that's warp, whether that's whatever the case is.

So I should be able to see. I've been told by some pretty smart people, bytes don't lie.

Everything else can be covered up, but bytes do not lie.

So we would see bytes coming in or bytes moving out. So we would see some level of activity.

Movement, exactly. We'll see some level of movement that we can say, oh, this is anomalous.

Suspicious, yeah. Suspicious, right? So once again, drilling onto that fact is that, hey, you have to secure yourself every way, which way you can.

But the point that you brought up based on our conversation we were having, it's not about if you can stop being attacked or hacked or compromised.

It's about when you become compromised, but really it comes down into what is your resilience?

We were able to show our resilience that, okay, yes, we got compromised.

We talk about this concept of dwell time, that an attacker is in your environment for sometimes very long.

They can be in your environment for months or years or whatever the case is, and then they do action objective.

But this incident for us is a testament that our dwell time is very little.

Because we had all of these controls in place, we were able to see it, prevent them from doing other things.

So the resilience of going back to normal was much faster.

We did all that stuff within the first day. In organizations, we need to think about, oh, yeah, I'm not going to prevent people from coming into my door.

I'm going to make it very hard for them to...

We used an apartment example before. So it's like, yeah, I don't know, if someone decides to scale 20 floors and come into your house, that sucks.

But then after your most crown jewels and you put them into a safe that's locked and then put that safe somewhere else, that's one more layer that is not going to make it easy for you just because you're in.

Many organizations focus on the perimeter.

But then once you bypass the perimeter, it's a straight path to take anything you want.

And that is something that we as an industry have to really focus on and understand that.

It's multi-layered, so that's just one thing.

Absolutely. I'm showing now the blog posts you wrote about how Cloudflare mitigated yet another Okta compromise.

The yet another is because in March 2022, there was also a breach on Okta systems and Cloudflare was also impacted.

And at the time, like in this day, in this situation, we concluded that no access was done by the threat actor in terms of systems and our data because of our use of multi-factor authentication.

So that was March 2022. And this situation was the second one.

But again, in our end, no impact in terms of our systems and Cloudflare data, which is always important.

And that's important. We pride ourselves to ensure the best good customer service to protect our customers' data.

So even if you think about the Okta situation, I'm surprised that they're not watching.

That's a critical application. We give some recommendations to Okta and Okta's customers.

Yeah, we give recommendations to Okta and then to their customers.

But I mean, let's touch on some of the recommendations for Okta. By the way, it's not finger pointing or anything like that.

This is so that our interaction with Okta is safer and then other interactions with their customers are safer.

We're all working within the same industry. We want to make sure that we can operate with a level of trust and certainty.

But I mean, one thing that happened that I want to note on, Okta didn't tell us about this incident until we reached out to them.

Maybe they were waiting, maybe they were waiting to get more information.

But they knew that Cloudflare was targeted. And they knew that all this stuff happened.

It wasn't until we've created that support ticket, it wasn't until we had that meeting with them that they're telling us that we're attacked.

Mind you, we're scratching our head the day before like, how'd they get in?

This is weird. And in our end, we wrote this blog post on the 20th of October.

So just a couple of days after we saw that happening. And we waited until they released the public statement.

So I think that's also important. We're not in the business of calling out other vendors.

But we definitely want to add more context of what happened.

And how did we react to this? And how do we protect ourselves from this?

So that others can learn from us. Exactly. And communication externally for customers, even for others in the industry, is definitely really important in this case.

I think we mentioned briefly that we also introduced this HAR sanitizer.

So trying to put actually a system in place to avoid these types of files to be the problem that happened, right?

Yeah. So that's something, right? If you can trust, you can have this layer of trust with whatever vendor that you want.

And you fully expect them to put that same level of trust and the same level of careful certainty in all the interactions that you have with them.

But in this case, we mentioned that HAR files, although it has a good definition of what kind of HAR files are, how they're used.

But HAR files do contain sensitive information in it.

Sometimes that sensitive information is required to troubleshoot a thing, but many times it's not.

And we should not be including that into any submission that we do.

So what we did is we developed a tool that essentially helps you sanitize that HAR file.

So using this tool, if you were in a situation where you needed to update something to Cloudflare, or you need to upload something to Optum, whatever you want.

You choose the HAR file here, it will go through, find any secrets in there, sanitize that secret, and then send it to whoever the recipient is.

So then when they have it, if someone else was to compromise it, well, you can't use it, there's no active session.

It's amazing, not only because it was done in so little days to put out there, but also it's available not only for Cloudflare customers, but to everyone that wants to use it at no cost.

And it's under an open source license in this case. So it's for the use of those who want it, right?

Exactly. So anyone can use it, whoever needs it, they can take it.

It was a great work here in terms of short notice.

The team really put effort first in finding the problem and mitigating it, and then putting actually a solution in place.

The lessons learned, you would say, of this situation?

We think alike. So I was just going to touch on some of the mitigations that we put in place, and then lessons learned from those mitigations and how to do things better.

Let's be honest, this incident is the actualization of a third party risk.

It's like you as a company or us as Cloudflare can take every single measure to ensure that we protect ourselves.

But if our partner organizations are not taking that same measure, and any sort of interaction with that partner organization, if that's not taken into account, and if that's not protected, then this fate is going to continuously happen with everybody.

We all talk about third party risk, third party risk, third party risk.

We've seen that with SolarWinds. This is a supply chain attack. People from SolarWinds attack whoever else.

But this is very similar, this is exactly that. This is that you attack the supply chain, you attack a critical vendor, an identity vendor, and then from that identity vendor, you move on to the customers.

Because you know that there is a connection from that end into your instance.

But there's multiple things that we need to consider. Some of the mitigations that we took is the actual session lengths.

We shortened that significantly.

So that even if a session exists somewhere, it's going to be expired if someone doesn't use it immediately.

Additionally, if you're going to access certain things.

So yes, we have our tokens. But organizations need to think about what other layer of protection or what other layer of authentication do I need to add.

Yes, you have hard tokens, but what if to access Okta as an admin, you have to authenticate with your hard token and then authenticate again.

So if one works, basically this attack is in between that initial authentication and then what happens afterwards.

But if you add another layer in there, another layer of protection, that's one more hurdle that they will have to get over.

I never sit here and say that you will mitigate the attack 100%.

Or you will stop the attack 100%.

It makes it much harder. Exactly. It creates barriers for those who are attacking.

Exactly. So creating that barrier. Turning up kind of... Like I said, we caught the attackers doing that MFA action.

But there's a lot of work that our detection engineering team has done to layer those types of alerts quicker.

So we obviously use this concept of machine learning that you put in together.

Let's say that based on the timestamp that's happening, based on what is being targeted, these are associated alerts or tickets.

What our team is doing is expanding that concept into all technologies.

So essentially what will happen is that the correlation is going to happen even faster.

And I'm a big proponent of using automation, but don't let automation kind of close your tickets for you.

What we did in this incident is the enrichment of all that data. A lot of it was automatic, but a lot of it was manual.

But now we kind of know multiple data sources to tap into.

So when we receive an alert, I'm going to have all the information much faster now.

So that was lessons learned from there. The other lesson learned is that we rotated a lot of creds.

And when we were impacted, we're like, okay, these people are impacted.

Their password manager might have been touched.

We don't want to take a chance. Let's rotate them. But rotating anything in any organization will tell you, if you rotate a service cred, you might take them to service.

So maybe it creates a few minute delay. Like, okay, let me just double check.

But even on the mitigation slash containment side, we're adding more enrichment.

So that we kind of know, okay, this service credential is being used in all these applications here.

And we have that information, but that's like, all right, you have to kind of correlate that yourself.

But we want to automate that. So when it comes in, it's like, oh, I know exactly.

If I rotate this, this is going down.

So how do I say that? Well, I'll click on this, and then we'll generate a new token, push it in.

You want to use the computers to do all that stuff. So that's another kind of lessons learned that we learned from this incident.

So we're doing all that as well.

I mean, what else? I hate to say it, they were not successful in targeting or exfiltrating data or doing anything like that for us.

So we're lucky. So a lot of the other things that we will be doing is essentially a continuation of what we're doing.

I mean, I would definitely tell organizations that always look out for a few key things.

So any unexpected password and MFA changes in your Octa instance.

If you don't have detections for that, make sure you have detections for that.

Any kind of suspicious support initiated events. So if your IT admin is doing something strange, watch the watchers.

That's important to watch.

Any kind of password resets, make sure that it's valid.

For us, if you reset your password, it has to be a ticket.

It can't just be a random password reset. But make sure that that correlation exists.

Make sure that you're monitoring any new Octa users created.

In this attack, what they did is from Octa, they used the API to manipulate people's Octa instances.

And they're creating users using one account to create another account.

I've read a few of the blog posts as well. But watch for that.

If you're not watching for that, watch for that very closely. Is there any new user that gets created?

At least for us, if there's not evidence of why it's being created, and a ticket's a reference to why it's being created, then that's not valid.

We would shut that down. If that happened, we would automatically suspend that user.

And you can shut that down and ask questions after, right? For safety purposes.

Exactly. Shut it down first. My strategy, a lot of it is kind of like, maybe I'm American, that's why.

The cowboy approach. Shut it down, and we'll figure it out later.

Shut it down, disable them, suspend them, do whatever, and ask questions.

It's like that. If it was legit, someone from the company, sorry. Your access is backed on.

If not, you stopped something. Exactly. It's important.

We talk about Zero Trust. Zero Trust is not just a computational concept. It's not just a system concept.

It's also a mindset. With that mindset, we have automations that will suspend a user based off of a certain activity.

We will suspend them.

If it's legit, you should have a ticket. You should do the proper process. You missed something in the process.

That's why you got suspended. We have to be critical about it, and we have to take that into consideration as we're designing our detection and response programs, as we're designing IoT, as you're designing the overall security program.

It's quite interesting. One last note I would take is from everything online to one little hardware key that will make a difference.

Something on the real world that a user connects and clicks with their own finger to something that is online, those two combinations help here.

Or the two combinations of a lot of automations and an automatic system, for sure, those are really important.

But also some manual curation. Hey, this is weird. Why is this happening?

Those two combinations, manual, automatic, virtual hardware, are interesting as a concept, I think, to be safe.

Most definitely. I hate to say it, there's not one silver bullet to anything.

If you think there's one silver bullet, you're in trouble.

Especially in hardware tokens. Hardware tokens is one aspect of it.

But in order for you to make hardware tokens effective, you have to have pretty good security controls everywhere around.

You have to know what does good look like in your environment.

What account should actually access certain applications as an admin?

You have to know all these things. Even if you're alerted about it, by the time you figure out everything, it's too late.

Yeah, the problem is there, and the attacker possibly had access to something that is damning to you.

Yeah, exactly. This was great.

Thank you so much for your explanations, your technical explanations. Have a good Halloween.

Happy Halloween, everyone. And that's a wrap.