Hacker Time

All right, we are live. Welcome to Hacker Time, the number one security show, not just on Cloudflare TV, the number one security show anywhere out there.

Today, we've got a jam-packed show, planning to talk about secrets management for a bit.

I'm walking through an old talk that I gave at Besides SF, and this is a few years old for the talk, but all of the information is still super, super relevant.

And actually, that's an interesting data point since it all still checks out today.

And we'll walk through kind of the details of that and what just the basics and the most important pieces of secrets management.

We'll check out the vault docs and some of the technologies, their docs and the details about them and what's important to know about those.

And then if time permits, we will play a chess puzzle or two, maybe.

And that's about the show. So I'm really excited to get started.

Hope everybody's wearing a mask and staying safe as the coronavirus continues to get worse.

And I'm gonna be staying in this weekend watching the PGA Masters Tournament at Augusta, Georgia.

First time it's ever been in November. So I'm gonna be staying safe indoors, wearing a mask on my couch.

So let's get started. The talk that, I actually gave this talk with Maya, who is, she works at GitHub right now.

And it was a really great talk. First of all, it was a great talk because besides SF, this was the first year where it was at the Metreon in San Francisco, where you're on a movie screen.

So our slides were 50 feet tall. It was incredible to present in that kind of glamorous fashion.

And also the rooms were really small.

So there weren't many seats because it's in a movie theater and it's super comfortable.

So the rooms are just jam packed with people, which was really cool.

So you kind of feel like with the big screen and the stadium seating, you feel a little bit like a big, like Jay-Z or something.

So it was really cool. But the content of the talk, the environment was great, but the content of the talk was also great.

And it starts with why is secrets management important? And this, we put together this collage slide because it really just shows that most of the breaches happening today are related to stolen credentials, stolen secrets that people accidentally leave lying around without usually not any complex software issue.

It's usually just a best practice that wasn't followed and then the secret got made public somehow.

So here are some examples. All of them kind of have to deal with cloud services as well.

So how I've generally seen these breaches take place, somebody has a credential in their source code that the source code gets made public somehow or somebody gets access to that secret.

If it has a lot of privileges, it can talk, if that secret can be used to talk to a lot of different things.

So say in AWS, if it can be used to pull down data from a S3 bucket, all of that data gets exposed when a hacker discovers that credential.

So it's really important to have some policies and procedures around secrets management because when you allow developers to put secrets just anywhere in their software and some of that gets exposed, you can't really manage the problem.

It becomes really unmanageable to make changes, to understand how risky an exposure might be.

And it's just the security gift that keeps on giving every couple of weeks you hear about an exposed S3 bucket.

Since then, since this talk was given, the big one that was in the news was the Capital One issue.

And that was a really serious issue of slightly more complicated.

It wasn't quite secrets management, but it was an S3 bucket and it just constantly happens.

So how are secrets typically managed? And Maya made this slide and I think it's really spot on.

There are two methodologies you could embrace, which is decentralized.

Just have your secrets stored right in the code, right next to all of the rest of your code.

It's no different than any config that you have baked in.

And this is basically completely unmanaged and it's decentralized because your code might be, your secrets might be in 50 different repos.

And so your code is secret, essentially. Centralized, meaning all of your secrets are in one very specific secrets vault.

So you're using some secrets management tool that was purpose -built specifically for managing secrets, where it has all of these desirable security properties and that's different than your code.

And then, wait, people actually keep secrets in code? Totally, they do.

Everybody does it, just nobody talks about it. A lot of people do put secrets in their code, me included at times.

It's a lot easier to just put the secret in the code and worry about it later.

It's really easy to feel like, this isn't really a secret, and then get in the bad habit with some things that are less important.

And then as time goes on, you're more and more likely to put more and more security critical things in your code.

And also when you're a startup, it's just not high on your list of things to worry about.

If you're a really tiny startup, you've got to worry about going out of business.

You've got to worry about finding product market fit, shipping regularly, shipping quickly, and your secrets management just is like probably problem 1,000 on your list.

So common mistakes made in secrets management.

These are interesting because some of them are more common today than when we gave the talk.

And I thought Maya had great forethought including this in our talk that we gave together.

So putting secrets in code, don't do it because it's just becomes an unmanageable problem.

And so that's obviously mistake number zero.

Mistake number one, zero, whatever, zero index.

Not rotating secrets. This has become way more common than it used to be. Rotating secrets is something that most cloud services are starting to have to think through how people rotate their secrets when they give you an API key.

So most of these secrets that we're talking about are usually either cryptographic material.

So something for encrypting and decrypting something really sensitive in that regard like a private key used for TLS.

Or it could just be, could be an API key to AWS, to Cloudflare, to any cloud service.

And most cloud services are starting up to think through when we give a customer an API key, what are the properties of that API key so that they can rotate that API key safely without downtime, without any availability issues?

Should they expose it? Should they need to rotate it for whatever reason?

I think Stripe does a good job here where when you deactivate your Stripe secret key, it's still good for four hours, I believe, or when you generate a new secret key, your old one's still good for four hours.

So you have time to go restart everything.

You put your new key in your secrets management system, you restart everything and you still have, you don't have any downtime or you're still processing payments throughout.

So rotating secrets is something all cloud services should be thinking more about whenever they're building their API features.

Not backing up secrets. This is a really good one. What would you do if you just lost all of your API credentials?

Think of, especially in a centralized manner.

So talk about in Git, in actually not managing your secrets.

This is a little more recoverable because if you don't manage your secrets and you put them all in code and it's all scattered throughout, you always have it in Git history.

And in Git, if it's in your Git history, it's also stored locally on your machine.

So if you have a secrets management system, like one central server where all of your secrets go to, you have to be sure that you're backing that up because if all of those just suddenly disappear or you lose the keys that unlock that secrets management server, you are in a load of trouble availability-wise because nothing will be able to start.

Who knows what's gonna, like everybody's situation will be different should that happen, but you're in for a bad day.

If you, just like a database, you lose your production database and you don't have a backup, you're in trouble.

Not having the concept of identity. This is the one that's the most important to me because a lot of people talk about doing secrets management and doing it well.

And they talk about also, I think it's no different than role-based access control or any identity and access management problem.

If you're going to restrict access to something and you're going to make sure that only the people who need access to something should have it, you really need to know who, you need to have a strong sense of identity first for each service getting secrets and each person in the environment.

So if you boot up a service and it's in a Kubernetes cluster and it's supposed to be talking to your secrets management system and pulling down only the secrets that it needs, you really need to be able to identify the each service in your cluster separately and then be able to restrict access around each secret that the service needs.

So there's a lot of prerequisite work around identity before you can have a strong sense of security in your secrets management system.

And I think that gets overlooked. Ian Hacken of Netflix gave a great talk.

I believe also at B-Sides SF. Potentially it might've been an Enigma conf, but he did a great talk on this where he kind of said, it's turtles all the way down.

How do you identify? What is the root of identity? And he was operating in the AWS environment where it's a role, an IAM role is your identity.

And it gets a lot harder, even on bare metal, you have to go all the way down to TPMs and your chips and your hardware level.

So AWS makes that a little bit easier, but it's still identity is a key prerequisite.

And if you're looking for that talk, Ian Hacken is H-A-K-E-N is, I believe how you spell his last name.

He did a great talk on that subject.

Protecting secrets the same way you protect everything else. There's secret.

So common mistakes made in secrets management. What was she saying here?

Common mistakes made in protecting secrets the same way you protect everything else.

I'm not sure. We've got the notes here. What did she say? I'm not sure.

I think secrets have, the point of this bullet point I believe is that secrets need more stringent controls around them, not, and because of what they are and how sensitive they are, it's really important to have extra controls around them, to have something that is specifically for secrets, not just a database of, it's not an Excel spreadsheet with secrets in it.

There are special properties that you need.

Oops, my slides are going crazy. There are special properties that you need in your secrets management system to make it good.

So what are those properties?

One, the prerequisite is identity. Two, auditing. I think we talked about this last week.

We were looking at enterpriseready.io and auditing is something that everything needs.

In security, everything should have an audit trail, a very robust audit trail.

That's really important. And then these are the critical ones.

So encryption. One of the key differences between a secrets management system and an Excel spreadsheet is everything in the secrets management system is supposed to be encrypted.

And you want it to be encrypted before writing it to disk.

And you want it so that anybody could have your encrypted secrets vault and be unable to recover it, just like any other secret message.

You're sending over signal or something or a encrypted backup.

Whatever it is that you want it to be encrypted, just like any other secret that you might have.

Rotation.

This is, we touched on this just briefly, but at the time this, when we gave this talk, I believe in 2017, this was kind of a new idea in this world of secret management where Vault had kind of built one of the key technologies we'll talk about in a moment as well.

And they kind of started adding support for this, but now this is kind of considered, kind of considered a must have in a lot of environments because one of the risks of having a secret that is static for so long, like an API key to AWS or whatever it is, is you can start to misuse that.

You can copy it and use it on your local machine.

You can expose it somehow and it sits in logs for a long time.

And then someone's able to find those logs. Like when you expose it, there are so many, when it's such long lived, the chances of it being exposed go up and up and up.

But if you're constantly rotating, the risk becomes a lot more manageable.

So they're starting to add, a lot of people are starting to be rotating their Postgres passwords all the time, their AWS credentials all the time, all of these really important things.

And that's part, like you need rotation in any system that you're building today.

Isolation, if I'm service A and you're service B, you need to make sure that only service A can see service A's secrets, only service B can see service B's secrets.

And that one's pretty straightforward. The whole point is that you have strict access controls.

And so this is where I think things get really interesting.

So depending on the environment your code runs in and depending on who your company is, what they do, what their goals are, what their future is, what their current systems look like, really dictates how you should be solving the secrets management problem at your company.

And so let me first talk about HashiCorp Vault.

And HashiCorp Vault is kind of, at the time it was becoming ubiquitous as the thing that everybody turns to for their secrets management system.

And actually at Cloudflare, we're using this internally. We have some really great folks working on this on our core operations team.

Shout out to them.

They're doing a great job and working on this system for us at Cloudflare.

But Vault is really heavy duty. And so despite this being ubiquitous, it still might not be the solution that you need for your company.

And it definitely is the right solution for us at Cloudflare because, well, I'll talk more about that.

But the thing to note about it is if you're rolling out Vault, it is heavy duty.

The way it works is it's a big encrypted data store. It can store your secrets in encrypted files in many different ways.

You can protect all of your secrets in many different configurable ways.

You can make it so you need five people to open your secrets vault.

You can make it so you need a specialized piece of hardware if you get their enterprise version of Vault.

But the thing that's really important, and it does have all the best features you want.

You have many different ways of backing the identity, which I was saying was critically important.

So you could use an AWS role. You can use a PKI certificate. You can do so many different things.

And all the properties of Vault are really good.

Just the thing to note is maintaining it, standing it up, it's a lot of work.

And unless you have people and a team working on it and operating it and owning the system to make sure that it's up, it's backed up, it's configured to be highly available, it's configured to be graceful when it restarts.

Unless you have people who can work on it and invest in that, it's probably not for you.

So if you're a big enterprise and you have systems in the cloud, you've got systems on-prem, you've got systems all over the place, you've got a dedicated ops team, like Vault's probably the right thing for you.

But if you're a little startup, if you're in the cloud, if you're fully in the cloud, if you're, I mean, just those two things.

If you're either of those two things, Vault probably isn't for you. So Vault is a great one.

And then in AWS, the world of AWS, they have a few different ways.

One is KMS, Parameter Store, Secrets Manager. There are these three ways, and under the hood, they're mostly the same thing, but with different packaging around them.

But the way, people have solved their problems on AWS in different ways, but if you're running all in on AWS, you probably want an AWS -tailored solution for secrets management, because it's native rather than standard.

So if you're setting up something like Vault, and so Parameter Store is one method, Secrets Manager, they have this feature.

At the time, it was new, but Secrets Manager, and then KMS.

Under the hood, it's mostly all the same thing.

It's just using a role to give you access to only the secrets that you need, and it's pretty straightforward.

I won't spend too much time on that. So here's what the Secrets Manager looks like.

Here's what Parameter Store looks like, and there's a lot of good software out there to do that.

And so GCP, if you're in GCP, here's the theme.

If you're all in on a singular cloud service, like they probably make, they have features for you to store your secrets there.

So GCP, it's the same thing.

If you have a bunch of Kubernetes plus GKEs running in GCP, they make it very easy for you to protect your secrets within GCP.

You might not want to have your secrets elsewhere.

So if your entire workload is in GCP, you've got some risks in terms of having a singular hosting provider.

You might be vendor locked in, and the secrets aren't really the thing keeping you vendor locked in with GCP or AWS, but they probably are what you want.

You probably want to keep your secrets there as well.

Okay, so there's a bunch of stuff here. Kubernetes, this is an interesting one.

I actually know very little about Kubernetes, and I haven't gotten a chance to play with it too much, but Kubernetes integrates with a lot of different things.

It's extendable, and there's many different ways to do it in the AWS or GCP world.

You can integrate it with those secret systems there in the Vault world.

If you're running a Kubernetes cluster on-prem, you can store your secrets in Vault.

If you're running a Kubernetes in Vault instance on-prem, so it's pretty extendable.

And yeah, I thought this talk was really good, and I'll bring it home with the things that really matter.

I would say that I've seen a lot of companies say that they need secrets management when they aren't ready for secrets management because the prereq is identity of your workload, and that's way easier.

Identity of your workload is way easier in the cloud than it is on -prem.

So in the cloud, you just kind of press a button in GCP or AWS or whatever your cloud service provider is, and it tells you the IAM role of your service.

And so it's really, really easy to start building a secrets management system around that.

On-prem, when you have a machine, like an actual server, you don't want to use an IP in these days to identify that workload.

You probably, if you've kind of embraced the DevOps world where they're cattle and not pets, then you definitely don't want to do that.

You want to have all of these DevOps ideas in your system where it handles restarts gracefully, it handles shutdowns gracefully, it handles scaling up and down gracefully, and no different than handling all that is needing to handle identity, which is a challenge when you're running on-prem.

But there are many different ways to do it. People use PKI certificates, people run a, like, there's a million different ways to do it.

But before you go down the secrets management path, if you take away one thing from this 30 minutes, it's you should ask, have I solved my identity problem of my workloads?

Auditing, super important.

Who saw what secrets at what time? So important that if, in case something goes off the rails or in case something breaks and you want to know, why did, why did, why am I suddenly getting authentication failures?

Why can't my service talk to AWS anymore?

Well, it looks like somebody changed the secret here.

So audit logs are super important for any type of forensics or anything like that.

Encryption, super important as well. You want your secrets to be encrypted and protected from, when they're at rest, should somebody recover your secret vault.

And that goes for backups too. You're going to want to be backing up your secrets vault, and you're going to want that to be encrypted as well.

So you have to think through the recovery, but also you have to think about how the data is protected and where those keys are stored, where the keys to unseal your secrets vault are, which is another key itself.

So you kind of end up in this turtle problem when it comes to encryption.

And there are ways to solve that as well with HSMs and specialized hardware in the vault world.

And you don't really have the same problem in the cloud world, but that is an important property of secrets vaults that you don't get from an Excel spreadsheet or just from a MySQL database with the table name is secrets, and there are some extra restrictions around it.

You've got to make sure that you have this control in place. Rotation is the big one that is new and really up and coming in the world of secrets management.

It was in 2017, and I believe it's fully arrived where now you're just better off never having a static secret.

So instead of giving a system a static secret that won't change ever until you change it, why not give it a secret that's only good for a day or 24 hour, an hour or five minutes just for the time that it needs it.

And when it needs another one, it requests another secret. And that way you just never have this problem of exposing your secrets to source code and ending up in a situation where that source code goes public.

So instead of even doing this whole static secret rigmarole, just start with a completely rotated, completely ephemeral secret that just is no good after five minutes.

And this is just never a problem.

And then isolation and identity so tightly coupled here where to do the isolation, you need identity.

So that when I'm service A and you're service B, I can only see my secrets.

You can only see your secrets. I can't see yours. You can't see mine.

And that's super important from a security perspective for isolation of should something go wrong.

If my code, if I'm service A and my code has a bug in it, I'm not exposing you and we are out of time.

Thank you so much.