🔒 Security Week Product Discussion: Securing Cloudflare with Cloudflare

Hi everyone, and welcome to this Special Security Focus segment of Cloudflare TV. My name is Jacqueline Keith and I oversee the security engagement portfolio here at Cloudflare.

In celebration of security, we Cloudflare's Innovation Campaign highlighting exciting security related developments in our products and services, I'm speaking with two of my friends and colleagues about one of the unique ways that the security team contributes to product improvements.

Every company has a security team, but not every company security team allows all those products to be tested by them.

So Molly and Evan, would you like to tell the audience a bit about yourself?

Yeah, I can start.

My name is Molly.

I am an engineering manager on the Security Strategic Projects team focused as one of the responsibilities on thinking about how we apply our own products internally to strengthen our security posture.

And my name is Evan Johnson.

I manage our product security, application security and enterprise security teams here at Cloudflare, Security Engineering Director.

Good.

So today we're going to talk about a few exciting developments. But one of them is a concept called dogfooding.

And so eating your own dog food or dogfooding is the practice of using one's own products or services.

And many companies have product testing divisions.

But since Cloudflare products are security focused, we've taken it upon ourselves to be a major player in the product testing environment.

Evan, can you tell us about the history of dogfooding at Cloudflare?

Yeah, I'm happy to go into all the gory details, and that is the most important first point about dogfooding is it's not pretty.

Eating dog food is a sloppy endeavor.

I've heard people talk about dogfooding like they're drinking their own champagne or that's.

I wouldn't complain about that.

I mean, it'd be great.

But you're you're drinking a finished product and that's not dogfooding.

And so dogfooding has always been kind of a little bit of a thing at Cloudflare as long as I can remember, having joined the company in 2015.

And then it really took off under our old VP of Engineering, somebody by the name of Ben Fatty who – shout out to Ben if you're watching this, who sometimes still sends emails to a lot of us.

But he under Ben, we really took to heart that if we were going to dogfooding – if we were going to offer these products to customers, then we should know firsthand the pain points.

And he shared really personal stories from his time at Microsoft where they were dogfooding Exchange and back in...

I think this is around 2001 when this was happening and his personal stories were were awesome.

They were basically the whole Microsoft. All of Microsoft would just stop getting emails when Exchange stopped working.

And that kind of made the product go from completely not working to they have their skin in the game and the Exchange team got their got their house in order.

I don't know if all of those details are true and correct, but at Cloudflare, the history is we've always kind of dogfooded our DNS products and a lot of our baseline Cloudflare products.

The proxy, Cloudflare has always stood in front of cloudflare.com for denial of service prevention and all of all of this basic DNS, all of that stuff.

And then where our security team really got involved is a lot of the newer security products we've been releasing since Access in 2018.

We really said this is the product that we are going to dogfood to the fullest to the extent that we can try to.

Try to use it, try to make it better, try to figure out what it's lacking that customers like us need.

And we've tried to position ourselves as the first best customer of that product especially, and more security products to follow.

I had no idea that was the full story.

I know I asked the question, but wow, it's more interesting than I thought it was going to be.

So, Molly, have you ever been someplace that dogfooded had before you were at Cloudflare or what were your initial thoughts when you kind of learned about this coming to the company?

Yeah, I'd always heard about the practice of dogfooding, and in prior roles, I'd definitely seen tools that had been built out that were kind of already in the hands of customers be used for like more thorough internal tooling, but there wasn't necessarily as formalized of a process in terms of giving feedback or bubbling up feature gaps or anything like that is kind of inflicted upon you when you're part of the test group and are experiencing dog feeding to the fullest extent, like Evan is suggesting.

I think my biggest impression of being at Cloudflare is something that Evan also alluded to, that we're in just a really unique position by being a security team at a company that builds security products.

So that also means the stakes are a little bit higher for us in some ways when we're using our own products to actually better our own security posture.

That means that we fully believe in their ability to protect our company as much as we believe in their ability to protect our customers.

And maybe I'm getting a little bit ahead of myself because I feel like what comes first, right, is the dogfooding, testing the product while it's kind of like undergoing the actual engineering effort to make it into a general release.

But then also, ultimately, it's not just dogfooding, it's how we use those products in GA to better secure our posture as a company.

So I think it's really like an exciting space to be in.

I feel like this is a company where people really respect and understand the value of security because of the space that we're in.

And so people are excited to be able to kind of get the feedback that our team has being that number one customer and kind of being able to pick their brain for thought.

So this is definitely the most unique environment I've been in regarding dogfooding.

Yeah, it really benefits us when I speak to customers sometimes and they want to know about product experiences or sometimes they ask for customer referrals and we're able to say to them we actually use it ourselves so we can tell you some of the pain points.

And as painful as it sometimes is to do it, it is really helpful when you're talking to customers and you know exactly what they're talking about because you've been through it and you can verify that maybe some updates have been made or something similar has happened that way towards empathy and highlighting the importance of all the features of the product.

I also think you bring up an interesting point when we talk to customers, right?

Like it's not just the technical solution, it's also the process oriented solution.

How did you roll this thing out? Right?

How do we make sure that our users kind of like understood better how to use this new tool or maybe ensure that it didn't interrupt any of their flows?

So being able to speak on the process side from that dogfooding experience also puts us in a really valuable position where we can speak from experience.

Yeah, yeah.

So we've blogged about some of the products that we've done, but today we wanted to kind of drill down, I think, on one particular product and it's just so applicable to the current work from home slash hybrid workplace world.

I mean, can you tell us a little bit about Cloudflare Access?

Yeah.

I can give you the whole story there too. So Cloudflare Access.

We've been using it extensively since 2018, as I mentioned, and it's grown a ton from being something that used to be called edge off where the code is open source, you can go find it on GitHub.

It was about 120 lines of Lua that we ran in Engine X and it's grown over time to be something a lot more robust.

And in 2018, we made the decision that we really wanted to focus on improving our identity and access management posture, everything from our identity provider to groups and roles, to having one unified solution with the hopes of eventually transitioning fully to a zero trust type of architecture.

And we learned a ton along the way using the Cloudflare product, and that we were building as we kind of flew the plane.

And Molly mentioned this earlier.

We are trusting ourselves to build a secure product and offer it to the world.

And really have a lot of skin in the game when we're protecting ourselves with that.

With that, because we're not only protecting our systems where protecting our customers systems by extension of that.

So with access accesses and identity aware proxy that authenticates users logs in with identity providers allows you to make expressive policies over which services certain users can reach.

And over time, it's grown to include Secure Web Gateway, to include DNS filtering, to include full...

There's a few partnerships on the EDR and the ADR space that the that the product focuses on and a full on client running on all the machines, doing a secure connection to all of our authenticated connection to all of our systems.

And so they started with this identity where proxied that you can see.

And then if you read some of the early Beyond Corp papers about what Zero Trust means, they are building out the full suite around endpoint protection and trusting your endpoints.

So it's a really fascinating thing to be a part of building from scratch, but also as an implementer of that at Cloudflare and as somebody who is supposed to be helping secure that product while they were building it, it was really cool for me to be able to just call Sam, the product manager for Access, and Kenny on the phone and ask for different features and say, Hey, we really need X and we really need an answer to how do we make sure that it's only Cloudflare Machines accessing our systems?

So it's been really fun to work on that and to and it's kind of nice that it just helps us better our security posture in the process.

Yeah, that makes sense.

So for a case overview of this, so that's the product and now we're kind of thinking about the process of dogfooding in the product.

So Molly, this was a pretty significant undertaking for the team and I remember being part of the initial group and having to learn pretty quickly what it meant to be part of a dogfooding group, right?

Like sometimes something doesn't work.

Sometimes you have to call somebody, sometimes you're in the middle of a presentation and something goes sideways and you have to call it to remedy.

So I think there's a lot of labor of love that goes into being part of a dogfooding group.

Can you take us through the decision making process and highlight some of the major findings when we decided to undertake access as a dogfooding product?

Yeah, I think Evan's tenure kind of covers a little bit more of the Access side, but when I joined was sort of the rollout of Gateway and Warp on top of our existing zero trust suite and capabilities.

And that was for all the more visibility right into our work from home setup to try to mirror it as closely as possible to the security guarantees that we had when we were in the office.

And I think it's a really helpful case study for thinking about how we conduct successful dogfooding and rollout.

So I think I think in part, right, those products already existed.

So we went ahead and rolled out what was already GA and available to customers.

But then we started to kind of dogfood more features and things like that on top of it to kind of test the cutting edge of what was being created on top of these products.

So some lessons learned from dogfooding. I think communication is really, really key.

We have a great IT team and enterprise security team who has a very strong sense of protocol when it comes to making sure that we're informing employees of kind of what is changing on their machines, what they can expect might be interrupted, whether they should see no interruptions, just really the ideal things of that nature and who they can contact if anything goes wrong.

So having not only strong communication but also really strong, solid pathway for reporting any sort of bugs, because not only does that help make sure that we're mitigating any problems so that our employees can continue to do the work that they need to.

So also the most valuable data and feedback for our product teams about what is going wrong.

So setting up a dedicated JIRA space and things like that.

And then I think the crux of this is really the timeline of the rollout.

So who do we think about as our dogfooding groups as in like who are the trusted testers that we're going to start out with who kind of opted in to testing this technology?

All the way up to thinking about deploying it to the entire company. Right.

And every layer of people that we increase the dogfooding size to, maybe their work is like a little bit more production focused where actually their stakes are even higher.

Right, if we were to interrupt their flow. So as we gain more confidence in the thing that we're dogfooding, we can expand access to people.

But I also think on that note, too, when you're doing pure dogfooding, having that off switch available is also very important just in case something goes wrong.

And I also think that also breeds a lot more ability to kind of take risks on the engineering side, see if something works when you have 85% confidence as opposed to 99% that it's not going to interrupt people's workflow.

So again, I think it all goes back to communication and that was pretty impressively orchestrated, especially given that we were all remote during this Gateway and Warp rollout.

And again, I think strong communication and feeding back to the product teams really were the key lessons learned there?

Yeah, I can I confess to being one of the people that sometimes is called and just been like, just take me out of the group but actually learn more about it.

And I think I should say a Cloudflare and realize that's just part of working here.

It's really important to be flexible and important to see that end user benefit to being part of the group.

I got better at it, but you're totally right. Sometimes I wouldn't read the email.

Sometimes I didn't know what to expect and when the bad thing happened to me.

Of course, you know, it just teaches you a new way of working and a new set of customer empathy, right?

Because if it happens to them when they're using the product and it's deployed, like they don't get to just DM somebody and say, you know, take this out for me right now, they have to go through a whole thing.

How often in the beginning?

Oh, go ahead, Evan.

I just want to add that one of the most important things that I found with dogfooding and that I learned through the access dogfooding process, was that having buy-in from everybody that the pain was that they were going to feel pain and that was OC and, and like tell them that their whatever workflows might change or that the security model might change and that like that's what we're doing anyways.

Embrace the change.

Embrace the pain is super important for dogfooding because it's just so easy to like bite off a small piece and say, Oh, we're dogfooding.

But until you really have skin in the game and tell you can't get on a Zoom call because something is enabled or something, then things don't really get fixed.

And at Cloudflare we did this by the very first true success story with dogfooding was the dog colo and our and it wasn't our office, but all of our office traffic went to one colo that had the newest Cloudflare software running on it.

And so like we were bought in that sometimes the whole Internet would just go down at the office and it was because somebody released a bad change and sales people are like, I have a customer call a demo in 10 minutes and we're like, Well, might not work for that.

Yeah, we're super committed to it from that front right now.

That's a really good point and sort of reminded me of something as well, which is I think articulating why is so important to I kind of went through some of the how to do a successful dogfooding run through, but especially when those pain points are encountered.

Yes, invariably there's kind of the frustration that gets that you get bought into when you're participating in a dogfooding group.

But hopefully those are those controls are in place to mitigate that.

But going back to the why, why are we doing this is so important and like yes to to better our products is certainly one big reason.

But I think having a strong articulation of here's why ultimately deploying this thing is going to be better for our security posture is sometimes a more effective right like understanding kind of going to that higher level picture where everyone on the company can get on board about why this really is the best option for us moving forward.

And we're not just eating pain to eat pain, but really this is something that is going to make our company stronger, more secure and then also better for our customers as well.

So super important to have that Why documented, I've found to make sure that everyone is on that same page.

Yeah, I kind of have this philosophy about being a security professional and it's that you've got to have a thick skin and a soft heart because a lot of it is doing things for the greater good that do not feel good initially.

So we are we are currently living that.

So I was going to ask about what it means for customers that we dogfood and what we found through Access.

But it does kind of seem like we've covered a little bit about that.

Is there anything that's hugely beneficial that we haven't covered with our Access dogfooding that we should let customers know about?

If there's not, we'll move on.

But just in case, we missed a huge opportunity to plug the security of Access.

Nothing. All right.

All good. All good.

So this actually leads us into another really exciting development that we have on the Cloudflare Security team.

So, Molly, seems like a good time to talk a little bit about the concept of our security innovation team.

Would you like to tell us more about it?

Yeah.

So we are building out a new function called security innovation on the security team.

And the idea is to really formalize a lot of the things that we just described through dogfood.

So really that closing the gap between security product and go to market.

And I think it goes back to everything that Jack and Evan said, which is really we're in this very unique position, being a security team at a company that builds security products.

So how do we go ahead and bubble up the kind of feedback and knowledge that we have as a security team to better our products and work with the product teams directly to do that?

And then ultimately, how do we sort of speak as that number one customer in order to help make sure that we are really fulfilling the needs of the customers that are coming looking to purchase Cloudflare software and kind of being leaders in the field in terms of both that technical and that process oriented standpoint.

And so I think dogfood creates a rich set of opportunities in this space because it really allows us to bubble up some of those gaps and those feature requests that we that we might have.

So definitely, I'll give an example and I'm sure Evan has a bunch as well.

So a perfect example of this and working with the Access team that we've written about on the blog is that on our security side, of course, internally we have all of our access controls in place, but as anyone who is in charge of setting access controls is well aware of.

They don't often tell the whole story, right?

Often those binary access controls can not really help capture how data that's sensitive or domain sensitive should be treated.

And so purely an internal conversation.

Joe Ah, so we were discussing sort of how do we contextualize access controls a little bit more, how do we make sure that people are really only accessing the things that they need?

And so out of that discussion, I had spoken with the Access Team about the creation of a purpose justification, sort of pop up box for people to justify why they were accessing certain domains, which is actually one of the kind of qualities that the GDPR looks for in its article about purpose, limitation and being able to justify use.

And then from that, the product, this feature on top of access kind of expanded it expanded out to time based access controls and sort of more granularity on top of what was just binary access controls.

And all of this came out from a need from the security team where we said, hey, we, we like this product a lot, but it would be even better if we had X, Y and Z feature.

That was something that aligned really nicely with where the product team was heading.

So we kind of provided that rationale and that voice in the room to help them build this out.

I think that's a perfect example of what the security innovation function aims to do, which is really to close that gap.

We have such brilliant people on the security team, so it's like, how do we really just amplify those voices to get them back to product?

Because Cloudflare moves really, really quickly.

And so it's a challenge within itself to make sure that the left hand kind of knows what the right hand is doing.

And we're really capitalizing on all of those brilliant sort of instincts that we have in place.

Then Evan, if you have any other kind of thoughts around sort of features that that we've helped influence on any products that we've dogfooded, I would be really curious to hear as well.

Oh, just so many.

The purpose justification is a good one.

The enforcement of security keys is a good one.

I'm regularly in conversations about it, it comes up a lot when products are getting built.

And the question is, well, where what part of this security product is doing the security?

And it comes in it comes up a lot in that where I say I want it to work this way and.

And that's kind of an interesting like low level one where it's it's like, how do you know that such and such a user has access to this?

How does that flow work in business logic behind the scenes actually work?

But in terms of features, that's a that's a good example.

Yeah, it's I think over the probably the past decade, there's been this concept of shifting left, right?

And that means you're getting in when the product is being thought of as a security professional, we're telling people, you know, it's a lot more expensive to go back and fix bugs.

It's a lot more expensive to patch. We should be getting in really, really early.

So this is a really interesting way to think about moving left earlier in the product development lifecycle that I don't think has been addressed by a lot of companies.

Right. We have sometimes places might have somebody who kind of has some ideas, sometimes they might go through some security review.

But this is a way more humane way to like add a human layer onto this concept of shifting left.

So will there be openings on the team or is a roadmap built out or what?

Thank you for that and thank you for that perfect plug.

So currently we are hiring for a security engineer on the Security Strategic Projects team with sort of a special eye to the security innovation function and thinking about using our own products internally and the coverage that we have there opportunities to feed back to products, gaps that are currently in existence for where we kind of want our products that are already in a really strong place to even be stronger from that security perspective.

So please take a look at the Cloudflare jobs posting if anyone has any interest.

There's a lot of work to be done here.

So very excited to get someone on board sort of as soon as we can to work through this.

That sounds awesome. And it seems like we'll have a big opportunity to work together on some security engagemenT.

to kind of talk to different teams about making better security decisions, and you're going to be developing some of those security decisions.

So I'm really looking forward to partnering on that. So that should be a good time.

We still have a couple of minutes left.

I thought, do any of you have any maybe interesting bloopers or something you would have changed about the dogfooding process?

Is there something that was just like maybe a funny story that happened?

Don't talk about me asking and crying for mercy to get out of the dogfooding group.

I've already exposed that one.

Anything else that you're thinking?

Or maybe something that you would have done better?

Like, is there some documentation that you wish you would have had or would someone be able to come with like, come fill this your shoes and be able to replicate the process?

I would add.

Go back to the embrace the pain thing.

I think that there's always a it's always easy to say, Well, we don't want to interrupt anybody's workflow, but it kind of defeats the purpose because if you're not interrupting, then you're not doing it in a meaningful way.

And so really getting everybody on and the communication piece that Molly brought up, I think those two go hand in hand, communicate that you're going to be interrupted and to expect it.

And that's, I think that we can always do a better job with that because yeah, that was a big learning and something I saw regularly with Cloudflare access.

As we were rolling it out, we went from three sites behind Cloudflare access to something like 500, 600 internal sites behind access and just an explosion of usage.

And that made a lot of people wary because it's a big change.

It's a massive architecture change for us.

And so we could have probably done a better job being really loud and proud that this is the way that we want it done because it's best in the long term and there's going to be some pain points along the way.

Yeah, I agree.

I think it's like a top down kind of culture shift and I think we do have a culture of it at Cloudflare, but I think it's like we can always do a better job of kind of embracing what that means in reality, because I think, well, everyone can be on board with the idea of improving our own products by testing them per evidence point when the pain hits.

Right.

It's making sure that everyone at every kind of level is, is aware that this is part of the process and sort of like the necessary pain in order to get to the best end product.

But with that being said, I wouldn't roll anything new out over Christmas, I think is, you know, one of the key learnings, although I feel like that's like a foundational learning for anyone in any sort of change management engineering positions.

So it still applies to dogfooding. Yeah, maybe it might make an interesting section in our interview process.

Right.

Like, what's your threshold for pain? Yeah, exactly.

Can you work over Christmas? There we go.

Wait until January 3rd when all the other changes that went over the that got built over the change freeze get pushed.

- Oh, yeah. January 3rd is probably the number one day for outages.

Oh, man.

Well, Molly and Evan, thank you for lending your time and expertise. So many of the philosophies that we embody at Cloudflare that keep me interested in my work and engaged.

And for anyone else that wants to learn more about our dogfooding, you can check out our blog post titled Securing Cloudflare Using Cloudflare.

Molly and Evan are the authors of that post.

It was just released, I believe, today.

We'll also have more Cloudflare TV segments coming soon and we'll deep dive further into other security focused products, other dogfooding adventures, maybe a few misadventures.

We'll see.

So thanks again and stay secure.