Infrastructure and Product Security at Cloudflare

All right. Welcome, everybody, to Cloudflare TV. I'm Evan Johnson. I lead the product security team here at Cloudflare. And with me is Blake Atkinson, who leads our infrastructure security team at Cloudflare. We're here walking on treadmills, walk and talk with some security time. It used to be the case that in the offices, when we all worked in offices, we would go on these long walking one-on-ones. Well, maybe not too long, an acceptable length for a one-on-one, but a half hour to do your one-on-one. And I would get home, check my watch and see that I had like 30,000 steps, something absurd, which says I was probably doing too many one-on-ones. But also, it's great exercise. And so this is the fad that is sweeping our Cloudflare security team walking and working. So we're here doing that with you today. Blake, how are you doing? Can you tell everybody who you are and give you an intro about yourself? Yeah, 100%. That's a great intro there, Evan. Yeah, we apologize for any seasickness that may be incurred by this, but we are getting our steps in and it's better for the health of the team in the long term, but we're also healthy. That's our opinions, I think. So my name is Blake Atkinson. I lead the infrastructure security team here at Cloudflare. I've personally been with the company for, I think, coming up on three years here now. My career is kind of recently, I guess, in the last five years, transitioned more focused on security. Before that, I've been a member on a few different engineering teams. I've worked in risk and fraud analytics detections in the financial sector, PCI things all the way back there. So kind of a weird career that's kind of revolved around security and engineering. And so we find ourselves now on a security engineering focused team here. Infrastructure security at Cloudflare is an interesting one. I like to think it's split between three core pillars here, which is the operating system, the firmware and hardware itself that it runs on, and also the network that connects it all. So kind of a large, kind of general pillars there. But within that, I think there's a lot of, you know, great meat to sink our teeth into. Yeah, we're trying to grow as much as we can into those unique pillars and mandates. Nice. And so you are, within the last five years, kind of focusing on security. I've always really felt that the people who don't start their careers in security are the best people in the industry, because they have experience doing other things and... Yeah, diversity of thought, right? It's so key to having a rich team, I think. I know it's easy for us to maybe come from those backgrounds and be like, yeah, ours was, you know, this is the best way to do it. But, you know, it does make a lot of sense. I mean, I know so many people in this industry that came from a myriad of different ways, no two stories are the same. But it is great, on the other hand, that there is so much, let's say, you know, maturity and, you know, education and, you know, rigor now around this field where there's... It just wasn't when I was in the school, you know. Totally. Yeah, being able to empathize with your stakeholders and security, having been on the other side of the table working with security people is, I think, one of the areas that I've noticed a lot of value. Cool. Thanks for the introduction. So I'm Evan Johnson. I manage our product security team at Cloudflare. The pillars of our product security team are three teams, application security, so the team that helps all of our engineers at the company build things securely and fix things when they're not as secure and have flaws. We have a team called product security tooling or product security engineering, and that builds a lot of tooling to automate and make all of our lives easier. So things for automatic ticket filing, things for automatic patching, all sorts of tools, identity and access management-related tools, kind of a litany of things, of those kinds of tools. And then last is enterprise security. And enterprise security is our key partners with, and security's key partnership with IT. And so IT runs all of the applications that make our business tick, and they work on keeping all of those things secure with our IT team. So pretty broad mandate, but suffice to say, if it's software, we're probably helping out on it and keeping it secure and having a lot of just conversations with people, understanding what's going on, how we can help, and trying more to understand what they're doing than having our own mission. Okay, so that's an intro to the two of us. We got into it a little bit, but I'm curious to hear, Blake, what is, we only have a half hour here, and we got to get to the meat of it on Security Awareness Month. And so what are your real core key security philosophies? So if you had to think about it, like if you had to build a house, what are the buzzwords that you're putting in the of the house to build the rest of a security team on? I love that. That is such a great lead-in. We've talked a lot about security foundations within and without our team, right? Like be a relatively young team or not, it's important to have a strong foundation to build on. And however you define that as a strong identity access management, strong detections programs, having all the tools in place. And that kind of leads me to one of the things I think maybe we don't mention explicitly is that it's a people first, relationship first role when it comes down to it, right? With you can't do that, you can't really build the foundation or the rest of the structural integrity of the building, because without that, you can't really drive the process and the technical decisions that kind of follow there. I think that's something that not to toot our own horn, but we do very well here at Todd Flair, very deliberately, I should say. And kind of taking that, let's say the construction analogy further, I think things tend to break in the middle after that. So it's more about being consistent and continuous improvement, as opposed to having kind of just like one big, hairy, audacious goal after another, and not really look back and make sure that you're closing the loop on some of these things. Yeah, that was a grab bag of all awesome topics. So just hearing, repeating back a little bit to try to organize my own thoughts about all those good ideas. One is like continuous improvement iteration. You can't just push one single boulder up a hill a year. You got to push a bunch of boulders up and continuously do that. Maybe a Sisyphean task isn't the right... Yeah, you're painting a picture. I mean, but you're not going to get it right the first time every time, you know, that's, you know, you'd be lucky if you did. And then even then, you'd have to make sure that you're still getting it right. That's one of those, you know, you very see the ops mentality kind of sneak in here, because it's a lot of what we have to do is changing all the time. And the foundation is always shifting in some ways. And if it's strong, right, you'll change with, you know, the ground and all that, you know, products change all the time, or they improve all the time, you know, the tooling will get changed, you know, all of that is super important to take into account. Yeah, I've actually always subscribed to the idea that I'm going to get it wrong the first time. Yeah, so the fastest way to get to the right answer, it's like the old comment about the Internet, like the fastest way to get an answer on the Internet is to write, like, write the wrong thing. Hey, did you know you could do this and put a bunch of misinformation about how to do something? And then someone will be like, actually, there's a much more efficient way. You can do this instead. And then, aha, why didn't I? It's a great way of saying, like, you know, you can get a lot done if you put like, some effort and to understand the problem you're trying to solve with the people you're solving it with. I think you kind of mentioned this earlier, a lot of what we do is so much, you know, like, across those different organizations and teams. And they are the ones that end up owning or rather having to run with the kind of long term security controls or the risk that is being solved by those controls, right? Oh, yeah, that's a that was another fantastic point you made is the relationship oriented kind of security. You're there to help everybody else do their job securely. And they probably know the best way to do their job. And your job is really just to latch on and say, how can we make this more secure? What are the risks involved in this and make sure that everybody's on the same page about those things? So I think that's a fantastic point. Yeah. Cool. All right. Well, if I had to give some of mine, this is actually from a few months ago. I did we did this whole startup thing on Cloudflare TV, where we interviewed a bunch of startup founders. And one of the people I interviewed was Calvin French Owen. So Calvin, if you're watching, shout out to you. One of the things that he said is founders don't spend enough time on their story. And I think that applies way more broadly than just startup founders where spending time kind of storytelling about the work that you're doing kind of helps get everybody on the same page, helps everybody understand the value of the work that you're doing, especially in something like security. That is like so binary. You're either hacked or you're not hacked. Like there's all these books on risk measurement and all of that stuff. But applying numerics to this kind of thing just doesn't really work. It's the dream, right? The dream is to have a quantitative assessment of everything you could possibly have. And there's certainly no shortage of literature on that. Oh, yeah. But yeah, it's certainly how you get there. I mean, I guess what you're saying, the qualitative approach first, right? How does that story really relate to the mission you're trying to achieve or the risks you are trying to mitigate, right? Totally. And then the quantitative stuff, you can extract down the line as you figure it out. But I've always felt like nailing the qualitative stuff and having a cohesive story around that is probably the most important thing that you can get right as a security team because otherwise everybody's just like, I don't understand what these people do. All that kind of thing. Right. And it's related to the dreaded specter of checkbox security where you're chasing literal missions that may or may not be actually telling the story you need to be, especially with the people that are going to have to keep saying the metaphor, the people that are writing it. There's so many things that you can negotiate around that aren't actually going to make you more secure. Plenty of examples of that in history. Yeah. I have no problem saying, yeah, this is a checkbox. When I'm asking for something that's totally, hey, we need this for compliance purpose, it's a checkbox. And sometimes it's okay. It's something that auditors look for, whether right or wrong. But what needs to go along with that is the real security that everybody can see and understand and you can demonstrate. But yeah, there's so many stories of security going wrong, not enough of it going right. Right. I love to quote this all the time. Have you heard of our blue team all the time? Because it is not to paint the negative picture of a Sisyphusian test, but are you building the right strategies to address the unknown or to at least react when it does happen? We use the term defense in depth all the time or the Swiss cheese model, what have you. But it is about are we building layers or are we just generating noise or are we increasing the signal to the noise maybe is the better way to phrase that. Totally. Okay. I have one other kind of personal security philosophy that I try to encompass. And that is another super kind of maybe a cliche in the startup world, but action. Being action oriented. I think it's really easy, especially on a security team to expect other teams to do all of the hard work. Security has got to be willing to get their hands dirty, not be, I sometimes use this somewhat negative sounding phrase, but like clipboard holders, like a Bill Belichick or like an NFL coach sitting on the sideline, which I mean, fantastic coach, Bill Belichick. I don't think he's, I don't know his personal history, but I feel like he's never. There's not check boxes on there and that's a different kind of thing that's on there. Yeah. He's not asking, did the defensive line drink their Gatorade this morning? He's got other things on that clipboard, but yeah, I feel like people need to be willing to get their hands dirty on a security dig in, set aggressive targets. Oh, this is another kind of diatribe that I have about there's not, I think you look at a sales team and every quarter there's this pressure to hit your number. You look at a product team and every quarter you need to ship something for the sales team to sell. So the sales team can hit their number. There are these pressures on other parts of the business that aren't always on a security team. It's like, don't get hacked, but it's so ambiguous that it can be easy to, I think, lose focus on that goal. And you have to really push at it and stay focused on that mission and give yourself a sense of urgency because there's no quarterly number for getting hacked. There's no quota or anything. Yeah. And it's funny, you're almost making the point for a quantitative assessment model, or rather highlighting the challenges in lack thereof of a true binary status of a company right when it comes to security. I was going to say something else here, but it is an interesting overlap, I think, between the approaches that all teams have to take here. Because it is when it comes down to everyone's job here. And it's really more about how are we moving everybody along together and feeling that responsibility. Sorry, moving security up into the right, let's call it. Shifting left, I don't know what you want to use here. But yeah, I'm using too many terms, I think. I'm picking up what you're putting down. Yeah. I like the call out that I'm trying to ascribe some quantitative stuff to this. And it would be great if it was like, we blocked x many things in the WAF. And like, we had a great quarter. Good job, everybody. Yeah, it would be great to ascribe numbers to it. But the reality is, you can ascribe numbers to like migrations to make you more secure. Like, we had x many websites that we put behind a security key this quarter. That's great. But in reality, it's that urgency isn't natural all the time. Yeah, I think you're touching on one of our next topics, important things that we like to talk about. And you talked about getting your hands dirty, having a hands on approach, because how do you really solve something without having that intimate knowledge? I think the same thing is true from an organizational standpoint, that security is one of the few organizations that sees how the glue comes together, or really, the things that it's gluing together. And that's just as important as like, you know, what's the next cool thing that you're going to actually ship or deploy to production, right? Like, it's there's the ability to connect that together. And, you know, move with urgency is extremely important, I think, for security engineering teams. So yeah, absolutely agree. Actually, I say that to candidates a lot when we're hiring somebody, I say something similar, I say we get to see it all because we're a security company, we kind of sit in the middle of all these different companies, really security oriented teams. So, you know, yeah. Great. So that was kind of our personal security philosophies and pills to die on anything else that you want to add to it? We can move a little bit further into one thing, I guess. And it was kind of like it touched on what you're saying about hands on this, but also that, like the ownership and accountability story needs to be, I guess, related there, right? Are the risks assigned to the right people are the engineering projects, you know, we have this conversation all the time, like, at what point is it the right time for this team or this team to be, you know, working on this project, or working on this particular risk. And then, you know, if you're not on the ownership, right, you're not going to get the right result in the end, I think. But yeah, cool. All right, so we have about 10 minutes left. I'm curious, what's new in the world of infrastructure security? What's going on on your team? It's great that we haven't talked about this all the time. But it is great to have this opportunity. I think one of the things, you know, like we said, those three big pillars, there's so many things to be excited about. You know, you can find information about our story on kind of moving our root of trust, you know, from software further into hardware. And that story is kind of, you know, transpiring, like all across, you know, what we call the platform. You know, remote attestation is something we're working on right now with a few different teams. It's really exciting stuff that, you know, is really tough to get right, if I'm being honest. As we may or may not have covered in Cloudflare, we work across, you know, 100% bare metal, you know, everywhere in the world, with, you know, just amazing engineering solutions that have been deployed that I've never seen anywhere else in my career. But the challenge of having, you know, that hardware, there's so much that is proprietary about it, so much that you have just no, you know, it's very opaque. There's just no way to know without very, you know, detailed or, you know, even reverse engineering, would be really exciting. But what if we could move towards a more transparent model? And that's something we've been working on with our hardware team a lot, specifically, excuse me, with face for management controller technology, we see in every one of our servers. So a huge shout out to those teams who've been helping us kind of move in that direction that makes it easier to address vulnerabilities at that layer. And that layer is like what's responsible for, you know, ultimately managing the system, you know, what's connected to the public Internet. So it's a, it's really important. It's really exciting to see and think about the opportunities there. I mean, I could take that a step further, and say that open technology is just something that this company takes very seriously and something that we're hoping to kind of keep moving in that direction. We're talking about, you know, how do we do, you know, production endpoint detection response technologies, what's out there in the world, we think we have some really exciting opportunities there. You know, more about that ownership model, that control and that ability to have more trust and say operational capacity. Oh, yeah. Yeah. Sorry, if I'm walking you out of breath here, you're just in much better shape. That's all I'm doing this for good reason. That's this is all a ploy for me to up the company's health premiums. Not really, of course, but the a lot of really good points there. One that stuck out to me that you just made is we're doing everything on bare metal. A lot of what we're doing is so unique. I think that's what kind of makes it fun and also challenging. Absolutely. Cloudflare is there's no RFC to write or to read that will tell you all the things that could that we have to worry about as a security team. There's no security consideration section that tells you but there's also like no drag and drop security model that you could take from a past job and just fit it in here. So like, if you let's say you work at a company that runs everything on AWS, you could probably just like take your your past cloud security kind of stuff, drag and drop. And your program will look pretty similar from one company to the next based on just the infrastructure that's running your product. But like with Cloudflare, we're running it all ourselves. We've got points of presence all over the globe. We've got to secure that hardware and makes it really fun but also really challenging. There's no API to call to just like set the security bit or fix the firewall rule or whatever. And it's awesome to see some of these amazing kind of security oriented internal projects that are ultimately powering Cloudflare's infrastructure. Like things that maybe you only see a handful of companies that we can't name right now that are able to do that. It's truly impressive to see how the punching above the weight class so to speak or rather trying and succeeding with what would be pretty aggressive security first technologies I think is always exciting to see. Absolutely. Cool. Well in my world, in the product security world, we've had some pretty interesting stuff going on. We've blogged a little bit about it over the last few months but we blogged about an attack that we mitigated because Cloudflare mitigated because we'd rolled out security keys. And since then we've kind of written about how we made the transition from no security keys just a few years ago to having security keys for all employees and that's how they authenticate whenever they're getting into our stuff. And so we've been a part of that storytelling going back to the storytelling. We've been a part of like a little part of that narrative. A lot of people helping on the partnership that we announced with Yubico. We weren't super involved in that but we definitely wanted to share with everybody, hey we used to have this archaic kind of moat-based security model and this is how we made things better. One service at a time. One small change at a time. And that blog came out and was super exciting. What else do we have going on? Over the past couple months we've been working with all of the engineering teams on all the exciting stuff in birthday and GA week. To different degrees sometimes we'll get really in the nitty-gritty details of how a product works. Sometimes we'll say, oh that sounds good. You all have a that's pretty low risk adding that feature. We'll spend our cycles elsewhere but we just finished that up. And then what's exciting and upcoming? There's a lot of exciting stuff. I mean we're always looking for ways to use our own product as part of our security model so that we have skin in the game. I continue with some proven story that you were talking about earlier. That's absolutely part of it. I don't know if you can't share specifics but I can tell from as another team working with you that I see the kind of steady improvements that there is to make towards a lot of this. I mean what we would call this plug the zero trust products or whatever we want to do here. But you can see that kind of steady continuous improvement within employment, dog fooding, whatever you want to call that. That's really the mantra that we have is for any of our problems but especially in implementing our own products we want to say what really matters in this problem space. And we try to solve for the 80 first and then the 20. I don't even know which line of work you're talking about because we've got so many projects going on at any given time. It could be any number of things that we're doing continuous improvement on. But yeah that's the mantra whenever we've got stuff going on. Cool. All right we've got about two and a half minutes left. So we were talking about running a security team. What's really important for teams? What do you think is one of the biggest traps that you can fall into? That's a good one. We talked about ownership, accountability, what have you. It's really easy when you go into some of these kind of larger meetings to shy away from the actual hard decisions or the big scary kind of looming elephant in the room. And it leads to that colloquial term of bike shedding about things that are more tractable or easy for people to have an opinion about. But if you've got to make hard choices about funding something or moving forward with perhaps an unpopular policy or something, it's just going to take time. It's definitely a trap to fall into I think on any engineering team to talk about the color of the bike shed. But when you're trying to design the nuclear reactor I think is the metaphor. It's important to deal with those hard decisions head-on or at least acknowledge that they're going to take time. Yeah I definitely like that. I think the traps are almost the same that we're talking about or that I would bring up. The one is kind of the opposite of continuous improvement. It's kind of like you plan, measure, measure, measure once, cut once. And that just never happens in the real world. So you're trying to build a nuclear reactor like... No I like that. I used to call this chasing the shiny things. You know like there's so much cool stuff you could do. But you know if you're going to build solid security foundations, it's not always going to be the shiniest, most fun project in the world all the time. But it shall hopefully be something to build upon and that will lead you towards those kind of innovative and novel ideas I think. Totally. I've always thought of shiny things chasing as kind of like short-termism. But I like that it's long-term and you can chase this almost utopia of a future of that'll solve all of our problems. All right well Blake, this has been fantastic. Thanks for having me. This is the Security Awareness Month. It was great having you on and really appreciative of hearing all the cool things going on in infrastructure security. And with that, we will see you later. Thanks for walking and talking with us. Thanks again.