A Year in Review: Security Operations Engineering Team

Welcome everyone. Welcome to this exciting episode of our year-end review of the Security Operations Engineering team. This is Aishwarya Ramesh Nagarajan and I have been with Cloudflare for the past two years now and it's been an exciting time so far. I'm a part of the Security Operations Engineering team and I'm based out of Austin, Texas and with me I have the captain of the ship, my boss Demetrius Green. Hi Demetrius, how are you doing? I'm doing great. I'm doing great. How are you? I'm doing very well. Demetrius, so I know like you don't need any introduction you know like within Cloudflare but for the context of this meeting and for wider audience I'm going to ask you to please do the honor. Absolutely. So hey everyone, I'm Demetrius Green. I am the director of this great team, Security Operations Engineering. I've been with Cloudflare almost a year. It'll be a year in two weeks. It's been a great journey so far looking for more. I'm based out of Los Angeles, California where right now it's about 85 degrees and sunny outside so I don't know where everyone else is but hopefully you have nice weather like we do here. Yeah, I'm excited to be here, excited to talk about the team and you know and where we're going from here. That's amazing. That's not honestly a right question for somebody from Austin. Demetrius, I mean I don't want to say the temperature out but I'm enjoying it. That's all I want to say. I was talking to Austin. Absolutely. Awesome. So that was amazing. So for just to set the expectations right, Demetrius and I are going to take the next 25 minutes, 22, I mean 25 to 30 minutes to walk you all through on all things security operations engineering. We're going to cover some of the major accomplishments, key milestones we have achieved in the last one year and we're also going to share some exciting updates and things we are really looking forward to in the years to come. So just tighten your seatbelts everyone. And before us diving in, I also want to give a quick introduction about our team, security ops engineering team and kind of like what we do at Cloudflare. So security operations engineering team, we also are called SecOps in short. We are basically one of the core security engineering teams at Cloudflare and we own a lot of security programs and we also help other teams, other security programs perform and achieve their goal. So some of the teams that we own are vulnerability management program, say for example. We own the vendor security program at Cloudflare and most of you might know that Cloudflare recently went public on its bug bounty. So it's us, we who own the bug bounty and we are sort of like the first level of respondents for the different reports that we get via our bug bounty platform. And we also do support other teams like the detection and response engineering team, say for example, for helping them respond to incidents. We help them write detections, fine-tune the existing detections that are based on SIM. And yeah, so we help automate, you know, improve our business processes inside and outside our, you know, the security ops team. So these are some of the exciting things that we do and we both are very excited to share with you all on the different milestones we have accomplished. So Demetrius, so let's tighten our seat belt and we are going to basically get into this time mission and I think time travel to October 2021, I guess. I think October 2021 is when you started, correct? Absolutely, absolutely. That's one of the days to mark in the calendars, I guess, because I should say it was like after Demetrius and before Demetrius, sort of a thing for a team. So I want to hear it from you, Demetrius. So when you started, sort of like what are some of the first impressions, you know, like you had when you joined the team with respect to people, tooling processes, we were getting matured at that stage. So what are your thoughts and how was the team before one year and how have you progressed? So what are your thoughts on that? Yeah, absolutely. And thank you for the introduction to the team. So prior to, I'll take a small step back and talk a little bit about the beginnings of the team because many people might not know that this is a fairly new team within security. So around March or April of last year, before I joined, there was a few members, senior members of our team that made a decision together to establish this team as a necessary and important part of the overall security program. Those members, for example, Blake, who's now the head of infrastructure engineering, led the charge to go and interview and establish this team. I know Susan also had a hand in that and perhaps many others in the interview process. So the team grew from basically no one supporting to about eight engineers within, you know, four to five months starting around March of 2021. That is rapid growth. And, you know, although there's a lot of risk, if not, there was also a lot of reward and success that came out of that. And some of that we'll talk about today. But that's kind of the landscape of where the team started before I joined. Prior, shortly before I joined, having conversations with some of those leaders who established that team, it was well, it was, you know, well communicated that I would be coming in and taking over a brand new team. And not just a brand new team, but some of the members being brand new to their first job straight out of college. Others, maybe this is their second job. So brand new team with mostly brand new, you know, people and engineers supporting it. But, you know, that didn't scare me off, you know, obviously, because I'm here. I saw that as a great opportunity to, you know, grow a team, mentor the individuals within that team, and just overall help Cloudflare and the Cloudflare security team achieve their goals. So from that day, from the day I started in October, and started, you know, getting more and more involved within the culture of Cloudflare, the culture of security, and the culture of the operations team, there was a lot of positive, a lot of positive, you know, work and achievements coming out of the team, even at that point in time. I mean, at that point in time, you know, the most senior person in the team was four months in the Cloudflare, and I came into a team that was already achieving great things. At that time, we were supporting mostly the security incident response process. However, there were plans that were already moving forward for that, for our team to take over the vulnerability management program, which was just established and led by Jenna. There are also plans for us to take our private bug bounty program to a public bug bounty program, and have the security operations team support the initial triage of those reports. So with those plans in place, you know, my job was to come in and make sure that they became reality. And it was such a great experience, you know, starting out and getting to know everyone, and rolling up my sleeves with the team, and working on establishing those programs and bringing those programs in into SOE, because ultimately, the team that we have is the best team suited to handle those day-to-day activities. And so far, they have been very, very successful with insecurity. That's amazing. I agree 100%. As I already mentioned, it was almost like before Demetrius versus after Demetrius, you know, when seeing how much our team has grown. And when I consider how mature our team has gone, you know, like, it's amazing. So thank you very much for that. So it should be, you know, like this whole timeline that you mentioned, Demetrius, like when you joined, and we, as you mentioned, we basically got a lot of programs, like vulnerability program, we got like HackerOne, Bensec, all these things merging into SOE. But we still did really well, you know, we still coped up and we adapted really quickly to all these different changes that we were getting, and we were awesome about it. So this is going to be a really hard question for you. But I still want to ask you, because I'm going to ask you, what are the two to three top accomplishments you think, you know, we have grown so far? It's a very hard question, because we have had a lot of accomplishments. But yeah, if you want to name some major key accomplishments, what would it be? Absolutely. I think, foundationally, I think one of our greatest accomplishments for this team is our ability to communicate and work together. I think when I first came, when I had my first meeting with the team, I remember this vividly, you know, I repeated to the team that what's going to make us successful is our ability to not only communicate within our team, but communicate outside of our team. Because of where we sit in a lot of our security processes, a lot of what we do, we have to interact with many different teams within ThoughtFlare. And those teams have to interact with us. And so our ability to communicate timely and communicate effectively to those teams is key. And, you know, I'm happy, you know, to share that, you know, that's something that I truly believe is one of our greatest accomplishments, coming from a team that didn't exist, you know, four months prior to existing and taking on, you know, many different programs, to moving those programs forward is, I think, one of our greatest accomplishments. And I think that's for, that comes down to our ability to communicate and work together. Secondarily, though, there are a lot of major projects and initiatives that we took on to help us become a more efficient and help mature our operations. One thing we focus on heavily is automation. You talked about that earlier, when you were giving the overview of the team, how important that was. And I'm here to say that it's one of the main reasons why we're able to grow organically with Cloudflare as Cloudflare grows, is our ability to establish automation initiatives that help our team and other teams improve their efficiency. A lot of the automation we focused on was around visibility. So, the ability for our team or other teams to be alerted when certain statuses or states are established within JIRA. And that really, given the workflows that we have, it's easy for us as a team to get distracted by the many different things going on. And having automations in place to help us focus and visualize what's happening at that time, without actually having to go into a JIRA dashboard or go into a JIRA Kanban board, is really key for us to be able to focus and juggle the many aspects of what we work on. We also did automation projects for different forms and also different ways to link and consolidate different tasks within JIRA. So again, there's so many small projects that really developed into something major for us in that arena. And there's still more to come in that regard. Another one, which is some people may see as a little mundane, but it's super important to what we do, is documentation. We, a lot of what we do, we have a process for, and that process needs to be documented. And one thing that this team did is we not only created new documentation for new tasks, for new initiatives, for new programs, but we also modified documentation that already existed and needed to change due to our maturity and the changes within the company or around us that were happening. Very, very important, even if it's not the most exciting thing to do on the team, it is very important to an operations team to have that sort of documentation available. Also tooling. So we've established quite a few tools over the last year plus that help us, again, either gain more efficiency in the processes that we already have or gain more visibility into areas that we just didn't have visibility into before. So for example, GitGuardian, there was a project that was led by one of our SOE engineers, Ari, out in Lisbon. She led a project to establish a program that brought visibility to our team around our public Git repos and being able to scan those repos and provide us with alerts where we could go and investigate whether or not there are, you know, keys and secrets that are publicly exposed and things like that. Again, that was something we didn't have before, at least not easily. And then after implementing that now, we could confidently say that we have at least some monitoring on our public GitHub. Also building forensic tools that help us in investigating different incidents. We had Miles and Calvin working together to develop some forensic tools to help our DNR partners in a lot of their investigations that require collection of evidence off of different areas of the business, whether it be on metals or on different endpoints. And also, HackerOne, you know, we developed some processes around that. But it was a big move for us to go from a private program to a public program. We had a drastic increase in volume. And we also took over the triage from HackerOne, which was a managed service, making it an internal process. So that, again, took a lot of effort to stand up. But now in that program, where we sit today, is now we are triaging and closing out reports faster than HackerOne did for us. That's always a good success story when you want to determine whether you should break something in-house or have it as a third -party managed service. And lastly, quickly on this, we also, you know, due to a lot of our different workflows that we inherited from the team, we looked at that and tried to figure out the best way to structure our organization in order to better handle the very distinct workflows that we have and also be able to handle the growth that came out of those. So we also went through recently and restructured our team, both the U.S. team in Austin and the team out in Lisbon, Portugal, in order to meet the increasing demand and allow us the structure to scale for the future, which is also very important. So that doesn't cover everything, but I think those are some of the core initiatives and projects and accomplishments that our team has. And I think it's paid off significantly in our ability to help Cloudflare. That's amazing. I think we need a separate 30 -minute discussion or a 30-minute Cloudflare segment just to discuss the accomplishments. So that's so amazing to hear all the accomplishments and also happy to be absolutely, you know, happy and proud to be a part of the security ops team. So, okay, so now we have discussed on how the team was before one year and then the major accomplishments. And the next key question for us to look at is what are some of the major opportunities for improvements, Demetrius, that we are going to take it to the next year, you know, like going forward. So what are your thoughts on that? Yes, and this is where my role gets really exciting because, you know, the large majority of my role is developing and creating a strategy for how to mature, not just the security operations engineering team, but also help security overall mature as Cloudflare is maturing. So a lot of what we're looking to do in the future to mature is really centered around some of the core tenants that I talked about previously around some of our accomplishments in the past year. It's really taking some of those projects and programs and initiatives and just expanding them in different ways to help mature a lot of our workflows. So, for example, one thing where we are looking to focus on is our vulnerability management program. We established a program that really met a lot of our needs at the time that we established that program. Now that we have, you know, a year into this program, there are many different ways where we can take that to the next level. What we really want to do is take it into more of a risk-based program where we are not just using, for example, a raw CVSS score, which is very generic. It may not or it may apply to Cloudflare, but we also want to be able to bring in our own context from different risk factors that we have established within Cloudflare. For example, is something Internet-facing? Is this endpoint attached to an executive? Is this particular user, does this person have elevated privileges to the Edge or to Ninja Panel or things like that? And kind of put all of that together and develop a more custom risk score. And this will help as we are sharing these vulnerabilities, whether it comes from HackerOne or comes from our third-party bold scanners, it will allow us to give higher fidelity information to our engineering teams so that they can get better and more efficient at the patches and fixes that are necessary to remediate or mitigate the risks that are associated with those vulnerabilities. Another thing with vulnerabilities, we're also looking to change how we label vulnerabilities and have that coincide with how we develop and share out our metrics with those engineering teams. So we want to get to the point where we are identifying vulnerabilities that are attached to certain products that Cloudflare has or certain colos or certain high-risk areas, for example, and be able to label those appropriately and provide those insights to engineering so that they have a more holistic understanding of potential vulnerabilities and how they are aggregated maybe towards one product or maybe one functionality of one product. So maybe there's an authentication type concern that we want to address and our justification for it is showing in our Vuln program. It's showing in our HackerOne bug bounty reports. So that's just an example, not necessarily reality, but that's something that we want to get better at so we are providing better information and data to our engineering partners. Tooling also. Tooling is very important, but it's not tooling just for the sake of having tools, right? We want to make sure that we have the right tools that meet the needs that we need and overall make us more efficient. And so we're looking at different tools within the Vuln management space to help aggregate those vulnerabilities a little bit better, again, trying to improve the information that we give to our engineering partners, IT and others. We're also looking at tools for managing things that happen within our incident response program. So for example, a case management tool that helps manage a lot of those incidents a little bit better and helps us get this information out to the appropriate teams as fast as possible. Also looking at different tooling within BNSEC, working with our procurement partners in that arena as well. So more to come on that, but again, these initiatives are really there to help us get to that next level of maturity. And lastly, as far as the team's concerned, one of our goals as an operation team is not only operational excellence, but also to be a 24-7, 365 global organization that helps monitor and maintain security around the clock. And in order to do that, there's team structures and we have to establish different members in different areas of the world if you want to meet that need. So that's also one thing that we're looking to do over the next few years is really establish that program where we have someone that's eyes on glass for 24 -7, 365. Again, that should improve our ability to respond, to really identify incidents and respond and remediate incidents that we see. So that's a very high level view, but I'm really excited about a lot of these initiatives. I can't talk about them all, but just know that there's a lot of exciting things that are on the horizon and I'm happy to be a part of it. Oh my God. Yeah, absolutely. I cannot actually wait to do another session same time next year to see what are the different things we have accomplished from all these checklists. I'm so excited for how our different teams are growing, especially as you mentioned the restructuring, that has really helped us produce better metrics and be more efficient and things like that. So I'm very excited for how our team is going to grow in the future. And if anybody from APAC is watching this program, this is also kind of an early news. Of course, nothing confirmed, but we will be expecting to grow our team sometime in the future. So just a very early it's up. So do you have anything for me, Demetrius? Yeah, actually, I feel like I spent a lot of the time talking. I don't know if that was by design or not, but you joined our team a few months after I joined. You came over to SOE from the GRC team. So I really want to know, I gave the insight to my experience coming over. Can you give us some insight into yours? Absolutely. And before that, going back to your question, this is actually by design because I always felt that you were the person who was asking me questions as a manager. So I was like, I'm going to take this 30 minutes to ask you the questions. That's fair. And coming back to your question, Demetrius, I think to be honest, I love everything about the team, our team, security team, and Cloudflare. The sort of culture, starting from the sort of culture we have, the sort of transparency we have, I think it's just built sort of a thing. So our culture is something that I absolutely value. And the amount of importance that you give to talented folks, I think that's also pretty important. It's not that you just do your job. Just take the Security Awareness Month as an example. So participating in CFTV sessions, getting opportunities to do these kind of things, getting opportunities to present in conference. Jackie from our team, she's always like, she wants people to present. She's kind of the person who's sort of like a gatekeeper and always somebody who encourages people to present yourself in conferences and improve personal land for branding. I think it's definitely important to be like, and especially when you are new in your career, like you are early in your career, as you mentioned, like our security ops team, you can do everything, almost everything in security. You can do vulnerability management, you can do incident response, you can write deductions, vendor security, hacker one. So these are basically, these used to be like different teams, like separate teams in different companies. But since we have sort of this hands-on to do all these different things, I think the kind of exposure that you get and the amount of things that you learn, I think it's humongous. And I cannot stress it enough on how much it has personally improved me to understand these different processes. And if I look back to see like how much I have grown, I think I'm pretty sure you will agree with this too. Absolutely. And I think for me two years ago, I think I figured out what I wanted to be when I grow up. However, if I would have had the opportunity that our team have here on this team to be exposed to such a wide range of security programs and initiatives, I probably would have figured that out maybe a few years earlier than two years. But I do completely understand what you're saying. And even as a leader of this team, I feel in a lot of ways the same as you do as a contributor of the team. And I think that ultimately shows that we are all sharing in the success of this team, of the security team as a whole at Cloudflare. And I think that's one of the key benefits of kind of how we're established within this organization. I think that really shows in the work that we do. Awesome. Yeah. And more importantly, if you want to be a leader in the future or something like that, you want to take up leadership, I think connecting the dots is pretty important. And I think having this view of working on each and every team and security team at Cloudflare, it's definitely going to be beneficial too, I guess. So I see the timer and we have close to one minute. So I think it's going to be a sad time. We need to end the segment soon. So before we end, I have two exciting updates. So one exciting update is obviously security ops team is hiring. We are looking for people and we have an open job role in Lisbon. So in case if anybody watching this is interested, please check out the job portal and you can see a security ops engineering role open. And you can reach out to either of us or anyone from our team to know more about it. So that's one update. And the second exciting thing is we are all doing this as a part of the security awareness month. Cloudflare and the security team, we give so much emphasis to security and that's the reason we are doing so. Do support us. We are going to have a lot of team members presenting in CFTV in the future days. So please do support us and thank you very much for your time and thank you very much for tuning into our session. And for the last 10 seconds, shout out to our SOE team, Myles, Calvin, El, Ima, Aishwarya, Ari, Jonathan, thanks for all your help and support. Bye.