Originally aired on February 12, 2021 @ 11:30 AM - 12:00 PM EDT
Join Evan as he speaks with guest, David Brumley, CEO, ForAllSecure, about Fuzzing, CTFs, and recent software security news!
David Brumley, received his PhD in Computer Science from Carnegie Mellon University, MS in Computer Science from Stanford University, and a BA in Mathematics from the University of Northern Colorado. Brumley became a tenured Professor of Electrical & Computer Engineering at Carnegie Mellon University in 2016 and went on to become the Director of CyLab Security & Privacy Institute. With over 20 years of cybersecurity experience in academia and practice, Brumley is the author of over 50 publications in computer security and has received numerous awards, including the US PECASE award from President Obama, the highest award in the US for early-career scientists and engineers.
In 2012, Brumley, along with his graduate students Athanasios Avgerinos and Alexandre Rebert, co-founded ForAllSecure with the mission to secure the world's critical software. In 2016, ForAllSecure went on to win the DARPA Cyber Grand Challenge with Mayhem, ForAllSecure’s autonomous cyber security system.
All right. Welcome to the number one security show anywhere. The number one hacking show on Cloudflare TV. This is Hacker Time and I'm Evan Johnson from Cloudflare's product security team. With me today is none other than the most famous guest we've had on Hacker Time. Our first guest with a Wikipedia that they didn't write themselves, I'm pretty sure, David Brumley, who is, I'll let him do his own introductions, but he's the CEO of a company for all secure and a well -respected security academic researcher. Thanks for being on today, David. Oh, super excited to be here, man. Mad respect for all the things Cloudflare is doing. It's an honor. Yeah. I'm glad you were able to make it on. And we wanted to have him on for maybe, I think the second or third episode. And it it takes time to schedule things. And it's been a little delayed, but sometimes waiting for something, the outcome is better waiting for it than getting it immediately. And so I mentioned you're a CEO now, you're, I don't know, are you still a professor too? I don't know, man. So, you know, I started off as a high school dropout, and then I was a CIS admin and worked at Stanford. Then I became a professor. I'm CEO. I've been on leave for three years. So we'll see how CMU looks at that going forward. Technically, I'm still listed and, you know, I could get healthcare. So I think so. Okay. So by the letter of the law, you're still a professor, but you're also CEO. Can you tell me about what For All Secure is and does? Yeah, it really started at university research. We were doing work at how do we automatically prove off the shelf programs are vulnerable. So we do things like automated exploit generation. And then DARPA came around with this thing called the cyber ground challenge. So we spun off a company really to address that research challenge of, you know, can we devote a couple of years to that full time? And then we looked at it and people were very excited about what we're doing. So we built a product out of it and we just released that product last year. So it's been a crazy ride, but overall the arc is like how do we get cool research actually in practice? And I think the real answer is you have to start a company if you're serious about it. Yeah, that's that's really interesting. So it started as just your research as a professor. And then that was just a research area of interest for you for a long time. This type of exploitation and all of this stuff. And then from the DARPA grand challenge, I think that was what, 2012 or something. It was 2016. Oh, the DARPA grand challenge. I was thinking about the cyber fast track DARPA. You did the cyber grand challenge and you won first place. We did. I mean, it was an awesome contest, fully autonomous offense and defense against not just like other competitors, but like colleagues and people who are in the CTF community, community like shellfish and so on. But yeah, we won. It was, it was a wild ride. It was super exciting. But like every DARPA grand challenge, like they had a self-driving car contest and a robotics, right. It shows the art of the possible, but that's quite different than getting it out there so people can use it. Yeah. So you, did you expect to win? Yeah. I mean, I don't want to come across as conceited, but in the qualification round, I think if you added up everyone else's score, they still wouldn't have beat us. They became much more competitive in the final round. Gotcha. Gotcha. Yes. And now the same technology is something that companies can, can purchase and use in-house at their company to to help find issues in their software. It's the same, same thing. It's the same thing. There's, I mean, so we actually in some ways stopped doing research and advanced automated analysis after CGC, we've done a little bit more. A lot of the challenges of bringing a product or like, how do you easily ingest new apps, right? Like how do you take a team from zero to one? Not how do you take an expert and get them from 90 to a hundred? So we put a lot of work in that. Yeah, that makes sense. We we're for all secure customers and use it internally and and really like it. But the thing that you're always I don't want to say battling against, but the hard part is not the the tech. When you have a bunch of engineers, it's getting them to want to use it, making it easy, making making it a product, like you said. And that's really interesting. So you won the DARPA Cyber Grand Challenge, you started a company and and how's how's it going? What what are the goals for your company kind of right now? So we're in the growth stage, right? So we're big believers that if you're going to do application security and everyone should be doing this right, like if you look at the stats, like 90 percent of hacks are due to software flaw, that it really should be fuzzing. And things like software component analysis are great, but like fuzzing is really the the technique that's going to get us into the future. Yeah. What about fuzzing? We do we have a what do you think about how not? Do you think that all applications should be fuzzing themselves? Like is that in the hierarchy of needs? Like where do you put fuzzing for for a Cloudflare? For us, it's really, really high up. But do you think that all companies have the same kind of profile and need for fuzzing? I mean, it's true. There's different value propositions of Cloudflare. What's remarkable about you is availability is a security issue. So typically in security, someone might just say, oh, you know, that's just a DOS or that just caused an application crash. But to you, that's like real dollar figures. Yeah. A lot of people should fuzz. But for anyone that's in the high performance game, it becomes even more important. Yeah, that's kind of how I've thought about it, too, where when it comes to anything where assurance and availability is really, really important. I use the term assurance and there's like a real definition of software assurance that you I might be butchering every time I say that. But I think about like avionics, autonomous vehicles, companies like Cloudflare, where availability is really high, necessary to be really high companies, any kind of infrastructure as a service or platform as a service company. Anybody really doing anything low level, especially when they're writing C, C++, Go, Rust? And it's a necessary tool in our toolbox, for sure. Yeah, absolutely. I mean, it starts with the developer, right? You got to have well-architected software. You got to people who have people who are knowledgeable. But, you know, after you have all that, you just have to test it. And I think, you know, people like Google and Microsoft have kind of pioneered how important it is. So to your question, I do think everyone can benefit. It's kind of interesting when we look at fuzzing. I'm a security guy. I look at it as like, can I find bugs? That's the number one thing. But I think one of the cool things about it is it also helps build a test suite. So in CGC, the Cybergun challenge, one of the reasons we won is because we take the fuzzing test suite and after an app was patched, we would replay it and measure if there was any performance diff. And for example, we wouldn't play a patch. We wouldn't feel the patch if there was more than 5% performance diff unless we were being attacked. So we had this like business logic, right? Like if something's not going to be too expensive and increase security, you do it right away. Otherwise, you're going to wait until there's a reason to do it. So I think a lot of security people get hung up on the vulnerability, but fuzzing is one of the few software testing techniques that provides that real benefit to the developer, right? Beyond security. Yeah, that's really interesting. I like that. Well, great. I'm a big fan of all the fuzzing that we're doing at Cloudflare. We have more to do. But for me, the the big value is kind of when you, at Cloudflare and at a lot of other companies that could benefit from it, part of our, we get asked by customers all the time, why won't, why aren't you going to get hacked? Or like, how can we think about your availability? And we need a good answer as to like all of the things that we have in place to prevent something bad from happening, from security controls to compliance certifications to everything. And to me, one of the most important parts of that that we we added is fuzzing. And it's it's something that's really, really close to the software itself and really, really effective in finding bugs. And so to me, it's a really powerful part of the argument when of our assurance argument to our customers when we say, hey, we're doing all of these good things. And on top of that, we're testing our code with with like 10 million test cases an hour or whatever. Yeah, absolutely. And the other part of it that you didn't mention, right, is if you go look at the professional AppSec people, the people are going and finding zero days, whether it be fluorescence as part of Pwn&Own or, you know, just just those people at that level. They bring a lot of human creativity. But like fuzzing is a very popular technique here. Right. So if you're not doing that in-house, then they're going to do it for you and you're going to be surprised. Yeah, they are very creative people. But also a lot of their time is spent fuzzing, trying to find where to employ their creativity and to get a they find the crash and then figure out how to exploit it. Yeah, absolutely. Like one of the people that I admire a lot is George Hotz. He was a CMU student. And I asked him, like, how do you find zero days? And what he said is, you know, I would look for like pieces of code that likely aren't tested, but complex. So, you know, it wasn't like he was trying to understand the entire app base. It was like, what's untested? Yeah. And that's where fuzzing can help, because it's like an army of robots helping you do that. Love it. All right. Well, we mentioned I want to I want to shift gears because I'm sure we'll come back to fuzzing. But one of the hot takes that I saw you make on Twitter yesterday was about security people in general and and how the security industry, there's a cost of doing nothing. Can you tell me about what you think? Yeah, it's so this was a big change when I was a professor, when I was a professor, I'm like, you know, what is the best security? And thinking people would reasonably evaluate security as, you know, the cost of a compromise. But in reality, like if we're going to as a startup go and sell to someone, the biggest challenge we have is people who just don't want to do anything. You know, they haven't been hacked yet. Whatever they're doing is working. You know, it's kind of like there's a Simpsons episode where you're like, you know, I didn't get hacked because I have these these air pods. So these air pods are my security. Completely unrelated, but like coincidentally related to the facts. So the cost of doing nothing is just high. And it's actually even part of our sales approach. When you look at when we're when we're talking about customers, it's something we evaluate. Like, are these people who who seriously just don't want to do anything? And those people who are just going to move on, right? Yeah, there's a it is way easier to do nothing. And especially I have the opinion that a lot of security teams out there, it's kind of in the same vein that it's easier to to like the business doesn't ride security people and doesn't report the progress of their security to Wall Street every quarter. And and so they don't have the same sense of urgency that a sales team does in closing deals or a a product team and shipping new products and engineering team releasing the new feature. There's a completely different sense of urgency. And I think that really is not helpful for a lot of security people's causes. But also, it just makes a lot of people who have too much time to complain on Twitter and about things and and who don't want to do anything meaningful. Yeah, it's annoying to me as well, when especially when you see well -respected security people, but more often than not, they're less. They'll just come. They'll just make generalisms and say things like, oh, you know, passwords are dumb. It's like, OK, great. What are you doing that isn't a password? And it's dead silence. They aren't doing anything. They're just, you know, parodying whatever they heard. And yeah, the blog, the bad. It's clear what's bad. But the thing that I that is missing a lot of the times from a lot of that attitude is what can we do to improve, improve the bad and what can I guess when there's complaining and when there's basically sitting back and and doing nothing, it's basically people spending their their cost of doing nothing. When that happens, I think the attitude that's missing is, OK, there are flaws, but what what can we be doing to improve it? So, yeah, my solutions are solutions oriented, man. It comes down to that, too. You've got to be solutions oriented, not problem oriented. Yeah. And so I'm a I'm a big fan of that attitude. I read the blog post and I really liked it. I'd never heard it described as the cost of doing nothing. I'll keep that in my back pocket for the next one. Next, next time I'm speaking, I and then also you mentioned in the green room, you asked me, what do you think about certifications, security certifications? And I kind of get a bad rap sometimes, I think, because in my LinkedIn profile, I'll tell the whole story in my LinkedIn profile. I have not a CISSP in my title. And that's actually kind of poking fun at a somebody I used to work for when I was an intern who was a Ph .D. And one day when I was an intern, he had his email signature, whatever, whatever, comma, Ph.D. And one day he got his CISSP and his email signature updated to be whatever, whatever, comma, CISSP. And he removed the Ph.D. And I asked him, why are you removing the Ph.D. and adding the CISSP? And it was because it was he was working on a government contract at the time. And he they didn't care or the contract didn't care how many Ph.D.'s they had working on it. But they did care if there was CISSP that it mandated some amount of CISSP. And so he thought that that one was more important at the time and he updated it. And since then, I have been kind of had a healthy skepticism for for security certifications. And I don't think that they're the most important thing to making a good security practitioner. I think they're definitely helpful, just like compliance. Compliance is really, really important for businesses to show that they know they know what they're talking about. But also, if you look around, some of the biggest businesses in the world don't have PCI or ISO 27001. It really depends on who your customers are and all of that. Just like it depends for security certifications, who your customers are. So that was a funny story. Yeah, I asked that because I run into the same problem, right? People ask, should I go get a CISSP? And I don't know what to say. You know, as an employer, I would care more how well you did on CTFs or look at your GitHub repository every day than CISSP. But, you know, if you don't have if you're just trying to break into the business and you're trying to look for, like, how do I get started in the next four weeks? You know, I can't argue against it, but it's funny. The Ph.D., the Ph.D. story is hilarious. Yeah, I was I was I was really an intern, so I didn't know which one is more. I did a bunch of research afterwards. I was like, which one's better? Ph.D. I thought it was Ph.D., but like at CMU, the average Ph.D. is six point two years. And but CISSP doesn't CISSP have a requirement of a five years industry experience or something? I actually don't know. I haven't done a lot of research. Yeah. So I'm I don't want to be certification bashing. There's definitely value to them and everybody has a specific a specific need and and it's going to help in some parts of the security industry better than others. And it's all about understanding that. But, yeah, that's what the story is from why I have not a CISSP in my LinkedIn. Yeah, I think that's how we how I got to know you. Actually, I saw that and I was like, man, that's pretty awesome. And then generally career advice, you mentioned that you'd rather see GitHub commits, you'd rather see somebody's experience. I've always felt that with anything in tech, having work samples and being able to show that you've done something and built something is a great place to start and get your foot in the door, because people can see that if it's on your GitHub or if you've written a blog about something. What's your take, though, on how do you show CTF experience or how do you show that that what are some other things that you might recommend or look for? Well, I mean, I'm a little bit biased, right? So if someone had a Ph.D., for example, that's great if they want an academic post and I love academia, right? But it's not necessarily going to translate into business on the business end. Typically, if someone says they do CTFs, they'll tell me the team name and I'll go to CTF time and I'll look to see if they're a member and CTF time that I'll try to look at their rank. The other thing that people will do, just like with this idea of a portfolio, is they'll publish like blog posts about challenges that they've solved a lot of companies. You know, it's proprietary software, so they can't share projects they've worked on. But you can always share like here was a small problem. This is how we went about solving it. So I look for that and then just code, right? Like anyone who has code or anyone who has committed to a project, set up, you know, LiveFuzzer, set up AFL, written about reverse engineering. That's what I look for because security is a practice to start. Yeah. So what about non -technical roles? Because you're you're a CEO and you have to hire all types of people. What do you look for when it's not a technical role? I mean, this has been part of the difficult transition for me, right? Academia, it's always it should be the best idea wins. But soft skills are extremely important. So I was telling you the green room, it's kind of funny. I had more authority as a professor than a CEO. And so what I'm looking for are people who are solutions oriented, have a growth mindset and can try to get people behind their ideas without telling them what they have to do. What does that mean, though, you have more authority as a professor just because you were advisor to people or. Yeah, well, like in a classroom, right, like a person gets an A or a B. And there's very little besides their work performance that will change that if they're a PhD student. It's very objective how they're doing. Are they publishing a top tier conferences? Right. And you could just be like, no, you need to go write that paper when you get into especially Silicon Valley with like a zero percent unemployment. There's. It's harder to just tell people what to do, you have to figure out, you know, what it's going to do. So they're motivated to do it. And this was a learning experience. I got like an exec coach at one point. And this was her number one piece of feedback to me was like, people say you're often right, but it comes across as as limiting you as an executive because you're telling them what to do and later they'll realize you're right, but how they got there was a bad way. And so now I spend a lot more time trying to think about how do I get the people to the place they need to be? That's so interesting that you so the feedback that your coach gave you was that. Basically, instead of just giving people the answer, you kind of have to lead them there. Yeah. Yeah. Which which is weird to me. Yeah, that's that is interesting, though. Like, how did you receive that feedback? How did you think about it? I went into a depression spiral for a bit, right? So this exec coach is like, you're terrible at your job. And you've been able to get where you are because you're like really good technically and you can speak to a vision. So I had to it took me probably six months actually to overcome just that mental block. And so now I try to ask a lot more questions. Why I try to be prepared to let people fail and just focus on like like I said, that growth mindset, like how do we get 10 percent better? And that's the typical question I ask as opposed to trying to give the answer. That's awesome. Well, I find that really, really interesting. and it makes sense, like as CEO, you're that you'd have less less authority just because. People don't have to work for you, it's a choice and also and also they might have a different opinion on it's not like writing a paper, like you said, it's they might have a different opinion on the best way to proceed with a customer or with the product launch or whatever it is. And it's about managing those those situations. That's really interesting. OK, so I think we have about five minutes left and I wanted to ask you about CTFs, because what's your advice to people trying to get into CTFs? Because for me, I was a college student a while ago and I tried to do it and it was I felt like I had a pretty good grasp on things technically, but it one, I felt that they were really hard. I was in a CTF once and I couldn't solve this problem. I spent all day on it. And the guy at the end of the day was like, I think you were. I don't know. You he was really a jerk, but he's like, oh, you were supposed to paste in search for the prime number in this certificate and then and I was like, how am I supposed to figure that out? And so I felt like I was just constantly a string string of situations of of of things that I would have never figured out. So how how do people get started? Well, it's easier now. So I had the same issue. So like we run at Carnegie Mellon a couple of CTFs and one that we've been famous for is called Platt CTFs and it's representative of a lot of them. They're run by experts for experts, but that there's like zero zero on ramp into them, right? Like the idea used to be like five years ago that you're going to sit there and you're not going to solve a problem for the first three CTFs and then you will. And some people have got a lot of people got through that, but I don't think most would. And so we started designing ones for like high school students, and what was surprising is they started even being used in like the U.S. service academies, like the DOD, where we were trying to be a lot more methodical about what are the concepts we want to teach. Like, I remember really clearly the first the first problem that we designed for the high school one was to teach people how to Google or give permission to Google. It was like, what is this error code? Try to Google. And as soon as you Google, like the number one result would come up and you paste the answer. So I think I think it's getting better. The first advice is go find those beginner CTFs and they're more clearly labeled now, like Pico CTF. Yeah. And then it's just a matter of practice. It's just participating. And usually you end up specializing, so you don't feel like you have to be best at everything. Like my own the CTF team at CMU always says they're terrible at Web. They know binary exploitation. They're awesome at that, but they know nothing about Web security. They know something. I mean, they do pretty well. But yeah, but I'm a big I guess how you get started, you go find those. And I'm a big fan of them because they allow you to practice security skills. Like you'll read, you know, how RSA works. And then you'll be like, oh, I understand RSA. But the CTF challenges are like, well, what if you pick a bad prime? What if you pick a low exponent, like all these super important, actually software implementation details usually that will completely negate the security. And so it's kind of just a fun way to learn those sort of facts. Yeah, it's I guess I did OK in quite a few of them. What was there's an NYU one that they put on that I really enjoyed. It was probably way on the easier side. Yeah. CSAW has one. It's an it's a really good one, right? So, yeah, a lot of people, you know, the the most competitive one are the attack defense, like a DEF CON. And definitely don't get started with those. You'll just get creamed like like when you're playing an attack defense contest, you're playing to win, which means you always pick on the weak first. You try or exploit there because, you know, if it works, doesn't work on the weak. It's not going to work on the strong. And by the way, they may grab your exploit. Yeah. So look for those on ramps and the jeopardy based ones. He saws a good one because CTF, there's more and more every day. Right. Yeah. What do you think the characteristics of a person like are people's brains wired differently who are able to really get into this? I do think that that's probably true, too. They'll spend a long time working on a problem and they find that interesting. So people who are very, you know, into puzzles and will spend a lot of time just thinking about puzzles seem to be really good at them. Um, you know, I actually am a little bit like you. I have a harder time staying focused. So if it doesn't come across in like the first two hours, I'll probably move on. Yeah, I'm having trouble staying focused now. My cat really wants some attention. Well, that's that's really interesting. We've got about a minute, 20 seconds left. I just want to thank you for coming on, David. It was it was great having you. And anything you want to say about for all secure or about how people could find your company or get started learning more about fuzzing? Well, I think, you know, go to for all secure dot com. We're providing great service, I think, to get started. That's what we're trying to make easy for people. And I really do think that the world's going to change as they add in fuzzing because it's it's finding bugs, but it's also adding value to the developer. So if you want to get started, just go to our website for all secure dot com or you could just search fuzzer and read a lot of the tutorials. And is it are people able to press like are people able to get started on for all secure today? Like, are they able to start fuzzing? They have to contact us because we host the infrastructure, we pay for the fuzzing costs and all that. But just, you know, enroll. We try to make it super easy for everybody. Well, make sure everybody go enroll. Tell them I sent you not not David and and I appreciate you coming on. I think the world will benefit from more fuzzing. I know we are and we're trying to trying to secure everything one one bite at a time. So thanks for thanks for watching, everybody.
