Originally aired on March 26, 2021 @ 11:30 AM - 12:00 PM EDT
Join Evan Johnson as he speaks with security professionals about recent security news!
This week's guest is Scott Piper, AWS Security Lead, Aurora. Scott Piper ran an AWS security consulting business for the past 3.5 years until he recently started working at Aurora, a self-driving vehicle company, to lead AWS security efforts.
All right, we're live on Hacker Time. This is the number one security show anywhere in the world. We have a very esteemed guest today, Scott Piper, joining me for Security Week. Thank you for joining us, Scott. How are you doing? Doing well, yeah. It's starting to warm up out here in Salt Lake City. Nice. The weather has been nice in San Francisco here too, this past week, which has been great. People have been able to get outside. But what, I mean, I'm from Cloudflare's security team, and I'm here every week, but can you give us a little introduction about yourself? Yeah, so for the past three and a half years, I ran my own consulting business just by myself that was focused on AWS security. And so I did that for up until about a month ago, ended up taking a full -time job with a company. But we'll talk primarily, I'm guessing, in the show about the aspects of running a consulting business and what all that's been like in the security world. Yeah, you're being far too modest about what you were doing the past three and a half years. You weren't running just a little AWS security shop. You were the face of AWS security in the industry. It is funny. After I decided to shut things down, I actually changed. So Summit Route is the name of my company. And on summitroute.com, I changed the homepage. I had to change it to something. And so I just put Summit Route was the world's greatest AWS security consulting business. There's a lot of competition for that. And it was probably true. We worked with you here at Cloudflare and only high marks having worked with you. But what was that like going from being thrust into the forefront of a really hot part of the industry? Yeah. So it was wild. I mean, the whole situation, a lot of luck, a lot of being in the right place at the right time, a lot of having the right connections and things. Because I had done primarily a lot of software development previously. And then just prior to starting the consulting business, I was the head security person for a startup, which meant I was the only security person, which meant I was doing absolutely everything. So I was doing not only our cloud security and our app sec and whatever else, but also our badge readers, surveillance cameras, physical security things. I remember they were asking me, when are we going to get armored vehicles? What happens if there's an active shooter? I'm not doing any of those things. I'm not prepared or qualified or anything like that for those. But yeah, so then decided to focus on one thing. I was like, I just want to do AWS security, just that one little niche there. And so yeah, I feel like when I first started doing it, there wasn't really a lot of activity, especially in terms of other consultants out there, there wasn't as much competition really. And so I really fell into a position where I became the only person doing it. And so de facto, the best person in terms of AWS security consultants, someone focusing just specifically on that. And yeah, and then ended up doing a lot of public things with it. And so released flaws.cloud, which is like an online CTF type thing, released CloudMapper and CloudTracker and a number of other open source tools that I developed with Duo Security. And so through all that, and then blog posts and things, was able to just kind of build up my brand, I guess. Yeah, that's really interesting. And so I was thinking, how did it start? It didn't start with flaws .cloud. How did- It did. Oh, it did? So flaws.cloud, I actually released while I was running security for that other company. And I had built it as like, I didn't know about AWS security. I didn't know a lot of the concerns there. So I'm looking on HackerOne, what are the bug bounties that have been AWS related? What are some of the research out there? What are the Black Hat talks and DEF CON talks and things like that? And then I was like, okay, I'm going to try and test these things out. Okay, I think I can make kind of a little challenge for our DevOps people on our team in order to teach them about these things. And oh, I'll just make it as like a challenge for myself to be able to make this a public thing. If I'm actually going to be confident about IAM policies and whether or not I've set them correctly, then yeah, I should be ready to put an access key out to the world that I'm confident is not going to get abused in ways in which I don't expect it to. And so yeah, ended up releasing Floss.cloud and that became super popular. And then there were various reasons why I wanted to kind of move away from being head of security of that company. Again, because there were like just so many aspects of different security things that I was not capable of, confident in, and just wanted to focus on one thing. And so I just decided, okay, I've never done consulting before. I just figured I'd jump into it and see what happened. And luckily was able to have like a lot of really good customers that also helped promote me and helped introduce me to other customers. And that just kind of helped build the business and made it sustainable. Nice. And so when you released Floss.cloud, was it like an overnight success or did it blow up? It was so weird. I mean, I think, I don't remember at the time how many Twitter followers I had. Maybe I had a few hundred because I used to do a lot of reverse engineering stuff. And so I had some followers from that. And so now I have like 10,000 followers. I'm an influencer, I guess. But yeah, I expected when I released it, like, oh, maybe if it's the right day on Twitter, it's a slow day. Maybe a dozen people will check it out or something like that. And instead, like the first month, 30,000 people checked it out. And I was like, oh my gosh, like there is clearly something here. So like with that, and then also I had this like meetup group reach out to me and they were like, hey, we want you to present in our next meetup, you know, we're in Chicago. And I was like, oh, that'd be cool. But I'm in Denver at the time, you know, so it's not going to make sense for me to fly out there, get a hotel, you know, like go through all that hassle. And they were like, okay, well, we'll pay for your flight and hotel. And I was like, wait, what? Like, I've never met this person before. And they're willing to write me a check for a few hundred dollars. Like I'm not making money out of that situation, you know, like that's not going to be like, you know, a sustainable business, you know, because I'm not actually profiting. But like someone who's never met me is willing to write me a check for a few hundred dollars, you know, on a whim. Like, okay, there is definitely like demand there for somebody doing these things. I'm not like super good at reading like the market and stuff. But like I could see like, okay, 30,000 people checked out this site, someone's willing to pay me, you know, people started emailing me being like, hey, can you help us out with our AWS security and stuff? Like, clearly there was demand, you know, and then also like part of the reason I dug into it was because while I was running security at that company, I had tried to find the consultant that was doing AWS security. I was like, this is an area I'm not super confident in. I would like someone else to double check my stuff. And I couldn't find anybody. And so I knew that there was, you know, this demand problem and no supply. And so I was like, let's see what happens with it. Nice. That is so interesting. And so you release Floss.cloud, it's an overnight success, you start doing this thing, you leave your previous job and you start your own consulting business. And I guess, how long did that take to figure out on your own? Was there any concern? I mean, I guess when you're instantly successful with it, successful enough with it, you feel like you're not going to starve. But was there concern about like, how do I run a business? Yeah, so I did get super lucky. My first client was Duo Security. And the way in which that had worked was I had interviewed with them like a number of years ago. And interviews went great. We're each like, yeah, let's work together. And they were like, you know, we want you to work, you know, in our office here in Ann Arbor, Michigan. And I was like, there are no mountains anywhere around there. That's not going to work for like me and what I enjoy in life. Can I work remotely? They're like, not at this time. Now they do allow remote work and everything. But so we were just like, you know, let's stay in touch. Like, you know, hopefully we can work together again at some point in the future. And so then when I basically kind of announced to the world that I was doing this consulting, they're like, hey, we allow remote work now. Do you want to, you know, do some consulting work for us? And so they were actually my first client and they were phenomenal. Absolutely amazing for so many different reasons. I mean, like, they at one point, you know, kind of had like a heart to heart discussion with me where they're like, hey, so like this AWS security stuff, the technical things that like we've hired you to do, like you're great at it. You're doing a real good job there. But like business-wise in terms of, you know, like, you know, you, you knowing how to run like a business and invoice and things like that. They're like, you have no idea what you're doing, do you? I was like, I was like, you know, no, I do not. Like I've never, I've never done any of those things before. They're like, you know, we, we work with a number of consultants and stuff. You know, how about we set up some time for you to talk to each of those different consultants and they can explain to you how to run a business, you know, like we've done a lot of work for them. They owe us some favors, and I was like, oh my gosh, like absolutely phenomenal. So the best client you could possibly ask for. And then additionally, like the work that I did with them was actually spread out, like did like a quick kind of like two week contract with them and then like a six month contract or something like that. And then there was like another follow-on contract. And so, so because of that, like, luckily I wasn't, you know, like starving because I had these long-term contracts with them and stuff. But, but yeah, I mean, there, there definitely has been like ups and downs in terms of you know, it's like feast and famine in, in this kind of world because different companies have you know, times in which they suddenly have money that they need to spend. And honestly, I still haven't entire, you know, I never entirely figured that out. You know, it seemed like, okay, end of the year. Yes. There are some companies that suddenly like need to burn off cash really quickly, you know, otherwise it's just gone, you know, and similarly at the beginning of the year, suddenly they're, they're planning their budgets and things like that. And so, you know, they're planning out like, okay, we want to, you know, bring you in for such and such we'll do this in like, you know, July or, you know, six months away in the future. So there, there are some things like that. So, so yeah, I mean, still trying to learn that, but I mean, even during my consulting, like it was, it was successful. Like I had, I had clients on like five continents or something, you know, like dozens of clients around the world, different industries, yada, yada, yada, super amazing. But at the same time, I mean, there were, there were periods of like, you know, two, three months where I'd go without any, any client work whatsoever. So no income whatsoever. And then suddenly, yeah, it would just hit and like, I'd be booked up and things would be great for a while. So. Yeah. That's interesting. I think a lot of security people see, there are a lot of consultants out there and people doing security consulting and it kind of just looks like a different change of pace for a lot of people working in industry on a blue team or internal security team. And the way I've always looked at it is every security team has a big consulting budget because that's just the way the industry works. You have auditors, you have third parties who look at your code, whatever it is. And so, and I guess it seems natural that, that people feel like they can, they can figure something out. And it seems like, seems like if you're a good security person, you probably won't starve. Even if you know nothing about business. It's a booming industry. And I mean, especially, you know, I think, especially if you can identify a niche of some sort and get good at that and market yourself in that niche, like there's going to be a lot of opportunities because, you know, we, we all recognize that, you know, in, in security and technology in general, like it is, it is moving so quickly. There are so many like advanced areas and things like that. And in security, if you, you know, make a couple of mistakes here or there, or, you know, in, in, in large enterprises, like you, you, you have so many blind spots, you know, you realize, and so there's so many different places where, you know, you, you have the opportunity to start consulting businesses. Like I've, I've tried to advocate in the past, like someone just needs to become like a G suite security consultant of some sort, because there's so many companies that like, don't realize how many, you know, publicly shared files they have, you know, or whatever other type of, you know, configurations they have for their G suite. Like do they have, you know, like SPF, DNS, DNS, what is it called? But basically like to, to disable spoofing and, and stuff like that of your emails or identifying like how many of your employees have, you know, installed various Chrome extensions and how can you audit those, or how many employees, you know, are forwarding their emails to a private account of some sort, you know, like all these different things that you can audit in just G suite, like easily, easily someone could, you know, sell some type of assessment, you know, for a couple thousand dollars, you know, maybe $10,000 or something like that. And, and be able to make like a decent business out of that. So. Yeah. I want to go two ways with this. The first is I mean, being a software engineer at heart and having a engineering background, didn't you want to build a product at the same time? Yes. But so, so there's like, so from what I'd heard from people, and I think this is true is that it is difficult to run both a consulting business and a product business at the same time, because it is two different businesses, you know, like if you assume like the success rate for a single business, you know, Oh, like for, for security, it's going to be pretty good, but overall in the world, like it's success for a single business is not that great. And now if you're trying to do two businesses, the one time, like your chances of being successful in both of them, like are going to become more difficult. And there's a number of reasons for that. Not only because it's two different businesses, but because you're being pulled in two different ways. And then there's also the fact that companies will be cautious. I think a lot of the times as to whether or not you, if you do try and do consulting for them, whether or not you're just trying to sell them your product, ultimately, you know, like if you pay someone to do this consulting and they come in and they tell you, you know, like, Hey, you know, what you really should do is you should use this other product. And they're like, did we just pay you to give us a sales pitch? Like, no, like we want you to solve our problem. You know, we don't want to also, you know, have you spend your time like marketing, you know, your product and stuff to us. So, so there's like that problem there. But I mean, what, what I did for my own business was I basically was building these open source tools, maintaining, and, you know, those open source tools, adding to them and stuff and using that for my consulting business. And that model was basically to advertise my consulting business via these open source tools, you know? And so a lot of times, like people would bring me in because I was already a trusted entity because they knew like, okay, this, this open source tool, this is the maintainer for it. Like, we know that he knows his stuff. We know that, you know, that, you know, I'm, I'm generally trying to do good in the world, you know, by, by making these open source tools accessible to people. But a lot of times people just don't have like the time to, to learn that tool that well. And so, so that model, I think, worked out pretty well for me. Yeah, that, that seems like it makes sense. I guess I've always thought about selling a product a different way. It's like, oh, I have this great product that sells, that solves your problem, but it, I'm sure it can come across the wrong way. Yeah. Selling your own. And there's like, there's so many different ways to like have a business and run a business and make money out of a business. So, you know, there, there's, I was oftentimes approached by a number of companies because I had, you know, this consulting business, I was interacting with a ton of different companies. And so a lot of product companies or SaaS vendors would approach me and say like, hey, you know, while you're giving training to a company, you should like tell them about our product, you know, like, we'll, we'll pay you to do that, or we'll give you referral fees or something like that, you know, or, or, you know, when, when I would do consulting and people would approach me and say like, hey, you know, you did a great job on your, on the AWS security for us. Can you also do such and such other, you know, maybe Azure security or something like that? And that's not a thing that I did. And so, you know, and I think that some of the other consulting vendors out there knew that, you know, I had a certain focus, a certain niche. And so they would be like, hey, you should refer us, you know, we'll give you referral fees or something like that. And so, so I personally, like, I did not have any of those business relationships. I did not want to, you know, I felt like, I felt like it could become awkward, you know, you could have some, some situations in which people felt like they were not getting the best value out of you and that you were trying to pitch things for other people. But at the same time, like, if you are, you know, one of the go-to people in a field, like you probably have connections with people that are doing adjacent type of work. And, you know, for things that you can't do, like, you don't want to just tell someone like, no, I can't do that. Like, go, good luck, go try and find them on your own. You know, like as, as just a good business, you want to be able to tell them like, sorry, I can't help you. But I do know that this person over here is pretty good at it. Maybe they can help you out with that thing, you know? And so, so I never had like referral fees, but I would oftentimes like try to direct people to at least somebody that I knew that could maybe help them with something. Yeah. That's interesting. I want to switch gears, but one last thing I'll say before we do is it's really interesting. People want you to advertise for their product. It is, it, one of my personal like pet peeves is you go to all of these like conferences and you see all of the marketing spend of a lot of security SaaS businesses and like they'll invest in parties in Las Vegas or a huge events, but none of them will invest in making it so you can actually sign up to use the product on their website. And I've been, I've been fortunate to have worked at Cloudflare with where you just like can sign up on the website in the free tier and try it out and, and self-service and also LastPass where it was the same kind of model. And, and it grinds my gears seeing companies that, that will say, Hey, we'll give you 20% if you advertise for us, but also like won't invest in making their product good drives me nuts. But I wanted to switch gears seeing cause you got to also look at all of these different companies, you set up five continents you tons of tons of tons of breadth in what you saw. I'm curious what you saw most people struggling, like the security teams, what were they struggling with? Not necessarily implementation details of like keeping their load balancers enforcing TLS 1.2 only for PCI compliance, but like what were their, the goals of those teams, the commonalities between them? So, I mean, so it's, it's like anything, like once you see how the sausage is made, like it terrifies you. And like, like, cause I, I worked for, you know, all these different companies and maybe, maybe my opinion is biased in some way, or my view of the world is biased because these are companies that decide to reach out to somebody, you know, and get help with things and stuff like that. But at the same time, I mean, there were, there were a number of companies that, you know, have, have strong security programs and things like that. And I would do an assessment for them and I would still find public S3 buckets and stuff, you know, things, things that you think are like the very basics of, you know, security, but it's, it's just hard, you know, you, you have, you know, people don't recognize that security teams oftentimes are, you know, just overwhelmed with, with a lot of, you know, the duties that they have and trying to keep up with these things. So, you know, some of, some of those basics. And then on, on top of that I mean, it's having, having other things like just public EC2s and stuff like that, that's like a hard problem to some degree to deal with on AWS and also with kind of just the classic way in which people have used AWS and the way in which people are used to doing certain things. Like you, you can restrict, you know, all of your developers from being able to make a public EC2. And, you know, there, there are ways of accomplishing those types of things. But a lot of people's environments, like they just don't allow those types of restrictions, you know, because they want, they want their companies to be able to move fast. They want their engineering teams to be able to try different things out. And, and so they kind of have to allow that and then have the security team hopefully be able to identify these problems and react to it quickly. It tends to be like the, the general strategy, whether companies want to do that or not, you know, whether they recognize that's their strategy or not, they're oftentimes reactive. Yeah. I feel like a lot of times companies, companies have to grow, especially a lot of these newer companies and younger companies, they have to grow and to grow, you have to ship and to ship you need velocity. And, and there's inherent risk with velocity. Like you have to, you, it's something that our security team tries to keep in mind because we try to stay in front of all of our big releases and, and, and are really intentional about, about knowing the things that we will compromise on and the things that we won't compromise on. And actually I've heard a great, and, and knowing where those guardrails are, and I've heard a great analogy of this, the car and the brake, braking systems in cars allowed cars to go faster because you could actually slow down when you needed to versus so the security team should be like the brakes of a car, not the guardrails, which I, I like that one. I forget where I heard it, but yeah, it is, it is a hard problem. Also like one, one of the problems, you know, specifically with AWS as to why this is, you know, done in this way is if you do try to implement the different guardrails. So if you do restrict your developers from being able to create public EC2s or, you know restricting them from creating EC2 that doesn't have the right tags or doesn't have an encrypted hard drive or something like that AWS provides you with no feedback as to why you were denied something. And so that I think is a very problematic thing from a usability perspective is when you do deny your developers, they just get an access denied. It's just generic. They don't know, they don't know, you know, was it because I didn't have the right tags? Was it because I tried to give it a public IP address? Was it because, you know, the, the hard drive wasn't encrypted? You know, they have to like manually try and check through all those different things. There, there's no easy way to understand why it was denied. And so that is something that, that myself and a number of other people that, you know, do a lot of AWS security have like, you know, harped on that AWS needs to have an improved system for that just because, you know, it is very difficult to have, to use AWS securely and in a usable fashion, you know, that's not going to, you know, upset your developers too much because there aren't getting any type of feedback when they are denied from things. That's interesting. We actually just came across an example exactly like this while we were going about our work at Cloudflare where we, we began enforcing security key usage a little bit ago. And so in order to get into any of our internal things, you have to like press your little security key. I have it right here in my, oh, it's blurry. I have right here. And we hated that the error message was just access denied. So we actually had to build a feature into Cloudflare access, the product that we're using to enforce the security keys that gives a nice, you can like link to something and it says, you have been denied by the security team because you didn't use your security key. Log out, log back in, use your security key. Cause yeah, I mean like debugging things like that is such a pain and AWS can become more problematic because you do not have the ability to see what, what the policies are that are blocking you. So from an account, you can't see what are called the SCPs. And so, so as a result, you as a developer do not have visibility into why something was denied in any way. You cannot, you cannot debug that on your own. You cannot like, you know, it's, it's a total mess. So there are some usability improvements that, that I believe strongly need to happen there. Love it. Well that's a, that's an interesting commonality of problems. Just, I mean, if you're getting to the point where you're denying developers, you have some amount of maturity in your program because you have like this role-based access control. But then that, that breeds new problems. It's really interesting. Well, I have a hot take that I try to, I try to always throw in my own opinions and see what other people think. This is a great opportunity to just, I don't know, get other opinions on things. I kind of brought this up just a minute ago where I'm tired of seeing security software businesses that aren't self-service where it's all locked away. You press like request a demo in order to see anything and try it. And you have to talk to like four people and do a sales call. Do you think like, why do you think that is in the security industry? Do you think that that's going to change? Do you think that things are going to get more self-service? Because the thing that I'm seeing is security is just so expensive for companies to try to address. Yeah. So, I mean, I don't have a lot of knowledge about whether or not this problem or how, what, to what degree this problem exists outside of security. But it is, it is super frustrating. I mean, I, I've been in positions, you know, in, in past roles and stuff where I, I needed, I wanted to buy a certain product. I knew that this was the product that I needed. I used it as a previous company, at a previous company and stuff, you know? And so I talked to, you know, the, the company, I was like, yes, I want to buy your product. How much, like here, here's is a briefcase full of cash. Let's make this happen, you know, within the next hour. And, and they cannot do that. You know, like they, you, you cannot approach a company and purchase some of these products within like an hour timeframe or something like that. You know, you have to get on multiple sales calls and it's going to take multiple days and they have to give you these demos. And, you know, like some of these like salespeople, they want to tell you about, you know, the Gardner charts and stuff like that. And you're like, I already know, I already know I want to buy your product. Like just give me a price and we can work from there. Like, what is your pricing model? Stuff like that. And that, you know, that, that all is a mess too. And I've heard you know, of companies as, as somebody has moved from one company to the next, they have realized that the company has changed the pricing model based on whatever, whatever new way in which like this, this vendor decided to price things. One of the most ridiculous ones recently was I heard of a vendor that was pricing things based on how much funding the company had received. So based on what their like most recent, like, you know, valuation was, or most recently, you know, like round of funding was this company would ask for, you know, a percentage cut of whatever that funding was, which is absolutely like insane, you know? But I mean, it's not much different from like, I've heard of, you know, a lot of AWS vendors will charge you based on what your monthly spend is, you know, or something along those lines. Like there's all these crazy different pricing models out there and it's a mess. So, but, but to your point, I mean, yeah, you, I think oftentimes we want to use the vendors that are easiest to work with. And part of being easy to work with is I can just sign up, I can start using the product and go. And I think a number of vendors maybe are hesitant about that because like, you know, they, they think that like that initial trial, that initial, you know, visibility into your environment or into an application is going to give away too much information about how their product works. And, you know, so I don't know. Well, Scott, we, we just ran out of time. I want to thank you for, for coming out and talking like you were a fantastic guest.