Hacker Time

Thank you for joining us. This is Hacker Time. I'm Evan Johnson. I work at Cloudflare. I work on our security team, working on all sorts of things from product security, application security, IT security, you name it. I kind of am involved in it in some way or another. And this is going to be my regular occurring segment at 8.30am pacific time. And we're going to talk about security. I'm going to have a guest each week. And this week, we're joined by none other than the famous security engineer, Stefan Edwards, would you like to take it away and introduce yourself? Sure. My name is Stefan Edwards. I go by handle logical. I am a practice lead at Trail of Bits. Basically, I help run our assurance practice with Dan Guido, who's the CEO of Trail of Bits. And then I also help out with various other things we do there. Additionally, I work in our like threat modeling practice, our app sec practice or sub practice and then help out with whatever random stuff comes up that that falls into my, my cannon wheelhouse. So I wear many hats at Trail of Bits, but it's, it's definitely a fun place to do so. So yeah, Trail of Bits is really a kind of over the last, how old is Trail of Bits? Like eight, nine years old? Yes, we started in 2012. So we're right at the eight year mark roughly. So nice. And they've really, I remember hearing of Trail of Bits from our NETSEC years and years and years ago. And that's how I first learned about it. But they've slowly grown and hired really good people over there from what I've seen. Yeah, it's interesting. When I started, we were like 25 or 30 people. And now we're like 60 to 65 people. So we've grown quite a bit in just two years. So that's fantastic. Well, it's great to have you on. And one of the reasons that I was really excited that you were interested in coming on was, I think you are really in the nitty gritty details of a lot of just, I wouldn't say obscure, but like, obscure. I don't think it's obscure. It's really important, but just really technical. And it takes a lot of ramp up time to learn. So like systems engineering and system security, as well as I know for a while you were on a cryptocurrency kick. Do you want to talk about like some of your areas of interest for research and what's going on there? Yeah, sure. So my biggest interest is programming language theory, program analysis, that sort of thing. So symbolic execution, property testing, which is just a fancy word for unit testing and fuzzing. Anything that basically allows me to consume code that people write. At Trail of Bits, we use that across various different areas. So blockchain is certainly one area that we do that in. We have tools to analyze languages and analyze programs and that sort of thing in that space. But we also do quite a bit now in Go and whatnot, which is a little closer to my heart. The core infrastructure of things, really getting into large blockchain code bases. And when I say that, I mean the things that power blockchains, not like smart contracts and whatnot, although we do both. And then things like Kubernetes and other core container orchestration systems. So it's been a lot of fun. We did the threat model and technical review of Kubernetes itself for CNCF. So we've been getting our hands dirty with a lot of different Go codebases as well this year. So it's fun. It's a very interesting set of things to do and apply all those neat techniques to large codebases. Yeah, what is a large cryptocurrency codebase? I don't actually have a good mental model of how... There's this whole coinbase thing and I bought some Bitcoin when it was going bananas. And I was like, I should probably buy some of this. But I don't actually have a mental model of how that all works behind the scenes. What is a large cryptocurrency codebase? If you're talking about the blockchains themselves and whatnot, they can be anywhere from 300,000 lines of Go to 1.2 million lines, 2.7 million lines of Go. They're quite large codebases. And the way that these are usually broken down are they usually have some sort of independent store, some sort of data structure that they use as backing system. For example, interestingly, Tendermint uses an AVL tree. So there's actually an AVL tree in the backing store there that we ended up analyzing. There are things that do distributed consensus replication. So there's all sorts of fancy consensus algorithms that people have to write, state machine replication, those sorts of things. And then obviously, interface description languages for all of these as well. So there's lots of implementations of proto buffers, lots of custom versions of binary wire protocols and things like that. So we get our hands dirty in all these very weird areas of computing, which is fun. What a mess. When you were getting into that, I really haven't gotten into that world at all. But when you were getting into that, did you notice all of the old security mistakes being remade? Yeah. So Dan and I actually have a talk on this that is like blockchain people need traditional security things as well. We've seen a lot of the same sorts of issues played out over and over again. And I think the blockchain world is struggling with the fact that, yes, they have some novel ideas, but they still need security fundamentals. And that's kind of what my role, whenever I do blockchain assessments is like, hey, by the way, you also need to know, what do I do if I lose all my keys? Or how do I back this up so that I don't accidentally expose keys to everyone? So we do a lot of like, InfoSec fundamentals, like InfoSec 101 with blockchain companies as well, just to help out with that sort of thing. Yeah, that's amazing. Every time there's something new, it's like, let's rewind security 20 years and make the previous 20 years of mistakes in one year. Uh, let's, let's fix all of those in one year and, and bring it up to speed. I mean, you want integer, integer overflow and underflow in your, your high level languages, right? I mean, you know, when you when you have integers, you want to not know by looking at them what they're actually going to do. That makes a ton of sense for a modern, you know, like developer, right? That's, that's exactly what you want. And yet, like, smart contract languages have gone back to very frequently back to what we had problems with in C, C++, that sort of thing. And we're, we're, we're starting over again with a lot. Yeah, it's fun. That's horrifying. Well, uh, well, that's awesome. What about the kind of symbolic execution and a lot of the work there that that you, I don't know much about the research you've done in that space. Symbolic execution is interesting. Uh, for those who don't know, symbolic execution is basically a way of, of, uh, either statically or dynamically walking over a code base in order to find constraints at specific locations. So the canonical example is if you have like a check that tests, if X is less than 10, and then the symbolic executor will actually take both paths and say, if X is equal to 10, we do these things. If X is greater than 10, we actually do these things. And this is the program flow. And that's glossing over huge amounts of information and things that are done and whatnot there. Uh, but at its core, symbolic execution is basically just path exploration and being able to find all the areas there, either as I said, statically or dynamically, um, where we use it at Trail of Bits is a few different locations. We have Manticore, which is a, a, uh, symbolic executor for native code as well as smart contracts. Uh, so we use that quite heavily when we're trying to prove out some mathematical things that we don't want to manually do ourselves. Uh, but fuzzing may be inappropriate for, uh, and then we, we've also written symbolic executors to deliver back to clients. So clients might have a custom language. We, we write one and hand it back to them. So we, we actually do quite a bit with, with Symbex here, which is, is always interesting and unique, you know, part of, part of what's exciting for me, you know? Yeah. Um, this is kind of a little off topic, but the, how much do you see different people writing software kind of embracing, um, these, all of these more advanced, more studied and, uh, and what the industry used to look at as academic ways to provide assurance for their software, like fuzzing, like a symbolic execution, like, uh, and then the big one, formal methods that I'm hoping to see more of in the future. So I think in the blockchain world, we see quite a bit of, of formal methods, including up to, and including formal specification. So probably at the top, at the top of any formal method system that you have is, is formal, formal specification where you're actually writing something in a spec language and then extracting code from that. So we see quite a bit, uh, there in, in, uh, in the blockchain space, it, in the infrastructure space, we're starting to see more of it. Um, we've seen quite a few go code bases, adding invariants, adding, adding pre and post conditions, adding core logic to things. Uh, there's, there's quite a bit going on there. So we're starting to see it. However, having said that that's caveated by the fact that trail bits is known for doing these sorts of things. So obviously more of our customers come to us for those sorts of things. I do think in industry though, the trend is heading that way. I know, uh, folks are excited about like mayhem and all sorts of things. There's, there is a lot of, a lot of new tools that are coming out here that help with like cyber reasoning or, or any of those fancy new buzzword terms that we have for, for these same tools. I haven't heard of that. What's cyber reasoning? Cyber reasoning is basically just program analysis for, uh, for mixed methods, right? So you, you might have constraint extraction, uh, like similar to, to symbolic execution, but then you may do a concretizing, you may generate actual data via fuzzing. So you generate some lightweight constraints and then you solve them via fuzzing rather than trying to solve them via a mathematical system such as S and T or, or any of those sorts of things. So it's, it's super useful. I know you're using it over there. So there's a, there's quite a bit going on, uh, you know, in that space. It's super exciting going forward, you know? Oh yeah. Okay. So, uh, that's the buzzword for, uh, one of the, what mayhem does, what the product mayhem does. Yeah. So cyber reasoning systems, um, it, it encorpuses, it encompasses, blah, words. Uh, it encompasses quite a few different techniques, I think. Um, like gray, gray box, uh, you know, gray box fuzzing or, uh, concolic symbolic execution or concolic execution, that sort of thing. There's a number of techniques that fall under this like cyber reasoning system, but basically it's a very nice way of presenting program analysis and, and, and giving, uh, security folks and developers tools to actually, you know, assess program state and program call flows and those sorts of things. Yeah. It's super fascinating. It, it's a really good tool. We, uh, are, are that, that type of like software assurance type of tool and process is really great. Um, the tool mayhem that we're referencing is, uh, a tool that a company called for all secure, uh, cells and Cloudflare has been, uh, has been working and on and using that tool. And it's fascinating. The, uh, the, we're going to have David Brumley, their CEO as our guest next week on the show on hacker time. And so if you have questions, feel free to, to shoot them over to me on Slack and I'll, it'll save me some work thinking of questions to ask David, but the, uh, it's, it's a pretty fascinating tool because it makes, it also makes things so easy for developers. They just take their code, they put it in a Docker container and it's getting fuzzed and it's getting, uh, and crashes are being found and they have test cases automatically generated. It's a really powerful tool. Um, and it's really simple. It's one of the hardest things about fuzzing for developers I think is you show them AFL and, um, and they're like, what is this? How do I actually get it set up with AFL? I have to learn this whole new tool chain. I have to keep my thing running for forever. And the hard part is actually getting set up to fuzz. But when there's a tool there that just like does it all for you, like there's no special compilation of your, of your binary or anything. It feels a little magical. Yeah. There's also techniques like property testing. You see this with hypothesis for, for Python. You see this with quick, quick check for Haskell and whatnot that, uh, make it much more like unit testing rather than having to set up the special harness and things like that, uh, that, that are hugely problematic. And, and that's, that's where fuzzing really can shine, but cyber reasoning systems or, or systems that try to, uh, try to help bridge, uh, symbolic execution and, and fuzzing, uh, though they really can, can make those sorts of things super accessible versus, uh, you know, versus the traditional, like build this harness and run things. And hopefully within the next two days, you have a crash. Yeah. So, yeah, that's, that's the hard part is it's a little bit of a waiting game. How long do you wait for a crash? It's, uh, uh, if I want to push the production, how long do you wait for a crash? Uh, to know it's okay. Uh, there's no amount of time. That's, that's good. Uh, but, but just like it's, it's called software assurance at like certain way. I think about it is at a certain amount of time, let's say 24 hours, you feel 90% confident at one week, you feel 98, 99% confident. Yeah. It's hard. I usually, to be quite honest, despite having access to many of these tools, I usually start with Redonsa or start with something super simple to do mutational fuzzing. And I usually can get a bunch of crashes right away. If there's going to be crashes. Yeah. And then I, I step up from there and do all the fancy techniques that take, you know, days or weeks worth of time to run, you know? Yeah. I mean, also at cloud player, we have the world's biggest fuzzer, which is live traffic. And so, uh, so if, if there's a problem, like we usually find out about it pretty quick and, uh, and somebody pulls me into a conference room or something. Yeah. Yeah. Things tend to break, uh, when you have scale of, you know, the entire Internet being pounded against something there. So it's much easier to, to figure out what's actually breaking, which is great. You know, humans are amazing fuzzers. Yeah. One thing that I think is, uh, interesting about, uh, the security industry is there it's really services heavy and trail bits is mostly a consulting company. Like maybe you can correct me if trail bits is like working on rebranding and like selling different products and whatnot. I I'm not sure. Uh, but like, what do you think about consulting in the security industry? And what do you think about, uh, just the security and industry in general, how, how it's so, um, services heavy? Yeah. So we, we do have a few products. Um, we, we sell, I verify, which is like a iOS, uh, you know, security checker. Um, and, and we have critic, which is a basically productizing some of our, our open source, uh, tools, but for the most part, we're a very services heavy industry. I've, I've been in infosec for almost 13 years now, uh, which is amazing considering that I haven't burnt out and left to go, you know, do something else. I mean, it's, it's honestly a depressing industry to be, to be a consultant in, right. You, you constantly see the worst side of people, the worst sides of applications, nothing gets better. Um, you know, and you're just slowly grinding away, doing this for your life, uh, and getting paid to do it. Um, but it can be rewarding when you have those customers that you can really deeply integrate with and help, uh, you know, help them fix serious problems or serious challenges that they have. But it is, it is not in any way, shape or form a secure, uh, like a easy industry to work in. So, yeah, I think as a consultant, you never really get to see things fixed, right. Your job is to break it and then move on to your next billable, uh, two weeks. Yeah. Yeah. We, we tend to have longer term engagements, so it's a lot easier for us to see fixes. Um, but even then it can still be very difficult if clients need a large amount of time to, to do something, it can be very difficult to see something fixed there, you know? Yeah. What do you think makes for a really good, uh, consulting engagement? The longer term, like what, what would you put at a minimum for, if I'm going to go hire a third party tomorrow for something, especially the type of work you're doing. If I want them to review some, some really low level stuff, what's the minimum amount of time that I should give them with our code? I mean, as much time as your budget can allow us. I mean, I'll take all of it. No, um, you're hired. Yeah, exactly. We, we generally, we generally scope things based on, uh, you know, the, the number of lines of code, the level of assurance that folks want, the level of tests that they actually have. So for example, we have customers that come in with specifications and would like us to review, uh, their spec versus, versus the actual implementation that they have and, and help them, uh, actually understand where the Delta between those two things are. So it, it really depends on the level of code, the sorts of things that clients actually want out of, out of there. And we, we tend to present a menu of options, uh, for approach for things. So for example, if they maybe don't understand architectural risk, offer them a threat model, lead the threat model into a technical review, lead the technical review, maybe into some sort of specification Delta, some sort of Delta fuzzing, that sort of thing. But, uh, firms that are able to offer, uh, design review, offer technical review, and then offer, offer the documentation help, the coding help, that sort of thing. Or if you're doing network side offer remediation help there, I think that's, that's where the biggest value for customers is when you can have multiple, multiple offerings to, to provide. Nice. That makes sense. Um, you, you just mentioned threat modeling and this is a hot topic, uh, uh, for my team at Cloudflare because we do threat modeling with our engineers on everything from like the crazy low level systems to like, we're adding a new feature in our dashboard. Uh, and it is, uh, I've kind of struggled with getting the level of detail, right. And maintaining them and the whole process just feels a little off to me. Like I'd love to hear your take on threat modeling and how to, how to make it really great. So we've done, we've done threat models for Kubernetes and Votes. Uh, you can see, they're both actually public threat models within our, our publication index. So I can mention that there's two customers that have threat models there. Um, I think, I think the thing that most teams struggle with is, is less the, um, it is less the, like, how do I, how do I find risks, right? As security people, we're able to look at things and just say, that's risky. I think the real issue is how do I cascade that out? How do I know where to stop at what boundaries, what concern points, those sorts of things are usually the more difficult part that, that I've seen in, in threat models there. So, um, generally I think getting, getting components down ahead of time, actually understanding interconnections and having the ability to, to rigorously drill into those things is, is where it is the thing that I think most teams struggle with, right? Like what are all my first order connections to things? But the problem that we have too is first order connections are great, but the actual vulnerability might be third order. So you have to know when you, when you have something like that and then be able to catalog it and consume it later when you're doing the thing that's third order, you know? Yeah, for us, it's like, uh, we start looking at, we start looking at two services that are talking and then there's all these, uh, how do you put a boundary around just those two services when maybe one's in a Kubernetes cluster and maybe one's, uh, at Cloudflare's edge or something. And there's so many other things at play, like the, the, uh, system that's running the service and the network that it's on and, uh, all sorts of these other things. It's tough to draw a boundary on it. And then it's tough to take that artifact of like, this is a threat model. We wrote it up and then make it meaningful in a month. Uh, and, uh, those are the two things that are really a challenge for, for me, at least. The thing I like to do for those is, uh, actually use Mozilla's rapid risk assessment. So if I know there's, there's a process that's going to change very quickly, Mozilla has this thing called rapid risk assessments. We've modify it to work with the way that we work at Trail of Bits. But basically Mozilla has like a 30 to 60 minute meeting where you can actually, um, you know, go through, tell me how this, tell me how this service works. Tell me this sort of data that it processes. And, uh, let's, let's talk through some threat scenarios to make sure that you actually, uh, you know, are, are solving things that impact CIA. The, and when I say CIA, I mean, confidentiality, integrity, and availability. Yeah. Um, at, at Trail of Bits, we've modified it to be the control families that clients have. So instead of asking like stride style questions or dread stock questions or pasta or got dirt or any of those sorts of things, um, well, it's almost like, like threat modeling is my job. It's true. It's true. Um, but no, so the, the, we ask a very control focused, like, okay, so what do you do for, what do you do for input validation? Like, let's talk about your input validation techniques. Let's talk about what sorts of data you have, what boundaries you have, what systems you may relay that information to. And we have output, output encoding or output, uh, you know, output validation controls, whatever that are, and just walk through those sorts of things. But I know folks will probably take umbrage with the fact that I'm using a control focused approach, but whatever you're doing to, to actually, uh, you know, get through a series of risks and make sure that you actually understand everything that's happening. That's, that's the really important thing for, for threat modeling itself. That makes sense. I, I have heard of the rapid risk assessment, but I'm glad you brought it up again. Um, because I haven't heard about it in a long time and I, it's like I'm relearning it. I just bookmarked it. Thanks for passing that on. Yeah, it actually, if the interesting thing about doing the Kubernetes assessment is that we released all of our core notes behind it. So there's actually a GitHub repository with all of the rapid risk assessments from the Kubernetes teams that we talked to, which is, is interesting as well. Not only did we release the final report, but we also released like, Hey, here's a bunch of random notes that we have with, with really smart developers from really smart places who work on Kubernetes. So you might want to read these as well. Nice. Yeah. Super interesting. I can link you in chat as well. Okay, cool. Um, I do want to get to the security news and, uh, that was all fascinating. Uh, but this, I, I thought that these vulnerabilities that have been coming out this week were really interesting. There's a few big ones. Um, Citrix released a series of CVEs. It looks like 10 or so CVEs of really serious issues and F5 big IP. There was, um, I was at one CVE or, uh, it looks like one main CVE, uh, and they're fascinating bugs. The, the F5 vulnerability was, uh, if you look at the proof of concept, the dot dot semi-colon payload, uh, is just brutal. I know exactly what's going on there. Yeah, it's 1999 again. Yeah. Yeah. It's, it's 1999 again. Everything is fine. I mean, and the, the problem is too, F5 is like very core infrastructure. They've been around for a while and it's, it's, this could happen to any of us that have legacy products or products that have come from legacy, I should say, you know, it's hard to understand what your, your thing is doing, but that's also just a brutal vulnerability to have. Like people are scanning Shodan for it right now, you know? Yeah. It's a bloodbath right now. The, uh, my hot take on this is that if I'm using one of those systems as somebody running my business with it, like I should, as a security person at that company, I should expect that at all times, there's like a non-zero number of criticals in those, these older kind of legacy products. Uh, and that's just part of the game. Yeah. Yeah. Well, I, I think if I recall correctly, F5's documentation actually says not to expose the management interface to the Internet regardless. So even if there was no, there, there was no, uh, vulnerability there, they, they actually say like, don't put this on the Internet. Now, naturally we have to access things remotely and therefore we put things on the Internet all the time. Uh, so, so that's part of the bloodbath there, but, uh, it, it's, it's very difficult when you, uh, when you have these sorts of things to, to keep things locked down and secure because as security people, we also get lazy too. It's not just Debs and it's not just everyone, you know? It's just one bad, um, IP ACL somewhere that you end up with, uh, with this thing totally on the Internet, having no idea or one bad, uh, like VLAN set up or who knows, who knows how you, how you do this? Well, and it can happen to anything, right? Like there's, there's NGINX vulnerabilities that are, uh, we may consider silly, but when you're trying to, when you're trying to write something, it's hard if you don't have a spec or you don't have tests or whatever to, to make sure that the thing does not do something else that you did not expect, you know? Yeah. Even with, speaking of NGINX, like even if you have a spec, the RFC for HTTP and all of these things are written and then you go read the fine print and you're like, well, I could see it going either way. We need like a Supreme Court for the, uh, for HTTP specs. Well, forget about HTTP specs. Look at the caching spec and then look at what Firefox does versus what Chrome does. Just look at those two. There was a Firefox vulnerability, uh, just recently in Twitter where, uh, DMs were being cached because, uh, Twitter was not sending no store, which if you know caching, Firefox actually requires to not store something. So if you, if you set no cache in Firefox, it will, it will store it, but not cache it. But if you set no store, it will not store it and not cache it, which is perfectly legal per the spec, but is different from what other browsers do. Yeah. So it's, it's a fun thing and these things are very difficult to do, to do, you know? Yeah. Uh, well that's my, my big hot take that like these, these products, you just have to them to always have criticals. Um, and, um, they shouldn't be exposed to the Internet. Like you said, it's in their documentation. Uh, but it's just like, so such like, I guess is the big one of the year. There's always a few, uh, maybe it's not the big one of the year. Like it's 2020. Who knows what's coming? It's early. Don't jinx us. Yeah. Knock on wood. Uh, maybe it gets worse, but these seem pretty bad. And, uh, whenever I see a CVSS of 10, I am definitely paying attention. Yeah. It's, it's, it's hard to, because if you're old like me, uh, you still remember the like F5 cheat sheet of all the things that you would get wrong in default F5 installations that you have to tell developers about. Well, thank you so much. I will, I really appreciate you coming on and it was, I thought a fantastic talk. And, um, I don't know if anybody's watching now anymore. I assume not, but, uh, next week we'll have David Brumley and, uh, it was, it was great having you. I really appreciate it.