Hacker Time

Thank you for joining us. This is Hacker Time. I'm Evan Johnson. I work at Cloudflare.

I work on our security team, working on all sorts of things from product security, application security, IT security, you name it.

I kind of am involved in it in some way or another.

And this is going to be my regular occurring segment at 8.30am pacific time.

And we're going to talk about security. I'm going to have a guest each week.

And this week, we're joined by none other than the famous security engineer, Stefan Edwards, would you like to take it away and introduce yourself?

Sure.

My name is Stefan Edwards. I go by handle logical. I am a practice lead at TrailerBits.

Basically, I help run our assurance practice with Dan Guido, who's the CEO of TrailerBits.

And then I also help out with various other things we do there.

Additionally, I work in our like threat modeling practice, our app sec practice or sub practice and then help out with whatever random stuff comes up that falls into my Canon wheelhouse.

So I wear many hats at TrailerBits, but it's definitely a fun place to do so.

Yeah, TrailerBits is really kind of over the last, how old is TrailerBits?

Like eight, nine years old? Yes, we started in 2012. So we're right at the eight year mark, roughly.

Nice. And they've really, I remember hearing of TrailerBits from our NETSEC years and years and years ago.

And that's how I first learned about it, but they've slowly grown and hired really good people over there from what I've seen.

Yeah, it's interesting. When I started, we were like 25 or 30 people.

And now we're like 60 to 65 people. So we've grown quite a bit in just two years.

So that's fantastic. Well, it's great to have you on.

And one of the reasons that I was really excited that you were interested in coming on was, I think you are really in the nitty gritty details of a lot of just, I wouldn't say obscure, but like, obscure.

I don't think it's obscure. It's really important, but just really technical.

And it takes a lot of ramp up time to learn. So like, systems engineering and system security, as well as, I know for a while you were on a cryptocurrency kick.

Do you want to talk about like some of your areas of interest for research and what's going on there?

Yeah, sure. So my biggest interest is programming language theory, program analysis, that sort of thing.

So symbolic execution, property testing, which is just a fancy word for unit testing and fuzzing.

Anything that basically allows me to consume code that people write.

At Trail of Bits, we use that across various different areas. So blockchain is certainly one area that we do that in.

We have tools to analyze languages and analyze programs and that sort of thing in that space.

But we also do quite a bit now in Go and whatnot, which is a little closer to my heart.

The core infrastructure of things, really getting into large blockchain code bases.

And when I say that, I mean the things that power blockchains, not like smart contracts and whatnot.

Although we do both. And then things like Kubernetes and other core container orchestration systems.

So it's been a lot of fun. We did the threat model and technical review of Kubernetes itself for CNCF.

So we've been getting our hands dirty with a lot of different Go codebases as well this year.

So it's fun.

It's a very interesting set of things to do and apply all those neat techniques to large codebases.

Yeah, what is a large cryptocurrency codebase? I don't actually have a good mental model of how...

There's this whole Coinbase thing, and I bought some Bitcoin when it was going bananas.

And I was like, I should probably buy some of this.

But I don't actually have a mental model of how that all works behind the scenes.

What is a large cryptocurrency codebase? If you're talking about the blockchains themselves and whatnot, they can be anywhere from 300 ,000 lines of Go to 1.2 million lines, 2.7 million lines of Go.

They're quite large codebases.

And the way that these are usually broken down are they usually have some sort of independent store, some sort of data structure that they use as backing system.

For example, interestingly, Tendermint uses an AVL tree. So there's actually an AVL tree in the backing store there that we ended up analyzing.

There are things that do distributed consensus replication.

So there's all sorts of fancy consensus algorithms that people have to write.

State machine replication, those sorts of things.

And then obviously, interface description languages for all of these as well.

So there's lots of implementations of proto buffers, lots of custom versions of binary wire protocols and things like that.

So we get our hands dirty in all these very weird areas of computing, which is fun.

What a mess. When you were kind of getting into that, I really haven't gotten into that world at all.

But when you were getting into that, did you notice all of the old security mistakes being remade?

Yeah. So Dan and I actually have a talk on this that is blockchain people need traditional security things as well.

We've seen a lot of the same sorts of issues played out over and over again.

And I think the blockchain world is struggling with the fact that, yes, they have some novel ideas, but they still need security fundamentals.

And that's kind of what my role, whenever I do blockchain assessments is like, hey, by the way, you also need to know, what do I do if I lose all my keys?

Or how do I back this up so that I don't accidentally expose keys to everyone?

So we do a lot of InfoSec fundamentals, like InfoSec 101 with blockchain companies as well, just to help out with that sort of thing.

Yeah, that's amazing. Every time there's something new, it's like, let's rewind security 20 years and make the previous 20 years of mistakes in one year.

Let's fix all of those in one year and bring it up to speed.

I mean, you want integer overflow and underflow in your high level languages, right?

I mean, you know, when you have integers, you want to not know by looking at them what they're actually going to do.

That makes a ton of sense for a modern, you know, like developer, right?

That's exactly what you want. And yet, like smart contract languages have gone back to very frequently back to what we had problems with in C, C++, that sort of thing.

And we're starting over again with a lot of these things.

Yeah, it's fun. That's horrifying. Well, that's awesome. What about the kind of symbolic execution and a lot of the work there that you, I don't know much about the research you've done in that space.

Symbolic execution is interesting.

For those who don't know, symbolic execution is basically a wave of either statically or dynamically walking over a code base in order to find constraints at specific locations.

So the canonical example is if you have like a check that tests if X is less than 10, and then the symbolic executor will actually take both paths and say, if X is equal to 10, we do these things.

If X is greater than 10, we actually do these things.

And this is the program flow. And that's glossing over huge amounts of information and things that are done and whatnot.

But at its core, symbolic execution is basically just path exploration and being able to find all the areas there, either, as I said, statically or dynamically.

Where we use it at Trail of Bits is a few different locations.

We have Manticore, which is a symbolic executor for native code as well as smart contracts.

So we use that quite heavily when we're trying to prove out some mathematical things that we don't want to manually do ourselves, but fuzzing may be inappropriate for.

And then we've also written symbolic executors to deliver back to clients.

So clients might have a custom language, we write one and hand it back to them.

So we actually do quite a bit with with Symbex here, which is always interesting and unique, you know, part of part of what's exciting for me, you know?

Yeah. This is kind of a little off topic, but the how much do you see different people writing software, kind of embracing these, all of these more advanced, more studied, and what the industry used to look at as academic ways to provide assurance for their software, like fuzzing, like symbolic execution, like, and then the big one formal methods that I'm hoping to see more of in the future.

So I think, in the blockchain world, we see quite a bit of formal methods, including up to and including formal specification.

So probably at the top, at the top of any formal method system that you have is formal formal specification, where you're actually writing something in a spec language, and then extracting code from that.

So we see quite a bit there in in, in the blockchain space, in the infrastructure space, we're starting to see more of it.

We've seen quite a few go code bases, adding invariants, adding, adding pre and post conditions, adding core logic to things.

There's quite a bit going on there.

So we're starting to see it. However, having said that, that's caveated by the fact that Trailbits is known for doing these sorts of things.

So obviously, more of our customers come to us for those sorts of things.

I do think in industry, though, the trend is heading that way. I know, folks are excited about like mayhem and all sorts of things.

There is a lot of a lot of new tools that are coming out here that help with like cyber reasoning, or any of those fancy new buzzword terms that we have for for these same tools.

I haven't heard of that.

What's cyber reasoning? Cyber reasoning is basically just program analysis for for mixed methods, right?

So you, you might have constraint extraction, like similar to to symbolic execution, but then you may do concretizing, you may generate actual data via fuzzing.

So you generate some lightweight constraints, and then you solve them via fuzzing rather than trying to solve them via a mathematical system such as SMT, or any of those sorts of things.

So it's, it's super useful. I know you're using it over there. So there's, there's quite a bit going on.

You know, in that space, it's super exciting going forward, you know?

Oh, yeah. Okay. So that's the buzzword for one of the what mayhem does what the product mayhem does.

Yeah, so cyber reasoning systems. It encorpuses it encompasses words, it encompasses quite a few different techniques, I think, like gray, gray box, you know, that sort of thing.

There's a number of techniques that fall under this, like cyber reasoning system.

But basically, it's a very nice way of presenting program analysis and giving security folks and developers tools to actually, you know, assess program state and program call flows and those sorts of things.

Yeah, it's super fascinating. It's a really good tool. We are that, that type of, like software assurance, type of tool and process is really great.

The tool mayhem that we're referencing is a tool that a company called for all secure cells and Cloudflare has been has been working and on and using that tool.

And it's fascinating the the we're going to have David Brumley, their CEO as our guest next week on the show on hacker time.

And so if you have questions, feel free to, to shoot them over to me on Slack.

And I'll, it'll save me some work thinking of questions to ask David.

But the it's, it's a pretty fascinating tool, because it makes it also makes things so easy for developers, they just take their code, they put it in a Docker container, and it's getting fuzzed, and it's getting and crashes are being found, and they have test cases automatically generated.

It's a really powerful tool.

And it's really simple. It's one of the hardest things about fuzzing for developers, I think is you show them AFL.

And and they're like, What is this?

How do I actually get set up with AFL, I have to learn this whole new tool chain, I have to keep my thing running for forever.

And the hard part is actually getting set up to fuzz.

But when there's a tool there that just like does it all for you, like there's no special compilation of your of your binary or anything, it feels a little magical.

Yeah, there's also techniques like property testing, you see this with hypothesis for Python, you see this with quick quick check for Haskell and whatnot, that make it much more like unit testing, rather than having to set up a special harness and things like that, that that are hugely problematic.

And that's, that's where fuzzing really can shine.

But cyber reasoning systems or systems that try to try to help bridge symbolic execution and fuzzing, though, they really can can make those sorts of things super accessible versus, you know, versus the traditional, like, build this harness and run things.

And hopefully, within the next two days, you have a crash.

Yeah, so yeah, that's, that's the hard part is it's a little bit of a waiting game.

How long do you wait for a crash?

It's if I want to push the production, how long do you wait for a crash?

To know it's okay. There's no amount of time. That's, that's good. But but just like it's, it's called software assurance at like certain way I think about it is at a certain amount of time, let's say 24 hours, you feel 90% confident at one week, you feel 98 99% confident.

Yeah, it's hard. I usually to be quite honest, despite having access to many of these tools, I usually start with radonsa or start with something super simple to do mutational fuzzing.

And I usually can get a bunch of crashes right away if there's going to be crashes.

Yeah. And then I step up from there and do all the fancy techniques that take, you know, days or weeks worth of time to run, you know?

Yeah, I mean, also, a cloud player, we have the world's biggest fuzzer, which is live traffic.

And so so if, if there's a problem, like we usually find out about it pretty quick.

And, and somebody pulls me into a conference room or something.

Yeah, yeah. Things tend to break when you have scale of, you know, the entire Internet against something there.

So it's much easier to figure out what's actually breaking, which is great.

You know, humans are amazing fuzzers.

Yeah, one thing that I think is interesting about the security industry is there, it's really service is heavy.

And trailbiz is mostly a consulting company, like maybe you can correct me if trailbiz is like working on rebranding and like selling different products and whatnot.

I'm not sure. But like, what do you think about consulting in the security industry?

And what do you think about just the security and industry in general?

How, how it's so services heavy? Yeah, so we do have a few products.

We we sell I verify, which is like a iOS, you know, security checker.

And we have critic, which is a basically productizing some of our open source tools.

But for the most part, we're a very services heavy industry. I've been in InfoSec for almost 13 years now, which is amazing, considering that I haven't burnt out and left to go, you know, do something else.

I mean, it's, it's honestly a depressing industry to be to be a consultant in, right?

You constantly see the worst side of people, the worst sides of applications, nothing gets better.

You know, and you're just slowly grinding away, doing this for your life and getting paid to do it.

But it can be rewarding when you have those customers that you can really deeply integrate with and help, you know, help them fix serious problems or serious challenges that they have.

But it is it is not in any way, shape or form a secure, like a easy industry to work in.

So yeah, I think as a consultant, you never really get to see things fixed, right?

Your job is to break it and then move on to your next billable two weeks.

Yeah, we we tend to have longer term engagements.

So it's a lot easier for us to see fixes. But even then, it can still be very difficult.

If clients need a large amount of time to do something, it can be very difficult to see something fixed there, you know.

Yeah. What do you think makes for a really good consulting engagement?

The longer term, like what what would you put at a minimum for if I'm going to go hire a third party tomorrow for something, especially the type of work you're doing, if I want them to review some, some really low level stuff, what's the minimum amount of time that I should give them with our code?

I mean, as much time as your budget can allow us, I mean, I'll take all of it.

No. Yeah, exactly. We generally we generally scope things based on, you know, the number of lines of code, the level of assurance that folks want the level of tests that they actually have.

So for example, we have customers that come in with specifications, and would like us to review their spec versus versus the actual implementation that they have, and help them actually understand where the delta between those two things are.

So it really depends on the level of code that the sorts of things that clients actually want out of out of there.

And we we tend to present a menu of options approach for things.

So for example, if they maybe don't understand architectural risk, offer them a threat model, lead the threat model into a technical review, lead the technical review, maybe into some sort of specification delta, some sort of delta fuzzing, that sort of thing.

But firms that are able to offer design review, offer technical review, and then offer offer the documentation help the coding help that sort of thing.

Or if you're doing network side, offer remediation help there.

I think that's that's where the biggest value for customers is when you can have multiple multiple offerings to provide.

Nice. That makes sense. You just mentioned threat modeling. And this is a hot topic for my team at Cloudflare because we do threat modeling with our engineers on everything from like the crazy low level systems to like we're adding a new feature in our dashboard.

And it is I've kind of struggled with getting the level of detail right and maintaining them and the whole process just feels a little off to me.

Like I'd love to hear your take on threat modeling and how to how to make it really great.

So we've done we've done threat models for Kubernetes and votes.

You can see but they're both actually public threat models within our publication index.

So I can mention that there's two customers that have threat models there.

I think I think the thing that most teams struggle with is is less the it is less the like how do I how do I find risks right as security people were able to look at things and just say that's risky.

I think the real issue is how do I cascade that out?

How do I know where to stop at what boundaries what concern points those sorts of things are usually the more difficult part that I've seen in threat models there.

So generally I think getting getting components down ahead of time actually understanding interconnections and having the ability to to rigorously drill into those things is is where it is the thing that I think most teams struggle with right.

Like what are all my first order connections to things but the problem that we have too is first order connections are great but the actual vulnerability might be third order.

So you have to know when you when you have something like that and then be able to catalog it and consume it later when you're doing the thing that's third order you know.

Yeah for us it's like we start looking at we start looking at two services that are talking and then there's all these how do you put a boundary around just those two services when maybe one's in a Kubernetes cluster and maybe one's at Cloudflare's edge or something and there's so many other things at play like the the system that's running the service and the network that it's on and all sorts of these other things it's tough to draw a boundary on it and then it's tough to take that artifact of like this is a threat model we wrote it up and then make it meaningful in a month and those are the two things that are really a challenge for for me at least.

The the thing I like to do for those is actually use Mozilla's rapid risk assessment.

So if I know there's there's a a process that's going to change very quickly Mozilla has this thing called rapid risk assessments.

We've modified it to work with the way that we work at Trello Bits but basically Mozilla has like a 30 to 60 minute meeting where you can actually you know go through tell me how this tell me how this service works tell me this sort of data that it processes and let's let's talk through some threat scenarios to make sure that you actually you know are are solving things that impact CIA the and when I say CIA I mean confidentiality, integrity, and availability.

At Trello Bits we've modified it to be the control families that clients have so instead of asking like stride style questions or dread style questions or pasta or got dirt or any of those sorts of things.

You know them all. Well it's almost like like threat modeling is my job.

It's true it's true. But no so the the we ask a very control focus like okay so what do you do for what do you do for input validation like let's talk about your input validation techniques let's talk about what sort of data you have what boundaries you have what systems you may relay that information to and we have output output encoding or output you know output validation controls whatever they are and just walk through those sorts of things.

But I know folks will probably take umbrage with the fact that I'm using a control focused approach but whatever you're doing to to actually you know get through a series of risks and make sure that you actually understand everything that's happening that's that's the really important thing for for threat modeling itself.

That makes sense. I have heard of the rapid risk assessment but I'm glad you brought it up again because I haven't heard about it in a long time and I it's like I'm relearning it I just bookmarked it.

Thanks for passing that on. Yeah it actually if the interesting thing about doing the Kubernetes assessment is that we released all of our core notes behind it so there's actually a GitHub repository with all of the rapid risk assessments from the the Kubernetes teams that we talked to which is is interesting as well.

Not only did we release the final report but we also released like hey here's a bunch of random notes that we have with with really smart developers from really smart places who work on Kubernetes.

You might want to read these as well.

Nice. Yeah super interesting. I can link you in chat as well.

Okay cool. I do want to get to the security news and that was all fascinating but this I thought that these vulnerabilities that have been coming out this week were really interesting.

There's a few big ones. Citrix released a series of CVEs.

It looks like 10 or so CVEs of really serious issues and F5 big IP there was was it one CVE or it looks like one main CVE and they're fascinating bugs.

The F5 vulnerability was if you look at the proof of concept the dot dot semicolon payload is just brutal.

I know exactly what's going on there. It's 1999 again.

Everything is fine. I mean and the problem is too F5 is like very core infrastructure.

They've been around for a while and this could happen to any of us that have legacy products or products that have come from legacy I should say.

It's hard to understand what your thing is doing but that's also just a brutal vulnerability to have.

People are scanning Shodan for it right now. Yeah it's a bloodbath right now.

My hot take on this is that if I'm using one of those systems as somebody running my business with it like I should as a security person at that company I should expect that at all times there's like a non-zero number of criticals in those these older kind of legacy products and that's just part of the game.

Yeah yeah well I think if I recall correctly F5's documentation actually says not to expose the management interface to the Internet regardless.

So even if there was no there was no vulnerability there they actually say like don't put this on the Internet.

Now naturally we have to access things remotely and therefore we put things on the Internet all the time.

Yep. So that's part of the bloodbath there but it's very difficult when you have these sorts of things to keep things locked down and secure because as security people we also get lazy too.

It's not just devs and it's not just everyone you know. It's just one bad IP ackle somewhere that you end up with with this thing totally on the Internet having no idea or one bad like vlan setup or who knows who knows how you how you do this.

Well and it can happen to anything right like there's there's nginx vulnerabilities that are we may consider silly but when you're trying to when you're trying to write something it's hard if you don't have a spec or you don't have tests or whatever to to make sure that the thing does not do something else that you did not expect you know.

Yeah even with speaking of nginx like even if you have a spec it are the RFC for HTTP and all of these things are written and then you go read the fine print and you're like well I could see it going either way we need like a supreme court for for HTTP specs.

Well forget about HTTP specs look at the caching spec and then look at what firefox does versus what chrome does.

Just look at those two.

There was a firefox vulnerability just recently in twitter where dms were being cached because twitter was not sending no store which if you know caching firefox actually requires to not store something.

So if you if you set no cache in firefox it will it will store it but not cache it but if you set no store it will not store it and not cache it which is perfectly legal per the spec but is different from what other browsers do.

Yeah so it's it's a fun thing and these things are very difficult to do you know.

Yeah well that's my my big hot take that like these these products you just have to prepare for them to always have criticals and they shouldn't be exposed to the Internet like you said it's in their documentation but it's just like so such like I guess is the big one of the year there's always a few maybe it's not the big one of the year like it's 2020 who knows what's coming.

It's early don't jinx us. Yeah knock on wood maybe it gets worse but these seem pretty bad and whenever I see a CVSS of 10 I am definitely paying attention.

Yeah it's it it's it's hard to because if you're old like me you still remember the like f5 cheat sheet of all the things that you would get wrong in default f5 installations that you have to tell developers about.

Well thank you so much I will I really appreciate you coming on and it was I thought a fantastic talk and I don't know if anybody's watching now anymore I assume not but next week we'll have David Brumley and it was it was great having you I really appreciate it.