Cloudflare TV

Hacker Time

Presented by Evan Johnson
Originally aired on 

Join Evan Johnson as he speaks with security professionals about recent security news!


Transcript (Beta)

All right, we're live. We had a bunch of, right before the end of this, or right before I hopped in here, we had some technical difficulties.

I have no idea if anybody's watching.

Maybe they are, maybe they aren't, but I'm going to do my best and do a half hour of programming with you all.

So thank you for joining me on Hacker Time, the number one security show in the world, anywhere out there.

And I'm going to share my screen.

I am going to additionally head over quickly to, see if this is working, see if it's not.

We'll see whether it is or not, that's okay.

And I look like I'm live. So thank you all for being here. When you're running your own live stream and when you're running your own TV channel, there's bound to be hiccups.

So this week we're going to be doing a bunch of programming, picking up right where we left off two weeks ago, cat free this time.

Cat cow was driving me crazy last time with all the meowing, but today she's in a nice quiet area with some food and water to keep her happy while I'm programming.

And so picking up where we left off, we were working on a honeypot, the cheapest form of detection anywhere and what we were doing was we had built an HTTP listener.

We built a Twilio notifier and then we were building out more listeners.

So I wanted to add an SSH listener and we were working on that. We kind of got the SSH connection thing working, but we have some improvements to do.

The first thing we need to do is, let me just open up a new tab here. Number one, I am in Vim mode in VS code.

And so number one, we need to make it so it can handle multiple SSH connections.

Make a listener handle multiple SSH connections. And then number two, we need to make sure that the listener is performing authentication properly.

Because last time when I was logging in, it was a little suspect.

I was logging in with like keys that it didn't have and all this random stuff and it was still working for some reason.

I'm not really sure why. Probably should understand that. And then anything else?

I can't think of anything. This will be some kind of low level debugging much more than a lot of the debugging we've done in the past.

And so this might take the entire episode.

If we have any time or if I get into some weird time, where I have like just a couple of minutes at the end, I might switch over to some chess puzzles and do some chess practice.

If I don't have time to dive into two after solving one.

Okay. So let's do this thing. We have our main program. We've got our SSH command.

We've got our HTTP command and this all looks normal. And here's our actual SSH connection handler.

And this was all completely copy pasta.

And so you know what that means. I have no idea how it works. We walked through each section of it and like kind of understood it.

But overall there could be quirks with the way this code works or handles things that we are not privy to.

So the first thing we wanna do is make it survive multiple SSH connections.

Right now I'll show you the current behavior.

So the honeypot is running. I'm gonna SSH to local host.

It got closed by the remote port for whatever reason. I wonder if it's because I don't have a key, which means we should do 22 local host dash IDRSA.

And the connection was closed.

Why is that? This is running correct. Okay.

Allow. We restarted. And this is no longer working. That's a great place to begin debugging.

Why is this no longer working? Let's run SSH dash B so we can see.

Oh, it looks like it's returning HTTP. So that's a problem, right? Why would this be returning HTTP here?

So it must be something wrong with my honeypot.

I've got HTTP running this. I think I need to do this H should be an S.

I think that'll fix it, but just shows my code is a little bit spaghetti-ish.

There's some quirks here.

Okay. Yeah, correct. That was the reason why now it's trying to load all of that SSH kind of stuff.

And we need to run this from a very specific directory, the honeypots SSH directory, I believe.

The reason is just because, oh, open authorized keys.

No such file or directory.

To run this from actually, we had an authorized key file.

So when the SSH honeypot boots up, I know what happened here.

We're having a problem because I did a get clean before I committed the SSH code and get clean because I didn't check in this authorized key file kind of blew away this required directory up.

Our file, cause it's reading IDR say, and what it needs to read is this authorized key file.

So this thing needs to exist.

So first things first, let's cat SSH. Where is my I don't know where my key went.

My keys are gone too.

Okay, that's fine. We'll just do an SSH key gen right here, right now.

And we will, okay, we've generated our public private key pair.

We're going to cat IDR say pub into a file called authorized keys.

And then our thing should boot. Oh, run CMB.

All right, our honeypot is running.

Now, when we SSH in, we shouldn't get HTTP back.

We should get an SSH connection. That was really funky. All right. So look at this.

Our honeypot crashed and it says RSA host key check has changed and you have requested strict checking.

Okay, and my host key verification failed. Look at this.

This is a great security mechanism to understand in SSH. So SSH, it's a trust on first use protocol.

So when you SSH in, the trust on first use is the client trusting the host key of the server.

And you memorize that, the client will memorize that key and stick it in a file itself and make sure that it sees the same key in the future.

So you're not getting tricked into connecting somewhere that you shouldn't.

So two things, let's clear that key from our client and it should be a known host.

I got a bunch of them here.

And this is the ones that we need to delete. So we'll no longer get this remote host.

Identification has changed. It is possible that someone is doing something nasty.

We won't get that error message anymore because we're basically saying, deleting the previous trust that we had.

We're gonna do a new trust on the first use now.

And I think we'll still get that end of file crash though.

That'll be a good one to debug.

Yes, trust this fingerprint. Okay, I didn't think we'd get this far.

Password is ASBF, permission denied. Okay. The password's in here.

We have built into the software. We have test user and tiger. Oh no, if user is test user and password is tiger.

So no matter what we type in, it should fail.

Let's type in tiger though. Make sure it fails. It failed, very good. Okay.

Here we go. When I killed the connection from the client, the server crashed.

So we did get a server crash. Password rejected for my username E. Let's try a different username.

Let's try test user. I have to run the server, allow.

Okay, test user.

Let's try, we're using password based off. Let's try password ASDF.

Permission denied, very good. Okay, let's try tiger. All right.

We got a crash, no pointer dereference here in our go code. Very neat.

But it clearly tried to go through the authentication flow and then something broke.

So we can be very sure that it's working. The authentication is working. Probably not very sure, but we're like reasonably sure.

And let's try to track down where the stack, where this dereference happened.

Looks like it was SSH go line 93.

Line 93. Logged in with key. Okay, so I bet this con permission extension map thing I bet this extensions thing is nil.

Yikes. I don't know anything about this SSH server con object.

Let's just take a look before I just delete this thing.

Let's see what it does. Go lang SSH server con. And it is of type server con.

Okay. Server con has a permissions, right?

Permissions as a pointer. And this should be a map.

Permissions is, has an extensions, which is a map string to string, which has all the like permit agent forwarding all of that.

And it looks like either this is nil.

Let's just print the whole thing. Let's just see. So you can kind of print a map and go.

It works for the most part. It's not going to print in the best format, but you'll at least be able to see kind of what's going on, kind of the contents of the map, unless it's just pointers, in which case you'll just see pointers, which aren't strings.

So it won't be super helpful. Test user tiger.

Okay, before it crashed, it said permissions was nil. So I'm not connecting with any kind of special connection stuff.

Before we do this, we need to do if con.permissions is not equal nil.


Oh, let's not do an else.

We wrapped this up with a, the if statement. So it shouldn't crash anymore.

Let's go to the next bug. We're just squashing bugs. One after the next.

Okay. This time it didn't crash. Look at that. I type in ls. We've got our echo shell again.

All right. Whatever we type in, the program will get right back to us.

So very cool. But now when I end the connection, it crashes. That's what we need to figure out why.

So how does this connection actually get started?

It's gotta be something with these go functions and the channels and stuff.

Why is that channel request new channel except service the incoming channel channel.

Wow. We've got a lot of stuff going on here. Consumes and rejects all requests from the past channel.

So how is this working?

We have a listener and then end con gets used for this new server connection, which returns a con connection, a channel and a sage request and an error.

And then when we get a connection, so this is going to be a very challenging way of doing this, but in the spirit of just hacking stuff together, I honestly wonder what would happen if we just put this in an infinite loop here.

This is not how you write production code.

Let me just tell you that first. That's not what we're doing here.

We're packing stuff together. This is fun. Okay. I actually wonder if this will work.

I'm not sure if it will, but the idea behind the theory in theory, what's going on here is we're listening on port 2022.

We're accepting connections and then they're initiating a server connection.

Looks like one.

And I believe this is like locked either here or here.

One of these is blocking probably here. That's why I put it here.

And so what this is going to do is when we kill the connection, it'll just restart.

And what I think that means is that at most only one person will be ever able to connect to this, but that might be okay for this little toy.

Let's just see if this hack worked.

So what should happen when I log in, kill the SSH connection, the server shouldn't crash.

It should, or it's kind of gracefully ending execution, but it should restart a new list.

It should listen for another connection. So let's see.

Test user tiger. Okay. Now when I close the connection, I'm going to press control C.

The SSH honeypot should not die. And it totally died. Use of close network connection.

Okay. So very, very nice.

We're close. I think this hack is going to work. I'm just going to move this up here.


All right. I bet this works. Okay.

Things working. Close it. And we can connect again. Things working. Close it. We can do this.

Okay. We fixed it. But here's the thing. Here's the quirk. I can only connect once, I believe.

So let us do, I'm connected right now here. If I do the same thing again, SSH.

If I run SSH test user at localhost-p2022 -i-idrsa.

Okay. It's just hanging. Actually, wait, I don't want this.

Yeah. Still just hanging. It's just waiting for the server.

And that's because there's only one connection. If we really wanted to make this work properly, just look at what we would need to do.

We would need to listen on a...

I'm not sure exactly what we would need to do.

There's definitely some open heart surgery we would need to perform on this section of the code to be able to handle multiple connections.

When an SSH connection comes in, it should spin off a go routine that handles all of the actual session stuff.

And it should continue listening. But for now, this is actually plenty because we're able to get a notification.

Remember for our honeypot, all we need is to have that one connection come in, be notified that the connection was made and then send off a notification to the notifier.

And so as long as...

If it was like an actual remote shell, like SSH is meant to be, that would probably be a usability problem where it can only handle one connection at a time.

That'd be an issue.

But for us, probably not. This is good enough. I like it. So, what now?

We've got about eight minutes. We fixed our first problem. Let's actually check in our code because it's been a while.

Let's go through git diff.

Look at all the problems we had to fix. Cool, cool. Some re-architecting.

This all looks good. It's been a while. Git add. So we just add all known files and we do wanna add these actually.

We don't wanna add this as a binary, but we do wanna add honeypots SSH HTTP and honeypots SSH.

That's from the re -architecting we did. And then we can commit working SSH honeypot.

Woo, okay.

What was the other thing we wanted to do? Check, this is done. Listener performs authentication.

We kind of made it do that, but let's, I think we should research more about how the key -based authentication works because let's see if there is support for keys in this code that we copy-pasted.


We have authorized keys, so there must be. We're reading all of the authorized keys.

So these are the public keys of the clients that are meant to connect to us.

We're parsing those. This parsing must be working because it's not returning an error here.

Otherwise, the whole thing would crash. Then it must be defaulting to the client.

And then we're going to do a password-based. I wonder if it should just work with the key if we pass it in because we have authorized keys and id are the same.

So that means we should be able to SSH with this RSA key.

Let's try it. Okay, we're going to run the thing and then we're going to, and it's running.

All right, hooray. Now let us, okay, cool.

We got the password thing. Oh, that's a good one. With, it crashes here. If you air, if you kill the connection here, the whole thing will crash.

So that's something for us to fix.

That's not handled by our infinite loop. Unknown public key for test user.

Let's try to do that. Failed to handshake. No auth passed yet.

Failed to handshake. So this is a log .fatal. Instead, we probably should do a log.println.

Oh gosh, what did I just do?

Oh, when you're in Vim and you press caps lock on accident.

Okay, let us do a log.println here and continue.

And that should work. Any of the fatals here, we probably don't actually want.

Fatal will just crash it and it'll never come back, which is kind of an issue.

And at least, oh boy, it auto corrected me.

Okay. And this should be good, right? Yeah, no more fatals.

Okay. Let's see if this worked. Let's try to do the same thing.

Password based off and then see if it will. We killed it and now it works. Man, we're fixing tons of bugs here.

Okay, cool. Cool. Now with our three minutes left, let's try key-based off.

F-I-I-D-R-S -A. Logged in with the key. Okay. It worked.

And now what?

Kill the thing. Looks like it survived another connection. Very cool. We have way made this thing more robust.

And then last, I want to generate a ROM key, try to SSH with the ROM key and be rejected.

I think that's a very important piece to make sure that the authentication is working properly.

So let's generate an ED25519 key, SSH keygen-type, ED25519.

Some fancy crypto, ED25519. So we've got two keys in here.

We're using the RSA key, but now let's try it with the 25519 key and we should see a failure.

All right.

So it went to password based off, which means it tried the 25519 key. It didn't give us a notification though.

It didn't like give us a log line. So let's try it.

I don't know why I just restarted the server. So we can see offering public key, the 25519 key and no bueno, it did not work.

So we were not able to log in with the improper key, which means this is working quite well, I would say.

That's pretty cool. I'm actually really surprised that just kind of worked.

So I would say done and step two, listener performance authentication. I would say done.

It's actually performing it properly. We didn't have to do any work on that one.

Let us go back here.

And so is there anything left to do? One thing that's kind of noticeably absent, the, I'm not getting text messages on my phone when these SSH connections are coming in.

What we need to do, this is probably the final step to this, which we can do next week.

And maybe we can talk about what the next project is after this, because I think this was kind of a cool thing to work on for a while.

We need to implement the actual messages to be sent.

And we need to make sure that there's no kind of funky off-ramps in the code where you can avoid notifying on an SSH connection.

So what I'm talking about is if I SSH as the client and I break the connection while typing in my password, I need to be notified.

The Honeypot shouldn't still notify the detection folks.

Additionally, when you make a full connection, it still needs to notify.

I also wonder since we're messing with the SSH protocol so much, if we can just make it a full-on shell just for fun to see, do that as an exercise, that might be fun.

I've never actually done that before.

So that'd be fun. But I really appreciate all of you joining me on Hacker Time this week.

This was a lot of fun. I think we did some cool debugging here and actually got this code working and hacked it together.


Thumbnail image for video "Hacker Time"

Hacker Time
Join Evan Johnson each week for the latest security news, programming tutorials, and more!
Watch more episodes