Hacker Time

Hello and welcome to Hacker Time. I'm your host Evan Johnson from the Cloudflare product security team and I'm so happy you've all joined me this week.

This week at Cloudflare is a very special week.

It's birthday week. If you're watching Cloudflare TV, you probably know what birthday week is, but it's the anniversary of the forming of the company and every year we have so many great announcements that we make of new products, of new initiatives, of company strategy, and it's a great cadence every year because all year people look forward to what we're releasing internally and externally and it's a lot of fun.

It's a lot of work on the things and I'm really happy to see them get announced and get released.

The announcement today being how Cloudflare fits into Web3, the Cloudflare distributed web gateways private beta.

A long time ago we launched an IPFS gateway and we're continuing to try to figure that out.

A few Web3 blog posts, some great announcements in stream, WebRTC related.

They're announcing Cloudflare TV as a service.

Anybody can run their services, their Cloudflare TV, their own version of Cloudflare TV using the same services that we've built.

And then Cloudflare for offices is a huge announcement. Moving hardware directly into office buildings around the world, moving Cloudflare closer to people than ever.

There's a ton here. Definitely recommend going and reading some of it, seeing what you're interested in.

If you're a candidate trying to get a job at Cloudflare, reading this is all really helpful in understanding the company better.

Yeah, so that's my plug for birthday week. There's so many great announcements.

It's hard to do it all justice, especially as somebody who works on the security team with so many of the announcements in pre-production when it's still just an idea.

So it's hard for me to do it all justice. So that's not what today's episode is about.

Definitely go check this out, but today we're going to do something different.

Today we are going to do a little bit of Zen programming. You'll see my Cloudflare, my CFCLI here.

I guess a personal announcement. This is a personal project, something that I built that works with Cloudflare's API command line interface.

And it doesn't have a ton of traction. It has 80 stars or so on GitHub, which it's not nothing.

It's pretty good. But I'm going to, I think, probably start working on this again.

And I'd like to bring it into the 21st century or the year 2021.

I worked on this three years ago for one week and then I stopped. It still works.

It's still a really good product in my opinion. But we've released so many new features, so many APIs that it's woefully out of date.

So if you'd like to contribute, feel free.

I'm going to go over how to contribute today and how to work on this, how to get started and try to contribute.

I'm going to try to make a pull request by the end of the episode.

And if not, that's okay. There's probably going to be some bugs because there's some cruft here.

You'll note that I'm using some older Golang tools like DEP and we'll eventually need to get rid of those.

But today should be a pretty low key episode because I've had only three cups of coffee instead of my usual five.

And so we'll just start programming and see where it goes.

And I'm going to try not to talk and type at the same time because I have, I'll show you this.

My keyboard is a Unicomp Endura Pro IBM Model M kind of copy.

And so it is deafeningly loud. I'm not sure if the camera and the microphone will pick it up, but I don't want to take the chance.

I'll try not to talk and type at the same time.

So step one to contribute to CF. First, you got to clone it.

So it's pretty easy to clone. You do one of them, get clones from GitHub.

You just copy here, get clone. Everybody's done it. Pretty straightforward.

And then building, to build, I run DEP ensure. I believe you need the old DEP tool right now.

And I'm already up to date, but this will download all of your dependencies that you need for the project and make sure that are all in this go lock package toml thing here.

I'll show you. So all the dependencies in here have actual version numbers or revision hashes, and it shows you exactly what you need.

So there's that.

So step one, I think some of this is documented here to get it built, but step one, you maybe this is out of date.

Maybe I should update this, but you definitely have to run DEP ensure after a clone.

Yep.

Okay. Step two is getting credentials set up. I think I've talked a little bit about the credentials before here and it's worth revisiting, but you run CF configure.

Let's say you build the program like me and it's running. Let me make this a little smaller.

You build a program and it's running and you can run CF and see the nice command line output.

Well, then you need an API key to talk to Cloudflare's API. And so the command to do that is CF configure.

And this will ask you for your email and API key.

And then what's called an origin CA API key. This is a specialized API key. If you're doing a special set of things with the API, it takes a separate API key.

It's a little wonky, but it's not a requirement that you add this in.

I normally just add the first two in.

And if I need to do operations with the origins CA API key, then I do that.

But this takes both our older global API tokens, as well as our scope tokens.

I think that's something for me to file up on.

We're getting some good things for me to file up on here.

I'll file an issue. Does this work with the new API tokens? And then additionally, I'm going to submit one other issue to the project to track to -do list items.

And that's move over to Go modules. You saw me using depth ensure earlier, but that's kind of an older way of doing things.

Go modules is the new way.

All right.

Now what? Well, we want to add the whole point of this massive project is to be able to support calling all of Cloudflare's API endpoints easily.

And so at the time, I actually had support for every API endpoint that was supported in the Cloudflare Go library.

So I think we need to update Cloudflare Go and try to get, try to see if this still works.

I think we're going to hit a lot of problems and we'll probably spend a lot of depth.

So debugging those problems, updating Cloudflare Go.

Because it's a dependency right now, I'll show you. Cloudflare Go is stuck on a revision in the past that's pinned here as a, it's stuck three years ago, the version of Cloudflare Go that this project is using.

And so when we update it, anything that's changed, any breaking changes might take a little work to resolve.

Because I know that people have been working really hard on Cloudflare Go.

So over the last three years, it's changed a lot. So I think that's going to be what most of the episode is.

And then we'll probably, you'll probably learn something throughout that process.

So let's go to Cloudflare Go.

This is the project that everything is built on it.

And you'll see tons of files have been updated as recently as two days ago, two months ago, 17 days ago.

Lots of Go files.

It looks like it's all, excuse me, one giant package, which is a way to do it.

There are definite pros and cons, lots of files, lots of sprawl, but kind of works.

And I don't mind that. So there's also a CLI in this called Flare Control, but it's kind of basic.

It doesn't have support for everything, just a few main things.

And also I'm not a big fan of the ergonomics here.

They're a little bit different than Cloudflare Go. Okay. So let me update here.

How do you do that with that? And sure.

Let's try add Cloudflare Go.

It's already there, but let's see what happens if I add it explicitly.

Nothing to add. It's already there. Okay. Well, let's try to update it and give it this argument.

I don't want to update everything. I just want to update this one.

The moment of truth, let's see how broken this is.

I'm so anxious to see, but it's clear that there's a lot of Git history that's being downloaded here.

That's what's happening in the background. And then they're going to probably try to run a Go build.

Oh no, my power is almost out. What a nightmare. I will be right back.

My power is going to go out any moment if I don't do this. So talk about programming under duress.

One moment. I know exactly where my charger is.

If I had lost this, that would have been a big, big, big problem, but I knew exactly where it was.

This is real-life programming, folks. This is real-life programming.

This is what programmers go through every single day. Okay.

We are plugged in. We've got power, but it says battery source. Battery is not charging.

That's not good. I think it's charging. Yeah. Okay. Okay. And in that time, Git finished.

Glad we averted one crisis for the day. And let's not have any others.

So next up, let's actually build the project. So with make, you basically just have this make file.

You basically run make all, and then it should work. So let's give it a go.

Okay.

I don't have go imports. That's a go problem. Let me get going. All right.

How do you go get go imports?

Go imports. Go get. I just want to run a command, folks.

Okay. Here. And do I have it?

Nope. I need to move go imports to my path. So CP go bin go imports.

And this needs to go to user local go bin. Nice.

All right. We have go imports. Let's try to make all again. Might get mad about go funct if I don't have that.

It's doing something.

Oh, it's really working.

And my battery is going up. It's gone from 7% to 8%. So we are in good shape.

Go build.

It is building. Wow. Go is really struggling today. I wonder what the deal is.

Here we go. So this is going to be something that we have to do at some point.

Because this is an error from the keychain library. When you put in your API token, I store it in the keyring.

And I do that using the 99designs keyring package that's written in Go that interacts with the keyring.

And it's kind of old. And I think the API has changed.

So I'll eventually need to update that, too. I'll make a note of it here.

New issue update keyring library. The hard part of updating that will be backwards compatibility.

Anybody who has used this for a long time and then has a new keyring library, it could break their existing setup.

So that wouldn't be a deal. But it looks like CF did build, though.

CF zoneless zones. Always allow.

All right. It's working. It's working. So we have, if you check here, we have done a big time update of, this is huge, of Cloudflare Go.

There's a lot of new functionality here and nothing broke.

I am super excited by that. We're in really, really good position to actually make a substantive change today.

So let's find an API to add access. Let's find it to add support for in the CLI.

There's so much to do. What's a simple one?

Something new from the last three years. And then gateway locations. That seems pretty good.

Just something simple. So I would like to be able to just list and get it working.

So gateway locations. Dang.

Okay. So I'm going to pull up the docs and then go to gateway locations and try to figure out how to list them.

Okay.

There's a lot of things here. That's not going to work. Teams gateway action is a type.

Okay. A lot of stuff here. A lot of stuff. Okay.

This is an ideal function for it. So we're going to dig around, try to find in the API, like list gateway locations and list, list, list, list, lots of listing.

Oh, actually log push would be a good one to add too, because I don't have log push here and that should be fairly simple.

So log push jobs, let's list it. Jobs.

Okay. So what we need to do are push jobs. We basically just need to call log push jobs, which takes only his own ID and it should list it.

This should be really easy to add. Okay. So the way to add to this is you have to go into this definitions directory.

Uh, let me do a good checkup. Uh, what's this branch going to be called?

Update CF go. Okay. So in definitions.toml is where the, all the magic happens in this program.

Okay. So we want to add log push jobs, a command for log push jobs.

What's the best way to do that? I think it's going to be, um, I think log push should be by zone.

I think it should be under zone.

So I'm going to do list log push. List log push in alphabetical order.

Um, one word list log push. Very, very verbose, but what, what's the worst that can happen?

And then, uh, we basically just need to do, uh, get zone settings, basically need to copy and paste this.

So we just need to copy and paste the command. So we need to add another command with the name that matches the above name, log push jobs associated.

And then we need to give it the function it's going to call, which is log push jobs.

And you do that as a string, which is pretty weird.

Uh, and the reason you do this with a string is because, um, there's all of this is just with string templating.

And I'll show you that in a second, we can walk through that.

Uh, but we pass it as own ID.

You list all the commands that are needed, which is just zone ID. And then, um, then you just have to write a little stub function.

So required equals true. You want to fetch the log push jobs, the zone ID you wish to fetch the log push jobs for.

And then, um, what is the command I need to run?

Some of this I'm refreshing myself. Um, I think in CMD API.

Sure.

I'm going to copy one of these. Okay.

And then all I have to do is call this log push jobs function. I think this will work.

Actually, I think this will, which is kind of funny because that wasn't much work at all.

Let's give it a try. Make will run depends clean CF.

CF will run generate. Generate will run auto generate, which will reparse all of that toml that I just showed you and rebuild.

Okay.

Let's see what happens. Let's go for it. There we go. Okay. Something broke. It has no log push jobs.

I got the casing wrong or right? Um, I got something wrong here.

It's like this wasn't updated. Let me double check. All right.

Um, looks like I have the log push jobs stuff here. So why wouldn't this work?

Did I just mess up the casing? So there's a vendor directory here.

Okay.

This one obviously does not have the library.

So how do we remedy that?

Definitely. Sure. Get a status on this dependency.

It's been so long since I've used that. I'm not really sure.

I remember what I'm doing here. Um, it says the hash is tied to this 9837, which is not what we have here.

So depth update single dependency.

Let's see how to do it. I think we did it right though.

Okay. So use depth ensure to update depth, ensure, add depth, ensure, add depth and ensure update.

This should work for what we ran earlier. Sure. Update. I didn't check in my vendor directory here.

So depth is the thing managing this vendor directory.

Um, so it should, this shouldn't be fixing the problem. I'm a little confused.

Let's try to do a, um, let's try to do a go build now. And then let's see if we, when we go to vendor, Oh, yikes.

Log push jobs. Yeah.

Yeah. I have the casing, right? Well, that is not good folks. That is not good.

Um, I'm not really sure why. Oh, I know. I think I know exactly why. Um, what I think is going on here is, um, CF or Cloudflare go.

I think depth will go to latest releases, which, Oh, it was 10 days ago.

Huh? Nope.

All right.

Well, um, this is what programming is all about running into problems and not being able to fix it for, uh, until you debug it.

Uh, let's go over what changed though with the last little bit.

And, um, and I hope you feel like you can contribute once I sort out this Cloudflare go issue.

I think I really need to move to go modules in the near term if other people are going to contribute.

So if you're interested in contributing, um, please reach out and we can talk about it, but really there's, it's really simple how to contribute.

This is all the templated stuff that, uh, got loaded into, um, that got loaded, uh, when I changed the toml.

So this is auto-generated code.

All you have to do is add, uh, to your toml, to the toml, your command definition, and then, uh, and also add the, uh, the name of the command that you want to do.

So this case it's CF zone list log push jobs, and then add a little stub function in CMD API that just calls that, uh, takes in the arguments that you need and calls the function.

And that's it. The rest is handled for you.

Once I sort out this problem, I'll be sure to, uh, sort it out and update you.

I'll, I'll have an agenda item to update you next week, but otherwise I appreciate you.

I hope you contribute to the project. I hope you read our birthday weeks, birthday week blog posts, and I will see you next week.

Cheers.

Thank you.