BGP hijack detection, exploited vulnerabilities, and how we build our products
Welcome to our weekly review of stories from our blog and other sources, covering a range of topics from product announcements, tools and features to disruptions on the Internet. João Tomé is joined by our CTO, John Graham-Cumming.
In this week's program, most of our Portugal team is responsible for welcoming everyone. We also cover some of the blog posts from the last two weeks. This includes Cloudflare Radar's new BGP origin hijack detection system, the most exploited vulnerabilities of 2022, and our Project Cybersafe Schools, which offers free security tools to small K-12 school districts in the United States.
We will also focus on a more general topic: the process of building things, from new features to the decision-making process of working on shipping new categories of products to the world. How has Cloudflare approached this, from ideas to demos to iteration? And how has the process evolved over the years? Workers, our developer platform used by many thousands of developers, is one example of this.
You can check the mentioned blog posts:
- Cloudflare Radar's new BGP origin hijack detection system
- Hardening Workers KV
- Unmasking the top exploited vulnerabilities of 2022
- Project Cybersafe Schools: Bringing security tools for free to small K-12 school districts in the US
- Introducing per hostname TLS settings — security fit to your needs
- Introducing scheduled deletion for Cloudflare Stream
- Debug Queues from the dash: send, list, and ack messages
Three, two, one. Hello, we're from Lisbon, Portugal, and welcome to This Week in Net. Say bye.
Hello, everyone, and welcome to This Week in Net.
We're live from Cloudflare's Lisbon office in Portugal.
It's the August 11, 2023 edition, and we're going to share a few highlights from our blog, but also go into how we build stuff in a way, the Cloudflare way.
I'm João Tomé, based in Lisbon, and with me I have, as usual, our CTO, John Graham-Cumming.
Hello, John, how are you? I'm fine, thank you. How are you doing?
I'm good, too. We're not far away. We're both in the Lisbon office today. Yeah, where are you?
Can I see you if I look? It's possible, because I'm there. True.
We have a bunch of visits today, actually, from the US in the Lisbon office.
Yeah. Dane, I just witnessed a fireside chat with Dane Knecht, our senior vice president.
He's of Emerging Technology and Incubation, so one of the engineering leaders in Cloudflare, yeah.
Exactly. It's busy for August, the Lisbon office today, and, of course, Matthew is here.
A lot of people are here, so it's good to see this proximity in terms of it's an office in Europe, but a lot of people from the US are here.
Actually, I don't know where you are, but I'm in a glass wall office where half the company is lining up over there getting lunch.
We're recording this, and I can smell the food coming into the room.
I can, because I'm further away.
Well, we were out for a few two weeks. A few blog posts to mention, so let me just share my screen, and then we can...
Yeah, let's do it. Let's do it.
This is the view of our more recent blog post, in a sense. Since we went away, in a sense, there's this Cloudflare Radar new BGP origin hijack detection system blog post.
Yeah, that's pretty cool, actually. Cloudflare Radar has just been growing and growing and growing with more and more stuff, more and more data about what's happening on the Internet.
Recently, we've had a bunch of stuff around BGP, but we've also been starting to do detection of hijacks, so when someone's BGP, where their ASN is misdirected.
We've done this internally for a while and now added it as a feature, so you can go in and look at the hijacks that are happening on the Internet, where traffic gets misdirected because of something happening in BGP, often deliberately and sometimes accidentally.
There's a new page on Radar, and there's a new alerting mechanism, so you can actually see how the Internet gets disrupted sometimes.
I think people don't necessarily appreciate that BGP is the thing that makes the Internet, in the sense the Internet is a network of networks, and how those networks find each other and know who's connected to who, and how to get from one place to another is BGP.
It lays out the route map of the Internet, how to get from place to place. Weirdly enough, there's no security built into BGP itself.
If you read the original BGP RFCs, there are no security features, so security features have been layered on, one of which RPKI, which as we mentioned in the blog post here.
But because there's no security, it is possible sometimes for people to deliberately or accidentally claim to be some network that they're not.
There was a very famous example a few years ago in Pakistan, where the government wanted to censor YouTube, and they accidentally said that they were all of Google.
A tremendous amount of traffic suddenly got sent to Pakistan, which is probably not a good idea, not what they intended.
But these things happen. And it was a disruption, right? Oh yeah, it was a disruption, yes.
And so over time, we as a community have worked to make things better, but we have built this system which can detect when we believe an origin is being hijacked, taken over by someone else.
That is to say, someone's network is being misdirected.
And read all about it in this very long blog post, which describes how we built the module, what data sources we're using.
And then you can just go to radar.Cloudflare.com, I think, slash routing, I think.
And you can see all that information on radar.
It is. Routing is the place to go. And if you haven't been on radar recently, look at that menu on the left there.
Traffic data, security data, adoption of protocols, Internet quality, which is really interesting.
We talked about it once before, which is around how good are ISPs in different axes.
Outage center, what things are out. The URL scanner, you can check a URL to see if it's genuine or not.
If you get a weird text message with a URL in it, well, you can pump it into here and we'll look at it for you rather than risking your browser or something.
So radar has really grown enormously. Yeah. So BGP hijack detection is the latest thing.
And for me, it was like a surprise when I got to understand that hijack is really hijack.
The name is correctly put there because it's all about the Internet, but it's something that hijacks the way the Internet was supposed to work in a sense.
So it's quite important. And these alerts apart in notifications model is quite important too.
Yeah. This is how we're going to be able to alert you if a particular network is being hijacked, for example.
And we've certainly seen hijacks be the source of outages in the past. Exactly.
There's a bunch here explaining that. And by the way, who is the typical user of these types of things?
Thousands of people that work in network, right? Yeah.
Probably the people who work in networking in general will be very, very interested to see this stuff as we put it out there.
So it'll be useful. But also, if you're curious, like your ISP doesn't seem to be working, you can go to the hijack page.
You go to your ISP's page and it will actually tell you, are we seeing an outage?
Are we seeing some issue that's related to BGP or some other issue? So definitely worth checking out radar.
There's so much in here. And actually you paused on the API.
I think this is the hidden gem in radar. It's this incredibly rich API where everything that's in the UI and more is available through the API.
Well, in particular, you can combine data in the API.
So you can look at two trends simultaneously and stuff like that.
So definitely worth checking out. True. I use the API all the time because again, it has more than radar front end in general.
So that's a good place to start too.
We also had some workers' perspectives. Yeah, let's do the hardening workers' KV.
Because I think that is really worth it. So we'd had a series of incidents during July with workers' KV.
So workers' KV is a product that gives you key value storage, can be used within workers.
And these impacted customers.
And so we really wanted to look into what the issues were and what was happening.
And one of the problems with KV is we use it ourselves. One of the good things, right?
We eat our own dog food. We drink our own champagne, right?
We use workers' KV internally. So if there's a workers' KV problem, then it affects customers who don't necessarily directly use KV.
And so this was just a blog post.
We have a habit of going through, there's an outage of really talking about the detail of it.
That's part of what Cloudflare does. And here we are.
I mean, we go through in a lot of detail about what caused these particular outages.
And so if you're interested in outage reports and how things go wrong, this is the blog post for you.
And this is if you were affected, you'll be able to find out what happened.
So, you know, obviously detailed timeline. And then, and so on.
And look at how, you know, how we dealt with the problem. It's interesting even to understand how these types of risks can happen, right?
How it happened and, you know, what improvements we're making.
Exactly. More that we had in the blog, some Zero Trust news.
Yeah, I mean, as you say before, there was the one about workers and Upstash, that's a nice integration with Upstash.
Here we have an integration with Datadog for the Zero Trust stuff.
So lots of the infrastructure that's around Cloudflare, because obviously we don't just work in isolation.
Our customers use us with other things.
True. And we also have an enmasking the top exploitive vulnerabilities of 2022.
Right. This in a sense, it's following up on a report from CISA, Cybersecurity Infrastructure Security Agent from the US.
They released a report highlighting the most commonly exploited vulnerabilities.
And in a sense, because we also have a very good perspective in terms of the Internet, we try to do our own analysis in terms of some of those vulnerabilities there.
And we did a popularity ranking in a sense of the top CSVs, the vulnerabilities in this case, for 2022.
Yeah, I mean, it sort of doesn't surprise me that Log4j and in general, right, improper input validation that caused Log4j is still popular because, of course, people are still scanning for it.
Because one of the things about Log4j is how long it persisted, right?
Many attacks are things that happen and instantaneously you've broken in.
But with Log4j, because strings could get passed around by backend systems, often those backend systems were written in Java, you could have something happen potentially days or weeks or even months later.
So it was worth kind of spraying the whole world with bad strings to see if at some point you got in.
So I'm not surprised that was around. And the Atlassian one, again, Confluence is very popular.
And people are trying to get into Confluence, especially because Confluence, because it's a wiki, likely contains a lot of sensitive information in a company.
So breaking into it would be very, very valuable for an organization, as does the third one, Microsoft Exchange.
I mean, if you can get into Microsoft Exchange, you can get into a company's email.
And also we saw recently some actually very direct hacks against email handled by Microsoft, which has been causing problems.
So not a big surprise there. And then obviously we carry on with other things where the big IP one is interesting because that's a piece of infrastructure, right?
If you can get into a piece of infrastructure, you can probably do stuff.
And we saw attacks in the past against physical infrastructure devices and so on.
I mean, you can see these kinds of things, but yeah, Log4j is the winner of 2022.
True. Not for good reasons in a sense, but these vulnerabilities can, like you were saying, can be exploited much later.
The Log4j one can.
Yeah, the Log4j one, absolutely. Yes. So it's always good to be aware and know that the protections are in place in the services you use.
Well, yeah. And also just the hell of going through every system that has Java in it and seeing if it was using Log4j.
It was a really big, big task for companies. And as with so many of these vulnerabilities, it dropped just before Christmas.
So we all got to spend our Christmas trying to remediate it.
I remember that we had a blog post about that, more than one.
I think we had two or three blog posts about what we're doing to protect our customers, what we're doing.
Because one of the things that happened with Log4j was initially our response was protect our customers, right?
We saw the scanning going on and then there was a bit of a cat and mouse thing with the WAF where people were adjusting the patterns that were being used in the WAF.
And then actually very quickly, actually right at the beginning, we were suddenly like, wait a minute, we use Java internally because we use Apache Kafka and various other things.
And so we ourselves had to go and patch and figure out if we were on also.
Makes sense. And again, we're trying mostly to augment a little bit that the CISA's vulnerability report, showing our own perspective there.
Yep. Tell us about CyberSafe.
You know about CyberSafe and I don't really know much about that.
So actually this is a great initiative from our policy team, in a sense.
Our policy team actually was at the White House this week presenting all about this project, in a sense.
This is a project where we're actually giving to K -12 schools in the US, at least with less than 2,000 and 2,500 students, I believe, are Zero Trust cybersecurity services.
And this comes to, in a sense, a preoccupation that the US has and this White House has in terms of making schools more safe, because education is one of those sectors that several times is vulnerable to cybersecurity incidents.
There's not a lot of money sometimes in some schools to spend on cybersecurity.
So we're trying to help there with this project. It's not the first time we have projects like this.
And this was announced this week, this one in specific.
There's also some data here that I actually helped put together. For example, in Q2 2023, we blocked an average of 70 million cyber threats each day, targeting US education sector websites, Internet properties.
And there was a big increase, 47% in DDoS attacks, quarter over quarter, also in the second quarter.
And we also have some examples here of some public schools in the US, the target of attacks.
So this is a trend, in a sense, that is happening and we're trying to help there.
A pretty sad trend. I mean, it's like we've seen hackers go up to hospitals and other bits of civil infrastructure.
And health is also one of those situations where it's pretty much common.
One of the big areas, yeah. True. And there's other projects actually related in a sense, like Project Galilee.
We already spoke about that in previous episodes.
And also Project Safekeeping, that it's actually for Europe, for example, and other countries is doing a little bit of that Q12.
So this one was presented in December and is actually helping those types of pretty much needed infrastructure that are vulnerable with our products in many countries, including Portugal.
So that's our project, Cyber Safe Schools for you.
Yeah, that was great. A great, great initiative for many of these things.
And more recently, we also introduced this feature in a sense. Yeah, it's a little feature, right?
If you're using Advanced Certificate Manager, you're an enterprise customer and it's just more configuration options, right?
In this case for TLS.
So there's one of those blog posts that's in our continuous improvement of the product kind of things.
It's not sort of dramatic, big announcement or something like that, but it's like, here's another feature.
There's actually another one going out today for Cloudflare Stream, which is our video streaming product.
We now added a feature where you can set an automatic deletion date.
And that will go out in about, well, by the time you're watching this, it's definitely be out.
So yeah, because some people want to stream video and have it available for 30 days or something and make it disappear automatically.
So we're adding yet another little cool feature that's coming, so.
Here it is. That one. Nope, that's not that one.
That's a different one. No, it's not. That's another one.
That might go out today too. True. Debug queues from the dash. Yeah, this is a very, very cool feature built by an intern.
It's really, really cool, but I don't know if it's going out today or not yet.
Maybe. Okay. Let's not dwell into that one.
We still have a few minutes. I want to talk about, we're doing sometimes these initiatives, like talking about Cloudflare history or Internet history or computing history.
And in this case, I was aiming to talk about how, in a sense, how we build stuff, how the process from idea to a little bit of science, to technology, to shipping a product works.
How would you summarize that process? Let me start, actually.
We have now something called Demo Hour with you and Dane Connect. And I've been fascinated by that because I learned a lot just by hearing the feedback.
You can see the collaboration going on, someone presenting something, someone that is building something, presenting something, and the idea is flowing.
Was that like that in the beginning? Well, no. The thing is, in the beginning, when the company is so small, you don't need any situation set up.
They don't need to create a demo because people will just show you stuff.
And in fact, our internal company meeting, which is called the BEER meeting, although there's no beer available, used to have product demos in it, like someone would demo stuff because right at the beginning.
And then, of course, over time, that no longer made sense.
The company's getting huge. So you lost a little bit that. So we reinstated this idea of Demo Hour, which is people come and there's just a bunch of people demo stuff.
Big, big, big focus on demoing, not producing presentations. Sometimes people have slides, but we're like, really?
I mean, the less slides I see, the better.
And just demo what they're working on. Yesterday, we saw a bunch of really cool stuff around R2, around how the core technology in R2 works.
And we saw a bunch of stuff around Privacy Pass, which is a really interesting zero-knowledge way of proving something to a website, such as you own an account, or you're human, or something like that, without giving away any information in either direction.
So the website doesn't learn who you are. And if there's an intermediary, the intermediary doesn't learn any connection.
And actually, we're seeing those kind of technologies get used quite a lot.
If you use iCloud Private Relay, which we help with some of the infrastructure for, again, that breaks the link between who you are and what's being done.
And I think that's actually a big future trend for the Internet in general, is to not make everything so trackable and so easy to link up.
I'm curious about, we have a research team, but in the beginning, there was no research team, of course.
Actually, it was the cryptography team, right?
In what way science plays a role in to finish products?
How is that decision made? Oh, we're working on this in terms of the research, of science, of patents, for example.
And then we start using this, thinking of this as a product.
How can we ship this? Well, I think we've always had a bias towards shipping stuff, right?
And you will have noticed that we ship and ship and ship and ship continuously.
And so we want to get stuff out early. So the whole company, I think, really is into shipping.
I think you asked specifically about the research team.
I mean, the research team, what we have in that team is a bunch of people who are, they lean more towards the academic side than the engineering side.
That's not to say there aren't engineers in there, but it's more working with universities, working with standards bodies sometimes.
And they tend to be working on things that are a bit further out.
So the fact that we were rolling out post-quantum cryptography is because the research team was deeply involved in the original algorithms and the testing of those algorithms in coordination with other people in the industry.
So they are a way of keeping us ahead on all technologies, right?
But ultimately, we will want to ship those things in our products, right?
So they're not doing a pure research. So they're not Bell Labs, and they're probably not going to reinvent the transistor and then suddenly realize it's the greatest thing ever.
But I think that they are doing stuff which looks us out five years.
And it's interesting, you were talking earlier about Dane Connects Group, which is emerging technology.
They're probably looking at things out that are a year or two, like slightly different bets, right?
So we have these different setups of different teams that are working at different risk profile and time to ship and all that kind of stuff.
Although there's a focus on shipping for everybody.
About that in terms of shipping, there's a lot of products, like you were saying, that we launched.
We were known for that. How is the shipping things and break things perspective in Cloudflare evolved?
Because shipping, for example, you ship, you see the feedback, we have a lot of free customers, that also helps in terms of giving us that feedback.
But how does the iteration work with so much shipping?
Well, I do think it's one of the most interesting things about Cloudflare, as we've scaled up, which is in the beginning, like when we were very small, we could ship something.
And if we broke all of Cloudflare, well, a thousand customers, it was bad for them, but it was not the impact it is today.
If you break all of Cloudflare today, I mean, we think we're handling about one in five websites and APIs and apps on your phone and companies' back ends.
We just can't break. So the interesting challenge is how do you keep shipping stuff super, super fast and not break your network?
And of course, the way you do that is a whole bunch of infrastructure and how you roll things out.
We have this thing where we can slice our entire global network into little sections, and then we can give different users different versions of the software so we can observe that around the world.
So that really would minimize a breaking change.
We have all sorts of testing stuff being done. There's a lot of new, actually, internal work being done on the testing infrastructure for everything.
So that is actually kind of a fascinating thing, which is like, how do you build a system that you can roll out new changes to and roll back really fast if you need to?
And even if you did break something, the impact is minimized. Because in the beginning, the sort of ship fast and break things is great.
Now, it's like ship fast. If you break something, make sure the impact of that break is tiny, and you can unbreak very fast.
Makes sense. There's this perspective in terms of choosing what to work on.
We're always shipping a lot, but we must choose because ideas, everyone has those.
We must choose what to work on. In terms of that type of strategy, in the past, that decision probably was more easy.
How is the process of, what should we focus on?
What should we build and ship next, now? Well, it depends on what level you're thinking about.
Because if you think about the announcement of the TLS thing we just talked about in advanced certificate stuff, there's a product management team, and then the associated engineering team working very close to them.
They have a planning process, and they're making trade-off decisions about what features and what customers want, what the market looks like.
They have a whole plan, and every three months, they know all this stuff's coming out.
Then we have emerging technology and research, which are doing things like longer bets.
Their planning won't be so detailed in the same way as maybe the product team does, but they'll also be perhaps slightly more making a bet around, okay, we think we should enter this market area, or we think this large feature area should be done.
They'll be working off in that direction.
They're doing less of the talking to all the customers, figuring out the very detailed product management stuff, and a bit more of the slightly pie in the sky.
We famously did something which was an essential failure.
We were very, very involved in Google AMP, and Google AMP is really disappearing at this point.
But that was a thing that emerging technology was able to do, was able to say, we're going to make a bet on this.
We're going to see what happens, where it goes, et cetera, et cetera.
To be honest, as a former journalist, I used a lot of Google AMP as a coordinator of a website, a journalist's website.
A few years ago, that was huge for a journalistic company. It was huge, and we were the only people who had an independent AMP cache.
And so, yeah.
But for example, there's also a twitch there. For example, we created new categories.
We, like Warp, like Zerotrek, those workers, those were created. What is the process there of, hey, let's go into this.
This is a new chapter for us. Let's go into this.
Well, to be honest, there's been a long list of things Cloudflare has wanted to do right from the very beginning.
We have essentially stayed on a plan that was formed 12 years ago.
Obviously, things have come along, and we've reacted to the market, and we've changed a little bit, but we have never gone through a big pivot and changed total direction.
So something like workers, we originally actually used to do custom code for our customers.
And it was a nightmare because we would write it for them, and it was actually part of our main software build.
So every time a customer changed something, we had to rebuild our software. Obviously, that was not viable.
And so workers came out of, there's a real need for this.
And then particularly when the Sandstorm team, so particularly Kenton Barder, who's sort of the spiritual leader of the workers team came on board, there was this idea that we could expand what we were doing around programmability of our platform to make it this really new type of development platform.
So I remember very well actually having lunch with a few people, Matthew, the CEO, Kenton, myself, and I think there was a couple of other people there in a Mexican restaurant in San Francisco where we're like, we should definitely do this workers thing.
Let's just do it.
Let's make the platform programmable. So it depends. Some of those things are like a little bit top down.
Some of those big strategic things, it sort of depends what it is.
And it's quite interesting, especially to see a product like workers, it's used by thousands of developers.
Millions, I think. Maybe millions at this point.
So I don't know the actual number exactly, but it's incredibly, incredibly popular.
Yeah. It's quite amazing to see that process. This was a great sum up, short sum up for how we do stuff.
Thank you, John. Yeah. Nice to see you.
You're out next week. So we will come back when you will come back too.
No, I think I'm back. I think I'm here next week. You are? So we'll do it next week?
Unless you know about vacation I didn't know about. I thought you were out.
No, I'm pretty sure we'll have another This Week in Net next week. So see you then.
Then we'll have it. What I am going to do is go and get lunch because it's right over there.
Good to see you. See you in a bit. Bye-bye. Bye. Say bye.