SECURITY SPOTLIGHT - Demystifying the Internet

Hello, everyone. My name is Arun Singh. I'm Product Marketing at Cloudflare. And I'm very excited to bring this episode of Security Spotlight to you today.

Security Spotlight is a series of curated security thought-provoking conversations.

And I'm very excited today because one, the topic, demystifying the Internet.

What a fascinating topic in itself.

But more importantly, I'm joined with Martin Levy, Distinguished Engineer at Cloudflare.

Martin, welcome to the episode. Welcome to the show.

And thank you for your time with us. Well, you're welcome. This is great. Good afternoon, or actually, whatever time it is that somebody is watching.

I mean, you and I are on nearly the same time zone.

Anyway, yes, this is good. Great. So Martin, before we get into the meat of the conversation today, which is demystifying the Internet, let's demystify the start that you had from being interested in the Internet, being fascinated by it.

What drove you towards the Internet and the infrastructure of it?

I was just at the right place at the right time. I got interested in computing.

I got interested in computing at about the time that people understood that you could maybe connect one computer to another.

And then through a series of very fortunate events, I ended up at a situation where I really was told to sort of take this to the next level.

And through, as I said, just being in the right place at the right time, somebody turned around and said, in 1982, get this computer to talk TCP IP to this computer, because TCP IP is this new protocol that the ARPANET is going to convert to at the end of the year.

And no one knew nothing. And there I was with two, three, four computers and what was at the time very nascent early ethernet.

We're talking half inch thick yellow coax cable underneath the floor of a computer room with vampire taps and ridiculously expensive boards that could push no packets per second.

It was very slow. And the software was just not ready for prime time, but right place, right time.

And I've been, I've been messing with TCP IP and everything about it ever since.

That's great. Exciting times when there's a problem statement and you are scratching your head and you're like, how does this whole thing work?

Figure it out. It's, it's one of the most rewarding experiences.

So you shared a little bit about, you know, what got you interested into the Internet?

Let's, let me ask you the question, what was the Internet built to do?

What it was built to do and what we do with it now are both very different.

And yet there's this very subtle part that's the same. Communicating between two different machines.

If you look at that early, early diagram of the early parts of the ARPANET, it was four computers.

And the goal was literally to get characters from one to another and then to work out how to essentially access computing.

In other words, login to somewhere else. That morphed into multiple different applications.

Those simplest applications in the early days were the ability to file transfer.

Very painful, very manual with a level of security that varied immensely through the ability, as I said, to log in and use the compute power if you had access to another machine.

And then certain very simplistic data formats showed up in order to query to a remote machine.

Some of the earliest ones were just to see who was logged in on another machine or to look at a particular database.

And even back then, you go back to those early TCP IP ARPANET days, and we were shipping around a file of every IP address and every machine.

There was no DNS.

There was no conversion. This was a scale that just would never work today. But this all changed as we know, and as we use today.

But the great part is that the underlying protocol, the IP protocol below the TCP protocol or the UDP protocol is still the core parts of it are still relevant today.

And although we have improved dramatically all of the protocols, the Internet is to the same extent, the same as it was.

End-to-end communication where the inside is just moving packets in a way, a very dumb way.

But more importantly, the end-to-the-end, there's no permission needed to move a packet from one place to another.

The aspect of build for this.

But as I look at my day-to-day transactions, everything's on the Internet.

I'm doing telemedicine on the Internet. I'm doing my banking on the Internet.

And so as the world went more digital, now in the current times, we are, you know, a lot of us are fully digital.

What were some of the gaps that started surfacing up as we went?

The biggest gap became security. We kept working on improving communication speed, communication cost, the ability to communicate with anything that by the mid eighties with inside the research community, universities with inside the enterprises, the large companies that had realized that this networking was important.

It became fundamental that a new box, a new computer, a new workstation must talk TCP IP.

This became absolute. But security became something that for multitudes of protocols just got left.

We'll do it later.

And in some cases, we kind of forgot to do it later. So when IP came about, even in the original spec, there are all sets of bits in the protocol level to handle one form of security issue or another.

In the TCP world, we understood that certain protocols, at least protocol ports meant something different from a security point of view.

When we did routing, which is really the area that I focus on, in the routing world, we just left security till later.

And as we know, in the HTTP world, in the web world, when the web arrived, we started with simple HTTP communication.

The S, the secure part of communication came later.

But the good part is that because that's what most people see now, the web versus the Internet, that has taken massive steps as other parts in the series have talked about to provide security at that level.

But at the low level, we left security until way after we possibly should have done.

Yeah, now that I'm listening to talk about this, there's this image, I forget which document it is.

But it may be about the Internet protocols.

And then at the end, it says security is kind of DVD.

I can find three or four examples of that. It happened at the BGP layer.

That's the layer which I want to talk about today, which is the border gateway protocol, the protocol that locks the whole of the Internet backbone together.

It happened in the web, as we know, with the early HTTP world before we went to HTTPS.

And it happened, it happened at just at these different areas. And you can find documents where there is just a simple note at the at the end of the document that just says, we'll worry about security in version two.

We don't do that, by the way, just to skip forward to today, we don't do that.

It turns out that security becomes baked in right from the start when you think about things.

That is that's the if you look back at the 80s and 90s, and you look at today, that's like some of the most fundamental difference in the way you do protocol design today versus, you know, versus 2030 years ago.

Yeah, I think that's what I was thinking as you're taking us through the journey through 80s and 90s.

And who was using and what was using it.

I was almost thinking, what's the state of the Internet today? How do you see it?

I know we have made leaps and bounds and other things. But what's your perspective?

Well, first of all, remember, my perspective is much lower.

I'm looking at the guts and the connectivity and the backbones and the parts that the end user doesn't see.

So at that level, where we had handfuls, and then hundreds and then 1000s of network operators that kind of knew each other, that could know each other that were very easy to find.

We now have hundreds of 1000s of network operators, with, of course, billions of end users.

And so the security has to be built into protocols and has to be built into best operating practices far more so than you would ever do prior to prior today.

So this massive jump of in 1997, when we had an Internet incident, we knew who to phone, and we phoned and we said, yo, we have to fix this.

Whereas now that's really not as doable anymore.

So the best common practices and the standards now take into account that.

So for Internet routing security, so imagine you're wanting to communicate.

In order to communicate between where you are and maybe where your bank is, independent of all the security that's thrown in to make sure your application is, you're talking to the right bank, you're talking on a secure channel.

Underlying that is this sort of roadworks of the routing that makes the Internet work.

And so the road signs, what we call this border gateway protocol, the BGP protocol, that is the Internet routing table, 800,000 entries and very dynamic and very live, all saying, if you have a bit, and it has this address in it, go this way.

Those road signs now have far more security built into them, a lot more to go.

But that is the area that I focus on security wise, so that you know that those bits are delivered to the bank, that they aren't sent to the wrong place, that they aren't returned back to somebody else that they shouldn't be.

We already dealt with the optimization.

We already made sure all the bits went in the right direction.

That part was very early on, and we made that much faster and much faster.

We made it a lot more reliable, but reliability and security are two different vectors to go work on.

So now we're very focused on the secure aspects of Internet routing.

Right. So you touched upon the genesis of BGP and why we needed that at the infrastructure level, which is great to hear.

And as we were building this out, there were also roadblocks that we had.

There were also some BGP route leaks that happened.

So can you elaborate a little bit more on what is the BGP route leak to start with?

And something that comes to your mind, which was one of the roadblocks that we had as a world with one of these BGP route leaks.

Oh, well, where do you start?

Where do you start? So first off, the routing protocols were built so that independent or what we call autonomous networks can communicate with each other.

You could run a network. I can run a network. We don't have to be the same organization, but we have a set of common mechanisms, protocols to talk.

When I run a network, I would take an IP address that I own or that I'm allocated, and I announce it to this global routing table so that you know where to send bits.

Even the intermediaries between you and I know where to send bits and send them to me.

Now it turns out that first off, how do you know I own those bits, those routes?

And those routes are owned and there's a registry that says that. But that registry back in sort of the first example of a route leak I'm going to give you is a very manual process.

The automation in it is not there. And more importantly, what you'll find out is that it's not as easy for a machine to know whether something is right or not.

And so we had a route leak in the late 2000s. It's the one that we use as an example constantly, where a mistake was done at an operator in wanting to block YouTube.

And in order to do that, they decided to add a route to their network that instead of allowing the route to be used by their consumers, they would route it to null.

In other words, they'd make every packet with that IP address disappear down a black hole and go away.

And that's a black holing technique, except for they made a mistake.

And they announced the fact that they had inserted this route, and they announced it to their upstream, to their network provider, who then announced it to the rest of the Internet.

And for 10, 15 minutes, all of YouTube disappeared from a very large amount of the world.

Because that route simply said, hey, YouTube IPs, come over here.

This is where I am. Which is wrong, because they should have been over at YouTube.

This opened a lot of people's eyes because it was so visible at the time.

But we've had multiple equivalents to this.

And around the same time, the beginnings of the RPKI, the beginnings of building a new security for BGP, were starting up in the standards body.

This is the Internet Engineering Task Force, the IETF. And RPKI was this simple idea.

You can build a database of everybody who owns an IP address.

It's operated, or at least the beginnings of it, are operated by the people that allocate IP addresses.

And that you can sign with a certificate the ownership.

And then only the owner can say, I'm doing something with this IP address at this part of the Internet, or that part of the Internet, or wherever.

And from that data, you can automatically filter and decide whether somebody announcing an address who doesn't own it is, in fact, actually the right person to do that.

In this case, it would have stopped that leak of YouTube. Fast forward through three, four, five, six major route leaks.

The RPKI software and its very extensive requirements for routing registries, so the people that allocate IP addresses, to start processing this and give the users the ability to manipulate and edit these certificates, which are ascertations, machine-readable ascertations of Martin owns a particular IP address and can do something with it.

That was very key.

On that point, as we sort of went through a year, a year and a half, two years ago, we still were getting the occasional route leak.

But there's one that's really important to talk about because it had a material effect.

From a nefarious point of view, somebody decided to announce a route to redirect access to a particular cryptocurrency exchange.

And they did this. They did this in the greater Chicago area.

And for a period of time, the website that was for this cryptocurrency exchange was not getting any bits.

They were being redirected to somebody else.

Through a series of, quite frankly, simple sort of sleight of hands with users, they grabbed a bunch of passwords from people as they were typing them in.

They then in the background stole a bunch of cryptocurrency and monetary effect occurred.

And this woke up a lot of people because it makes a difference when money is involved.

And it turns out that at the same time, the maturity, or at least the initial stages of maturity of RPKI, were moving ahead such that route security could be deployed en masse.

And this is what's happened. If I fast forward you a year, then we have, or year and a bit, we now have a large number of both major backbones and content players and access networks that are implementing the fundamental building blocks of this new route security.

Such that when one player announces a route, they know that a large number of players are filtering to make sure no one else can hijack that route, making for a much more secure Internet.

The technical parts about RPKI are a whole multi-hour long session in itself.

But the highest level explanation is you're using the same X09, X509 signed certificate model that your bank uses for HTTPS for giving us a secure level at the TLS level.

It is used to secure data so that when I'm running software on my backbone and you are announcing a route and have signed that route with RPKI, then you are able to bring the information into your router in such a way that says, allow this person to announce that route.

More importantly, disallow somebody from faking an announcement to that route.

It's one step of many. We still have a long way to go where we want to bring more security in and use this as a mechanism.

Right. That's very fascinating.

But when I look at the repercussions of this and you know, you give the example of YouTube, you give the example of cryptocurrency, of course, where there's money, there's always malicious intent because it's a very tangible asset that people can have.

So when we look at the repercussions of this, in general, how do you think the industry, the organizations and enterprises, how are they coping with it?

There's two answers to that.

Progress and sometimes a bit of skepticism. But let's go deal with the progress part and give a little light to the skepticism.

We'll mention it slightly.

The progress is that we can clearly see the classic up and to the right graph of the movement of signing of more routes, of filtering more routes.

Cloudflare built a website, isbgpsafe.com, in order to track this in one way or another.

And we clearly see more and more networks, large networks, major networks, signing up and doing the work that they need.

We're seeing Internet exchanges, which are a key part of the interconnectivity of the global Internet, of doing this filtering and detection and the like.

We're also seeing that the work within the protocols bodies, the IETF, for example, in this case, is continuing, in other words, to improve the capabilities, to add functionality to this fundamentally building block, this base building block that RPKI provides.

Skepticism. So skepticism is a really interesting problem that occurs at all different stratas of the Internet protocol level.

And we hear it in many different ways. But the simple one is, ah, but this doesn't solve everything.

So why am I doing it? And to that, there is a very simple answer.

It doesn't solve everything. This is solving a particular problem.

It's a big problem. It takes us a lot further down the path. It sets the groundwork for additional work.

But then we'll go and solve the next level.

And we'll iterate and we'll solve the next level. And we have done this time after time, whether it be if we talk about software development or we talk about how the Internet plumbing is built.

We build something, we improve it. We build something, we get faster, we get more secure, we get more reliable.

Internet routing security is exactly the same way.

So for example, although RPKI only solves the origin part of the route, not the path part of the route, right now in the IETF, there are standards proposed in order to improve path security using the identical infrastructure that RPKI provides.

In other words, all of the security and signing and ownership.

So the good news is that that part has been, um, you know, is active work.

It's not like this is something that's dead. But the reality is that we also have quite a few companies, Cloudflare, companies in Europe, companies in Latin America, who have built software bases that improve and expand the ability for RPKI to be deployed.

So all in all, one of the things that we know about the Internet is that the more people that use it, and this is within the protocol space, let's talk about the protocol space, the more people that use something, the more innovation occurs with inside the industry, such that either better software, better subsystems, better procedures, all come out.

So quite frankly, we may be still early on, we may only have a finite amount of the Internet in this secure manner.

Cloudflare has done its part, both at filtering other people, therefore making sure that other people are secure as seen by Cloudflare, and announcing our routes to make sure we are secure as seen by other people.

Both parts are important.

So we've done that part. We've done open source software. It gets iterated.

You go look at us on GitHub, you'll see lots of either issues or pull requests from people.

You'll see the same for the other software in that space. Two out of Holland, as I said, one out of Mexico, and some others that are showing up in this space.

This is all the standard stuff that we see in the Internet community.

We see innovation and growth, and then new ideas on how to improve things.

Security is front and center in this space at the moment. No one's doing anything that doesn't have an appropriate security component.

Yeah, security in general has been top of mind for a while, but now it's open center to everything.

Remember, this is still underlying infrastructure.

This is over and above the fact that if the packets go to the wrong place, and I'm going to my bank, my TLS will fail anyway.

And so we are dealing with stuff at that level, but we're wanting to secure core infrastructure so that DNS, routing, all of the base level stuff, we know that we can build upon that in a secure manner.

So these changes that you mentioned, Martin, one, that it is very critical to the mission of Cloudflare, helping build a better Internet.

And it's great to hear there are so many other organizations that are moving in that direction to help build a better Internet along with us.

You mentioned that we are still in the early stages of deploying some of these changes.

We as a bigger community, not just Cloudflare.

What is going to be the catalyst to leapfrog some of these changes?

Because everything that you're saying is so critical in the infrastructure today.

From your perspective in the past, your perspective in the current, what really is going to move this faster?

Well, the good news is that we are moving.

But it's interesting to measure this. If you do a basic measure, we haven't even turned up secure RPKI signing of routes for not even 20% of the Internet.

But it actually turns out that's not the measure you want. You want to see the major places where the Internet exists, where content is created.

So that's content delivery networks.

It is content social media sites. It is access networks that run whether mobile or fixed.

It is very core backbones, this sort of tier one backbone list.

And also Internet exchanges where there's a large amount of routing that is swapped between backbones and other networks.

And if you can get most of those in a secure manner, then it turns out that an Internet routing leak then has a very limited, if not near zero effect on the rest of the world.

And you can sort of build upon this.

So if you look at where the focus is at the moment, the focus is 100% at getting those large backbones, Internet exchanges, content players, and major access networks to sign their routes and to filter their routes appropriately based upon this.

The best common practices part that I mentioned is really being helped by the network operating groups and by, well, I mean, everything from our blog to many other blogs in this area, where you're teaching people how to run a network better.

And once you do that, you are able to see the stability and the reliability and the security of the whole Internet improve overall at that point.

Well, that's a north star. I'm hoping for more and more people, more and more enterprises trusting the Internet, one with the data, one with the business and making sure that their customers are having a great experience.

One last very short question.

I always come across this as you talk about BGP. Are BGP optimizers a potential solution for this?

I know it's almost like a... Oh yeah, this is a, I'll give a minute on this subject, but that's all I want to give.

The world of BGP is quite complicated and as any real world system becomes, you can describe the protocol in five, 10 minutes, but it can take you decades to understand how to opt, how to run it optimally.

And a lot of people want a short circuit to that.

So, oh, let me buy this box and plug it into my network and it's called a BGP optimizer and it'll pay back itself.

It doesn't. It simply doesn't.

And it doesn't because the mechanisms and the safety mechanisms around BGP optimizers just don't match the complexity of the modern day Internet.

We have a very concrete example of massive route leaks specifically caused by a BGP optimizer.

And it's just not the way to go.

So, we've actually really, we've actually spent some time, go look at our blogs.

I'm sure it's an easy Google search to find it, but we have, yeah, we've sort of dissected heavily what an optimizer does to the Internet routing table and why that's a bad idea.

That's great. And thank you for that positive note.

And Martin, once again, thank you for sharing your insights.

Thank you for the time today. It almost like I was trying in this episode to demystify the Internet.

I feel like I'm just scratching at the surface and there's a lot more to talk.

So, hopefully we'll have you again in one of our episodes and talk more, but thank you for your time today.

You're welcome. Anytime. You know that.

Take care.