🔒 Security Week Fireside Chat: Geoff Huston & Richard Leaning

Morning, Geoff. Nice for you to join us. Nice for you to join us in our cybersecurity week here at Cloudflare.

Firstly, I know you a long time now, Geoff, but just for the people who are watching in this morning or this evening, where you are in Australia, can you just explain who you are and where you're from and what a chief scientist is?

Okay, yeah, thanks. My name is Geoff Huston. I work for a weird mob called the Asia-Pacific Network Information Centre, APNIC, and I'm their chief scientist.

What does APNIC do? Well, interestingly, if you sort of unwrap all the covers of the Internet and look deep down into the bowels of the machinery, you actually find that the guts of the Internet is actually built up of two things, names and numbers.

Every packet that traverses this set of tubes actually has two little sets of numbers in it.

The first number is where the packet's headed to, and the other number is where it's coming from.

Now, you think that's weird, but you go, actually, no.

Some of you, and you and I probably remember the telephone network, remember that?

Every single handset had a number, and oddly enough, if you wanted to make another thing ring, you entered the numbers and the telephone network did all the rest of the magic.

Well, these telephone numbers, when we went to a computer-based network, actually became what we call IP addresses, and so every single thing in the network has a number.

Now, for humans, things get pretty confusing pretty quickly.

I'm like, remembering 10 digits is hard enough. Remembering a number between 1 and 5 billion or whatever is really, really hard, and so we decided to start using the names that people use, and so we started calling these numbers by a name, Cloudflare.com, and so on, and so there's these two pieces of infrastructure, names and addresses.

Now, let's forget about names, but addresses are fascinating.

Everyone needs one, but who gives them out? How do they get allocated fairly?

And of course, while the Internet was being built, how many do you give away today versus how many do you keep for the future?

And you go, oh, but numbers are infinite.

Well, yes and no, because in their wisdom, the protocol designers of IPv4, and let's just remember that protocol dates from the 80s, dude, when all the computers lived in warehouses and there are only 100,000 of them in the entire world, so they invented a 32-bit number space.

The number goes from 1 to 4 .6 billion.

But that's a huge number. How many computers are made every single year if you count the processing chips?

10 billion. And so it was clear, even from the early 90s, that 32 bits wasn't enough, and we were going to run out.

And the idea of how do you allocate numbers to make the Internet grow, but do so fairly, so you don't give all the numbers to, I don't know, some dude in the US Department of Defense or something, actually requires a little bit of balancing of policy.

And so these regional Internet registries, and APNIC's one of them, came up to try and actually do the hard yards about trying to allocate these scarce and valuable numbers in a way that kind of was fair and matched what people needed.

Now, part of that job is the feedback loop that says, how did we go?

How are we doing? And this is my job, doing the data analytics on infrastructure to understand the feedback between these grand policies we made at the time, let's give away addresses on a Monday to anyone who's wearing purple, and the results in terms of network growth, network fairness, parity, efficiency, and so on.

And so that's my job in APNIC, to try and understand that interaction between policy and technology.

That's my job. And having worked for you for a number of years, it's a very important job you do as well.

But you're also, I know you're a humble man, some may not say so, but I do say that you're a very humble man.

And you're more involved in cybersecurity, network security for many, many years, probably too long for you to remember how long you've been involved in the network security side.

But you're also been involved in the IETF, the Internet Engineering Task Force for a number of years as well.

And just out of interest, can you remember the first RFC that you've submitted to the IETF?

And explain what an RFC is, because I'm not sure everyone will know what that is and how the IETF works.

So the IETF was originally a research project funded actually by the US Department of Advanced Research Projects Agency, variously called ARPA and DARPA over the years.

And it was actually a set of folk who received contracts to work on the Internet for the US, and a bunch of fellow traveler researchers.

And they were taking bits and pieces of experience in networking, because we were working on mainframes, we were stringing them together, and trying to make them interoperate.

As we matured, and started to throw off the shackles of US defense spending, and actually became an industry body, we assumed the guise of a more conventional standards body.

Now, a standards body doesn't create technology, or it shouldn't.

And the technologies that standards bodies create are invariably rubbish.

Committees are lousy at this job. Oh, well, let's have A and B and C and D at the same time.

But that doesn't work. Let's have a vote.

Oh, everyone's voted in favor. So the IETF is actually about standardizing technology.

Why is that important? Because let's say a house, and you make PVC piping, and I make PVC piping, and someone would like to screw the two pipes together.

Oops, we've got to have the same thread, the same diameter, the same this, the same that standards.

In the physical world, we're full of standards.

And equally in the technology world, standards about technology actually make all these applications fit together.

Because bits of the software are made by you, bits by me, bits by other people.

How do you know it's all going to work? Well, if you adhere to standard specifications, then you've got a much better chance of making this all work.

So the IETF is actually these days, heavily into standardizing technology.

And it's imprimata, if you will, an RFC describes that specification.

So it's been going on for ages, the first RFC dates back to late 70s, I believe.

And that were very much let's, it was an experimental lab book, let's try this, let's try that.

I wrote a few early and early on, I wrote a charter for some engineering group.

And I mucked around a little bit in quality of service, and so on.

But I got that because it fascinated me. You see, the thing about technology, particularly the Internet technology is everyone is surprisingly open.

This was not technology that some large company developed and held the copyright on and you can't do this.

From the word go, the Internet was open technology.

And the American effort, oddly enough, was actually inviting everyone else to help them.

This wasn't well, it's our technology fences up barbed wire, go away.

Totally the opposite. This is a really fascinating problem. Come in and join.

Come to our meetings, join our mailing list. Let's discuss this, the more the merrier.

And you know, in technology, having such open technology in a conversation is brilliant.

So you know, that's how I joined this. Security, though, is the fascinating part of all of this.

Because here we all are in sort of the semi research cozy academic world.

And, you know, certainly it was at the outset.

And although we always regarded the students as the hostile enemy, they weren't really that hostile.

It really wasn't that rough an environment, it was pretty benign.

And the effort was not to design infrastructure that withstood deliberate attempts to subvert it.

The effort was to make it work. And so a whole bunch of assumptions were made that, you know, 30, 40 years later, are coming back to bite us something severe.

Because if you actually unpack a lot of the tenants of the Internet, most of the assumptions are frighteningly bad, incredibly simplistic.

And literally, you know, if you push the Internet and its application, just a tiny bit, just topples over.

We can't defend ourselves. Well, I remember going to my first IETF meeting in 2013, in Vancouver, which was November, which was just three months after Snowden's revelations in the summer of that year, and I was a government official.

And I walked into the IETF meeting wearing a suit.

And that's, and that was an interesting environment to be wearing a suit in as a government official in 2013, just after Snowden.

That was a sobering moment, because certainly, various researchers had come up into the IETF in the past, and Steve Bellivan was noted, who would stand at a plenary when they still had plenary meetings and say, the Internet is insecure, we must fix this.

And so every RFC that defines a standard specification must include a discussion about security.

Oh, say the geeks, security is not considered in this document.

Okay, let's make it better. Let's make it better. But it was never really serious.

Because to actually include the concept that other folk who you're chatting to a hostile, actually requires a very different attitude in the design of the protocol, than if you assume that everything's friendly fire.

Right? And Snowden was with a first of these sobering moments, when it was realized that a huge amount of this quite accessible technology was completely open.

And more to the point, it was being exploited by a bunch of folk whose motives were well being questioned at the time.

Now, the IETF has always been liberal leftist leaning.

But this was indeed for them, an outlandish invasion, but how dare they?

You see, the Internet is actually really crude. For a long time, you asked the DNS, totally open protocol, anyone who's on the wire can do anything with it, what the IP address is of a service.

And it gave you an answer. You don't know if it's right or not.

It just gives you an answer. Oh, thank you an answer. And then you send off packets to it.

And if it looks like your bank, you just type in your password, because everything I get from the address I go to is obviously the truth, isn't it?

Yeah, right. And immediately, you get quite pernicious attacks in routing that sort of go, you think you're going here, but you don't know, but I'm sending your packets to there.

And when you get to looks like your bank feels like your bank has exactly the same issue.

It's not your bank. It's just there to grab your credentials.

And this was getting to be pretty tiresome. Because in a communications network, you got to trust it.

I'm like, I can't dive down the wire, and actually make sure it's you, you know, exchange credentials, show each other passports, all that kind of stuff that we do in the physical world.

I've got to trust the wires.

But the problem is, you can't. And so these yawning gaps about how do we make this better, have now been with us seriously for the past, you know, eight years, and more so in the background for the past 20.

So yes, routing security is essential, because, you know, like I said, the two pillars of the network names and numbers, the DNS, the name system, we are running so fast and trying so hard to make that better.

And quite frankly, it's widely abused by everyone, including governments, that making it trustable and secure is a gigantic, hugely expensive operation.

But over in the number world, there's almost the same problem. There's less institutional abuse, but there's certainly some of it.

But trying to make sure that when you address a packet with an IP address, you go to the place you're meant to go to, is actually really hard.

And so industry, because it wants people to spend money on the Internet, obviously, the Internet's hugely good, worked around it.

And you know, that little padlock icon in your browser, when you go somewhere, a padlock appears.

Yeah, I was just gonna ask you about that. Because, I mean, would I mean, we're going to like, you know, BGP, the border gateway protocol, which some people have described as the glue of the Internet, you know, the GPS of a mobile, like the post office to make sure your packets go to where it wants to go through hopping from network to network to network.

And I assume that, you know, most people assume that that is safe, it's secure, nothing could go wrong.

Obviously, that's a naive way of looking at it.

And you know, when I look at my browser, and I see a padlock, I think, oh, great, it's all secure.

But is that the case, Jeff? Is it secure, as everyone believes?

Or is that enough? It's much harder to fake the padlock.

And quite easy to fake routing. Right. And that's why the padlock got invented.

The padlock is a cute bit of crypto that actually says when I'm going to www.jeff.com, or everyone that I go to an exchange comes up using digital signatures, that only the domain owner of www.jeff.com can produce one half of the key to unlock the puzzle.

And so even if I redirect the packets to www.nastyperson .com, who creates a website that looks at www.jeff.com, he can't create that padlock, because he hasn't got that half of the key.

Now, of course, I've got to keep my private key really private.

Because if I publish that, I'm lost, I'm gone.

So routing attacks can still take place. But I can't pass off one side as another, unless I also manage to subvert the certificate system.

So that's good. But routing attacks even so are annoying, it would be nice to stop it.

So yes, it's harder to actually make the routing system fool users, thank God.

But at the same time, we'd prefer it if folk didn't muck around with routing.

But that's really hard, really hard.

Because, you know, there's no map of the Internet, that is the external reference, no one's actually managed to get up in one of NASA's spacecrafts, or Elon Musk's these days, and, you know, look down on the wires and draw them out, we don't do it like that.

We actually build self mapping networks. And it's a cute trick, it actually dates back to the research work in Paul Barron in the early 60s.

And even earlier, Bellman and Ford in the 50s, of networks that discover their neighbors.

Now, this is cool, because if you discover your neighbor, and your neighbors discover your their neighbors, and so on and so forth, at some point, you've actually built a map, almost by rumor, because everyone tells their neighbors their best path to reach everyone else.

And, you know, any student of computer science would understand, wait, that's recursion, it works.

And so, as long as everyone tells the truth, the routing system actually builds and maintains a topology.

And it's great. The algorithm is now, wow, 50 odd years old, 60 years old, sorry.

And, you know, it just works. It's very simple. But the assumption is that every single participating router, every single entity, and these days, there are about 80,000 of them, are all honest.

Every last one of them, even the one I operate.

And, you know, I don't know about the other 79,999. But I wouldn't trust me.

Because if I lie, no one can tell if I'm lying. There's no ground truth out there to judge whether what I'm saying to my neighbors is the truth or not.

And so the original assumption of the Internet, we're all the same boat together, we're all nice to each other, lying and routing never happened.

That doesn't hold in today's world, quite the opposite.

Everything is deeply suspicious.

And correctly, so there are folk out there trying to fool you. And so how do you make sure in a rumor network, that the rumors are true, and that false rumors are instantly identified as false.

And, you know, in the abstract, that sounds impossible.

It's like trying to clean up Facebook, it'll never happen, you know, because you just can't do it like that.

And so the last, I don't know, eight years, 10 years of very intensive work has actually been trying to improve that situation.

By making sure that the folk who forward on these routing rumors, only forward on what they heard, they don't make stuff up on the fly.

They don't generate fake routing information.

And that's kind of where we're in the middle of right now trying to clean up our act with routing, extraordinarily difficult and complex.

Even in geek circles, as soon as you start talking about public private key cryptography, half of the audience is asleep.

And the other half is looking decidedly puzzled.

We talk about the padlock icon in browsers, because it's so damn difficult to actually talk about the underlying technology, and the problems that we have with the web of trust.

You know, this whole issue that the padlock is secure, glosses over a whole bunch of really scary incidents, when that hasn't been the case, and the padlock is misleading you.

And so the same with routing security, we'd like to deploy very similar technology.

And the underlying sort of foundational technology is actually a thing called public private key pairs.

Blame the mathematicians, oddly enough, invented just after World War Two over GCSE in the UK, but they weren't allowed to publish.

And so to Johnny come lately, Americans, Diffie and Hellman published the same work, you know, 10 years later, got all the glory, yay.

Such as history. But the beauty about those algorithms is if I encrypt something using a key known only to me, only the my public key can decode it.

No other key in the world. So if you know my public key, and you find a piece of cryptographic message, you know, two things, and they're both really strong.

Only Jeff said it, and he can't deny it. Because he signed it with his private key.

No one else signed this. And secondly, no one altered it. And so this is really powerful in communications, because it's like an envelope that once I seal it up, no one can touch it.

And the from address is real. You know, no one's lying here.

I sent it, can't deny it. And so we use that in the web, this padlock system, badly.

And we're trying to use it in routing. And so we're trying to actually create digital signatures around these fragments of information that build up the map of the network.

So that if someone tries to go, oh, I just thought of a good thing.

I'll just advertise to my neighbors, this false bit of information, then you're trying to make sure that that can be detected as a lie.

There's no associated signature.

So this is the big dream. It is a bit, but I've got two things, because not every browser has a padlock, do they?

It's only some do, some don't.

And for the, you know, I always, you know, think of my mum, when she goes on the Internet, she has no idea what the padlock means, or doesn't mean it's just, you know, majority of the users have no idea what the padlock is meant to signify.

So regarding that security you're talking about, Jeff, is there, I know, I know the answer to this, but I just wanted to get your view on it.

There's no one entity organization that's responsible for that, is everyone working together?

Is everyone doing the right thing and building these security, as you mentioned there?

And that sounds like a big challenge.

How do you get everyone to do that? Well, I could mention the traumatic protracted saga of v6 to say this is really difficult.

I should mention DNSSEC and DNS and routing security has the same problem.

The issue is in infrastructure, where everyone is doing it, if you want to change it, then everyone has to pay the price of moving with you, everybody.

Now, if we're just 10 people around a table at an old DARPA IETF number one meeting, you know, we can actually stare everyone eyeball to eyeball, are you going to do it?

Yes, I'm going to do it.

Okay, let's do it on Thursday. Yay, we're going to do all this. But in today's Internet, yeah, right.

And trying to persuade everyone to make that investment at about the same time.

For v6, it's taken us 20 years not to get there. You know, for DNSSEC, it's almost the same, it's 10 years.

And we still haven't got there because cost and benefit aren't well aligned.

And some folk go, well, you know, all's cozy where I am, why should I change?

And with routing, we can't change BGP, we can't change the protocol at all, because no one would do it.

And so what we have to do is bolt on these ugly adornments that become optional, because not everyone's going to play.

And this just makes it harder in some ways, because any new technology in the Internet, including security, has to be adoptable piecemeal.

You can do it and see benefits to you. And I can be a laggard and go, well, I'm okay.

I'm not excluded. I'm not shut out. But you need to see benefits of adoption and investment.

And it's trying to create technologies that can do that, that really are the challenge.

And so in routing, in particular, where everybody does routing all of the time, but not everyone is going to buy the crash hot router with the crash hot this, not everyone's going to train their staff, do their configurations, etc, at the same time.

So you've actually got to make technologies that you build incrementally.

So, so far, oddly enough, we've done the easy part.

We've done the really easy bit. It's only taken us 10 years. And that was the one that was meant to be quick.

We've done the bit that says, when you inject information into routing, you better create a digital signature to show that you did it.

Yay. The hard bit is to say, as that piece of information floods through the network, has anyone mucked around with it?

Is anyone lying about how it got through the network?

And that protection called path protection, is the big wall that we've been hammering our collective forward against, because to allow piecemeal adoption, to allow you to do it, but not me, to allow this sort of rumor mill to have almost little bits of more secure operation that detect lies, and other folks who are quite happy to hang out, you know, and go, I don't care.

And all work together is actually proven challenging.

So like I said, we're about halfway there.

But it's been the easy half, the tough half is yet to come. So how would you incentivize the others to adopt this?

I mean, how do we deal with the tough half, as you call it?

I mean, how, what's the future? How do we move this forward?

So well, you know, we've got what we wanted. Now, we got what we wanted in 1980.

When we said in 1980, that we were sick of regulating telephone monopolies, one per country, we were sick of regulated technology.

We were sick of folks saying, we're not going to do this, because we're the telephone company, we've decided not to.

We changed communications into a market. And literally, what drives the Internet is market dynamics.

Now, that may come as bad news to regulators and folk who think they're in charge.

The only thing that's in charge is investment economics.

Now, this is actually the driving force. So there will always be folk who have a differing view of cost and benefit to you always.

And trying to incent enough folk to create a momentum of adoption is actually the tough thing.

And so technologies that only work when everyone does it, are a failure.

And in that I include v6.

Because it only works when we're all doing it. We can't get rid of v4 until everyone does v6.

We're in a 20 year long hell. And the prospect is another 20.

This is insane. The only way we can make this work in routing is to actually create where if you do it, you see benefit that people can't muck with your routes, and play, you know, routing games with your services.

Now, that should be enough for you to go, I'll invest, I might not invest, I might not see the benefit, but that shouldn't affect your benefit.

And so if we can use technologies that actually create partial adoption, where adopters see incremental benefit that aligns with incremental cost, then yay, we're all winning.

And this will actually come forward by market momentum.

The second you go to a regulator saying regulate this, you're actually conceding failure.

You're actually saying this technology does not naturally self adopt, we've got it wrong.

And you know, that's a harsh lesson about the realities of deregulated market that is today's Internet.

You know, like it or not, money drives us all. Yeah, because I suppose regulation is something that no one really wants, because that would be a mindful in itself.

What do you regulate? What you don't regulate? How do you implement it? Well, you just don't want everyone actually to self-adopt.

Don't forget the Internet isn't taxpayer funded.

This is not the old days of telecom. This is actually funded by private investment.

If you start regulating private investment, you actually start scaring the horses.

And companies and countries that are heavily regulate are creating, if you will, adverse conditions to local investment, you actually start creating the stultifying regime that the telcos were so good at all over again.

It didn't work last time, it's not going to work again. The challenge with any security system, and routing is a very good example, is security is normally seen as a cost, a bolt on a taxation point.

You've actually got to turn it round to actually make it an incremental benefit, that the penalties you pay in time, and you do in computation, and you do offset by the benefits you get in not having quite malicious and capable third parties playing with your users, manipulating leaking data and doing all the bad things we've seen them do already.

And so that's the cost and benefit. Jeff, we're running out of time.

As normal, it's been fascinating speaking to you, and I would love to invite you back and we could have an hour slot, two hour slot, because you have many other topics that you can talk about in depth, and we can obviously, you know, unpick BGP a bit more.

So thank you very much for your time, and I will speak to you very shortly, and lovely pleasure.

Next time I'll fly over and do this in person.

Please do, I'll come to you. Thanks indeed.