Tracing DNS standardization: Insights from Ólafur Guðmundsson

Hello everyone and welcome to our We Are Cloudflare show. This is the second segment we do from my conversation with our DNS guru and VP of Engineering Ólafur Guðmundsson. After years of working on DNS and almost 10 years at Cloudflare, Ólafur retired in February 2024. In the first part of this conversation that's available on the show page, we've heard from Ólafur's amazing journey before and during Cloudflare. This is the second part where we go into DNS standardization, the need for politics in such a crucial area, and also Ólafur explains to us his job as one of the seven humans that control keys of the Domain Name System, or DNS. DNS is at the heart of the web, so this is the Internet's version of a telephone directory, a series of registers linking web addresses to a series of numbers called IP addresses. Without these addresses, you would need to know a long sequence of numbers for every site you wanted to visit. For example, to get to the Cloudflare site, you would have to enter 104.16.132.229 instead of Cloudflare .com. So, this is some of the topics we're going to discuss. So, standardizing DNS security took a long time, a really long time, if we're going to switch into the standards discussion. Before you go there, let me ask you, when did you start participating in the discussions on IETF, and when did you start really participating in that part of DNS? I went to my first IETF meeting that I went to was IETF meeting number 7 in 2000, no, in 1987. Cool. Okay, but then I did not start going regularly until 1990. For those who don't know, what are those meetings about? So, basically, these meetings are a gathering of technology people to talk about standards efforts. The IETF, when I went to it the first time, it was basically one track. There was one topic discussed at a time. Now, there are over 100 different specific working groups that meet, and there are like eight or nine that go on at the same time. So, it has grown a lot. When I went there the first time, there were maybe 100 people there. When it was the largest, during the dot -com boom, there were 3,000 people at a meeting in San Jose, because it was very easy to go there from Cisco, Juniper, and all of the other companies there. Attendance has fallen off, and I actually stopped going to IETF meetings in 2000, and my last one was in 2019. In this case, for those who don't know, IETF is Internet Engineering Task Force, right? And in those first meetings you attend to, what was the discussion like in terms of participating, in terms of new standards? How was the process at the time? The process was very informal in the beginning. The first meeting I went to, the main topic there was network management protocols. There were three proposals on the table, all similar, all very different, and each one of the authors of them was promoting their own, and they were saying, mine is better than yours, etc., etc. But at the end of it, one person said, mine is not any better than the other one, so I'm withdrawing. So the choice was then between the other two. One was basically OSI-inspired, and one was on IETF-like, and so the SNMP was picked as the standard protocol. And in the second, when I started going back, I was working on transport layer protocols a little bit, and I got also involved in email for some strange reason, because I was running my own email server, and I hated it. And when I switched to TIS for work, I started working with DNS, which I thought was a solved problem at the time. But it wasn't, right? No, by no means. And that was in what year, when you changed it? That is in 1994. So the web was starting to pick up in 95, 96. Yes, and at the same time, this is also because of the web, domain names are becoming very important. There is this other technology called keywords that people are selling, people are trying to create closed marketplaces, etc. And these days, domain names are free, but you have to write justification of why you need the domain names. And a year later, the network solutions that was the precursor of start selling.com names for a really inexpensive price of $100 per year. At the time, that was a lot of money, right? And once you have money going into domain registrations, that attracts a very different crowd. So the IETF becomes very crowded with people who want to set policies for whatever reasons about domain names. Fortunately, smarter people than me realized that the only way to solve this problem was to create another standards body called the ICANN, which administrates the domain name system. And that took care of getting all of the non-technical people away from the IETF. Try not to bring too much commerce and politics to that area, right? Yes, to try to keep it as technology neutral. And everybody who participated in IETF is supposed to be only speaking for themselves. But we know that it's not necessarily true. Yes. And the IETF has been very successful in many standards. Some of the standards they have created have worked really well. You use them every day, the IP, the TCP protocol. Now the quick protocol, IETF has taken over specifying the SSL or TLS protocol, which has been absolutely instrumental for bringing e-commerce and confidentiality to the web and the Internet. But a lot of people don't know that if the decisions at the time, the 90s, you were saying were different, the Internet could be a very different thing that we know today, right? So those decisions were crucial to keep it mostly free and available as we know it today, although there are services and all that. But in the backbone, it has a certain structure and philosophy, let's call it like that, that was not like a clear cut at the time, right? The decisions were made to make it that way, right? Well, the decisions were made to make it as open as possible. And at that time, computer security was basically a non-existent. There was a lot of people who thought that cryptography equals security. It was not true. There were other people who thought security was all about access. So, yes. So the world we lived in then was very naive to the world we live in today. But because the IETF, like the TCPIP and telnet and others, without any security features, it was possible to deploy them very quickly. Makes sense. And then the new layers of security, the things that were missing came. Yes, came later. And it took many iterations to get it right. And for DNS security, we were given a mandate of how to do it, which turns out held us back a lot and caused lots of problems. So the protocol should have been revised and we should have gone to version two at some point, but at the same time, we didn't know what we would have to do. So one of the biggest problems in DNS is the delegation point. We have the same record type, both in the parent and in the child. The parent is authoritative for the existence of it. The child is authoritative for the contents of it. Except most of the resolvers that are on the world that need to use this interpret it the other way around. They think the parent is authoritative. So when we have a delegated signing system in DNS, that's because it's a delegated protocol, where do you store the information about what the key the child is going to use? And that was done very badly in the first versions of the protocols that I implemented. And it took a long time to realize that was a mistake. And finally, in early 2002, I wrote up a proposal on an airplane ride back from Iceland to the US called delegation signer. We broke the model and created the first record type that could only exist at the parent. So this picture here at the workshop is showing the different proposals that were in play. Without the delegation signer, we would never have been able to deploy DNSSEC, because it is storing essential information that the parent can sign, and we can have a hand over. So the alternate system in the world is the certificate system. And the web people believe in certificates. I think they're wrong. We should be sending out all the authentication like that information through the DNS. But I'm a minority. For those who don't know, DNSSEC is mostly the security part of DNS, right? It's the authentication of the data you retrieve from the system. To bring like data integrity trust, things like that. Yes. And because of the totally distributed setup of DNS, you have to have a public key cryptography signing every piece of data, and you have to be able to validate the keys. But sign it, being authoritative for that entity. So at that time, after the very well-known tech boom at the time, the early 2000s, you played a crucial, very relevant role in what came next and what the DNS and Internet became next, right? Yes. So that must be like a curriculum achievement thing. I was a working group chair for the DNS working group for 16 years. And at the time I stepped down, that had been the second longest chairmanship of any working group. And I, for a while, held the record for most appeals against my decisions as a chair. Oh, that's a lot. Why? Yes. People didn't like change. But you kept there. So you were doing something right, definitely, right? That's... I like to think so. It's for others to judge. But the time will say that to us, for sure. So when I came to Cloudflare, I had been proposing in the ITF to people a number of different ways of doing DNS that did not go over well. Very radical thinking. So I believe that that the idea of sending signed zones through zone transfer all over the world is stupid. So I came to Cloudflare, among other things, to do online signing of all answers. And everybody said, it's impossible. It's going to be too expensive. We have been doing it for eight years plus now, and we don't notice. So you were right on that regard. And it got more than Cloudflare, right? It got to the industry, in a sense. Yeah. Now a number of other parties in the industry are doing the same thing. And for those who don't know, why is that relevant? Why do you say that change is relevant? It reduces the data that has to be copied all around the world. We just have to copy the exact records. We don't have to copy signatures whenever they are modified. Signatures have a short lifetime, and people are doing a month -long signatures. We only do one day here at Cloudflare, because that's all we need. Answer is valid and appointed time. Some answers, some records change every few seconds. Some change every decade. And we wanted to support the first case and make sure that, yes, minimize the potential of what we call replay attacks. That also minimizes data issues of privacy, regarding privacy. Convergence. It's all about rapid convergence. One of the biggest problems we have, or our customers have, in moving between providers that requires changes in DNS delegations is many TLDs have two-day TTLs. And that means they're a part of the world that takes two days for them to discover the new information. That's why Google and us and other public resolvers, we cap how long of a TTL we believe. So we are operating on a four to eight-hour TTL maximum. Anything longer than that, we ignore. And again, that helps to the mission, Cloudflare's mission, helping make a better Internet, in a sense. Yes. We are not in the Internet of the 1980s, where there are a few parties, something changes once a week. We are at the Internet today. Everything is real-time. People want things to be correct. It's just like, you don't want to see yesterday's stock price before you decide whether you're going to trade the stock. Absolutely. Absolutely. You want the two days to get an answer. And that makes a world of difference in this day and age, how quick it is, the changes. People say that any problem they have with their service, the cost to them can be measured in millions of euros. Absolutely. Even a slow Internet will make a big difference in companies, in individuals working. So productivity, all that. And imagine that you're trying to bid on something on eBay, and you have a slow Internet, and you miss getting the bid in just before the end. True. And on a work day, if you have a slow Internet, you will definitely see pages loaded, it takes more time, so you'll do less, and you'll be more annoyed and bored. So that's not a good experience. You were mentioning before, specifically about the politics sometimes on Internet standards, that sometimes those politics, those interests come about. And those who are technical, who know what they're speaking about, the specific protocols, sometimes have to be playing that game, to take their way, in a sense. How was your process of working through those politics with these standards, these really important standards, bringing consensus, evaluating ideas, things like that? How was that process over the years? I'm going to give you a slight background. While I was... I like politics. I have been a politician in various forums through my life. I did student politics when I was in school, I was in local politics when I was growing up in Iceland. Yes, so I like politics. I like to call myself a politician at times. And so when I was at the ITF, I was doing nothing but technical politics. And that is, yes, how do you get people to see the light or agree on things? And one thing that I learned very early on is words matter. In that sense, that if people agree on the meaning of the terms they're talking about, it's easy to get to consensus. But you can't assume that everybody understands the words the same way. Let's make sure that's the case, right? Yes, that there's a very common... that the understanding is very precise. Mm-hmm. So what I like to do is basically get people to talk. I early in my ITF chairmanship decided I was going to stop writing proposals myself and let others do it. And that helped me a lot, not being one of the people who is writing and proposing things. And it's only when I couldn't find anybody to volunteer to do it that I had to do it myself. Like that case, exactly. Yes. And then others came aboard and said, yes. So how you get the community where not everybody likes each other to get along? You have to create relationships. It's just like a normal life. You have to get people to talk. You have to build trust. The chair is not an authority. It is a facilitator. In a very rare occasion, our chairs are supposed to make decisions. And most of them are saying, yes, we have enough common... the working group roughly agrees on this. Therefore, it can go forward. Or no, we don't want this. Go away. Yes. Those decisions have to be made, right? Those decisions. And then it is listening to the outliers. Because sometimes they have a better idea of what is the problem than not. And I am very proud of I brought a number of people into the standard processing who had great ideas, but were afraid of going in there because they figured they would be mobbed by the old guard. So it is very important that the community gets refreshed regularly. And new ideas come aboard because of those that are thinking out of the box, that have a different experience, maybe. Yes. You were also a trusted community representative, key officer for DNS root key. I still am. You are still. What is that? For those who don't know, what does that represent? So the root of the DNS is published by the root zone operator, which happens to be VeriSign. But the entity that holds the ownership is ICANN. And the keys that sign the root zone are operated by VeriSign. But the keys that authorize those keys to do that work are held by ICANN. And those keys are kept on a very specialized hardware in a very secure location in Culpeper, Virginia and LA in California. And to operate them, there has to be a number of steps to get it. There's actually seven rings around the actual hardware to get it to operate. And I hold a physical key that opens a, let's call it a safe deposit box, like in a bank. And inside that safe deposit box are smart cards that enable the hardware that contains the keys to operate. If those key cards are not presented, the machines will not start. What does that represent? Do you need to do that to use your key often? How does that work? The machines generate the signature for three months at a time. So we do something every three months, every other time in Virginia and every other time in California. And that authorizes VeriSign to keep operating the root zone in accordance with the instructions they get. And because these keys are perceived to be the most important keys in the world, I could debate whether the key certificate authorities could be just as important, but people see DNS as a way to diverge it. So they are worried that new TLDs will be introduced, taken away or whatever. So there is a lot of fear that these keys get misused. Or there was fear or is fear, I don't know. And so there is a very transparent process about using them. And that three months, it's for those seven keys to be approving VeriSign's use, it's for security reasons, right? Every three months it has to be okay, everything is in place. We will approve this again. Yes. And if there's a problem with VeriSign's keys, there is a problem. And so we all have a short time that these keys are authorized for each time. During COVID, we made longer plans. Or you can adapt. Yeah, the system adapted. Yes. Yes, of course, somebody can break in and into all of these rings, but it will be quite obvious that they have done it. So it's not being the use, we're just making it hard to get hold of them. Makes sense. How was this system put in place? What's the story there? Well, there was a committee formed that I can set up. I was not with it. With when? This is like in 2007 or 8. And it created a proposal for the root key operations. And it got started to implementation started in 2009. And the first key ceremony was in June 2010 in Culpeper, Virginia. And I became a trusted key holder, a community representative in 2014, when one person had to retire because they got a job that was a conflict of interest. Makes sense. You can't be one of those and work for ICANN. Yes. Makes sense. And it's like an important role, in a sense, a responsible role. It is. I won't do it forever. I believe in turnover. I believe in term limits, let's put it that way. Any scarce moments about that mission specifically? Always smooth, always easy in terms of how it worked? No, but not scary. But there's challenges sometimes. Yes, there is challenges. Well, it's very annoying. Somebody shows up there after traveling from Europe and discover that they forgot their key. That's a problem. Hardware is not reading the cards. Oh, okay. Yes. These cards, they decay over time. Oh, they have to be refreshed. Refreshed. Yes. There was a lot of fun at one ceremony that I actually blogged about, where we were destroying old hardware. Oh, just a smashing old hardware party. Well, yes. If you can link in the notes of this blog, it has got lots of attention over the years. I will post that. There's a video embedded in there that shows the actual attempts to destroy it and what happened. That's interesting, Elzer. With all of this experience, of course, where do you see the future of the Internet challenges that are coming? You will have a very specific perspective there. Internet is always under a threat. Because it's so important, right? In a sense. Yes. And the threats come from many different angles. There are the ones we can talk about. There are the ones who want to bring it down or harm somebody who is on it. There is the governmental threats. Governments want to be totally able to dictate what their captive audience sees. There are businesses that want to rule it for their own benefit or to create a closed ecosystem. Something like America Offline version 2. Yes. Keeping the Internet open, keeping the Internet affordable is very important. It is also important to realize it has to be affordable and available to everybody. It doesn't matter if I have the Internet, but nobody in Namibia has it. That's a bad thing. It has to be available everywhere. There has to be sources on the Internet that everybody can trust. Something like Wikipedia. This information is getting very easy to do on the Internet. I'm not going to make any judgments about the hype about AI systems today. They can be good. They can be bad. But to some extent, I classify them as you feed the garbage, you get garbage out. The inputs have to be done right. We're seeing that in play already. Yes. The future is bright and the future could be dark. It depends on what people vote for and what bad actors get away with. Predicting the future is hard. It's more like an iteration. You see how it is going, the usage as you're saying, and then you iterate. You change, you see the adaptations you need with the players at hand. America Online failed because they could not adapt to the openness and the speed. They drove the users away by making the life miserable. Many of these ad-supported sites today, I think, are becoming basically like AOL, unusable. I agree. Journalists had a problem there. If somebody can finally figure out a micropayment system for viewing sites, that would be a massively great thing for the Internet. Absolutely. Make it easy to use will do a world of difference. There's, when I was learning journalism, there was this saying that was recurrent from Marshall McLuhan, which was, the medium is the message. And sometimes if you make the Internet really quick, easy to access without entropy, you will make people use it more. If you make it difficult, you'll make people try to use different services that are not as difficult. So that plays a big role, right? Yes. Any final thoughts before we go on your experience, your amazing experience over the years, like the biggest lesson learned you want to share possibly with the audience of 2023, December 2023? It's all about the humans. It's all about that everybody has a voice. What I tell my teams, whenever you're working on a problem, you never know where the good idea is going to come from. So it is the process of talking things through, listening to each other that makes the difference. Allows us to build the best possible thing. Today, I am fearful that lots of people are starting to tune out anything that is not within their agreement and becoming more and more narrow-minded. And that is a bad thing. You have to keep an open mind at all times. Yeah, I've been very fortunate. And I really liked the experience that I've had over the years and being able to work with and talk to amazing people and getting people to work towards common missions. It has been a total privilege. It's all about the humans. It's all about that everybody has a voice. And that's what I tell my teams, whenever you're working on a problem, you never know where the good idea is going to come from. So it is the process of talking things through, listening to each other that makes the difference. Allows us to build the best possible thing. Today, I am fearful that lots of people are starting to tune out anything that is not within their agreement and becoming more and more narrow-minded. And that is a bad thing. You have to keep an open mind at all times. Yeah, I've been very fortunate. And I really liked the experience that I've had over the years and being able to work with and talk to amazing people and getting people to work towards common missions. It has been a total privilege. And it was great just to hear your stories about it and see the progress that was done over the years. Yes. Thank you. This was fun. This was fun. Thank you, Oliver. Thanks a lot. Thanks. Bye.