Latest from Product and Engineering

Music All right.

Hi, I'm Jen Taylor. Welcome back to another episode of Latest in Product and Engineering.

How are you doing Usman? I'm good, Jen. How are you? I'm Usman at Cloudflare and with us today is Sergi Isasi, which is always fun talking to Sergi.

Sergi, say hello and tell everyone what you're in charge of. Sure. Hi, everybody.

My name is Sergi. I am one of the directors of product at Cloudflare. I run a mixed group of products, so DNS, load balancing, bot management, our certificate management, our research team, and also our IP address management team.

List them all out.

That's an impressive chunk of the infrastructure, Sergi. I may have forgotten one, to be honest.

I feel bad about the one I forgot if I did. Obvious steam list of things that you just mentioned.

What would you like to talk about today?

I would like to talk about DNS today. I've been working on DNS for three years now at Cloudflare.

I still remember when you asked me to take it on, Jen.

Totally welcome to Cloudflare. Here you go. Yeah, it's my second oldest product for me, and I think it'd be fun to talk about.

We don't spend a lot of time talking about DNS, although we spend a lot of time working on it.

We do, and it's very fundamental.

One of the things we like to do on this program is just quick refreshes.

Let's start from the beginning. Let's start from ground zero, Sergi. What does DNS even stand for?

What is it? Why is this key thing that is so important to all of us?

Sure. It's domain name system. As we like to say, I actually don't like to say this, but it does work as an analogy.

It's the phone book of the Internet.

It takes things that humans are bad at remembering, which is a sequence of numbers, and translates that to something that humans are good at remembering, which are words.

You don't remember the IP addresses that Cloudflare.com sits on, but you do know the words Cloudflare.com.

You put that into your browser, and it goes through a pretty elegant process to recurse down and say, all right, you want to go to...

I don't remember IP addresses, but that pair that connects you to our website.

You don't need to know those IP addresses. That's what DNS is for. That's right.

The only ones I remember are 1.1.1 .1. All right. There you go. I think that I can see the, yeah, I really like this analogy.

There's a lot of good reasons for that, because a phone book, it's not a phone book.

It's way more interesting, and it's got way more...

But the part of the analogy that I think holds is, in the same way that you ask, or you go to a phone book with a name, and are looking for a number, that part of it is similar, in the sense that your computer starts with whatever you typed into the browser, or the URL, the letters, and the name of the website that you're trying to get to, or the host you're trying to get to.

That name is called a domain, and so hence the name, domain name system.

Okay. But hold on a second.

There are lots of aspects of DNS that live within the Cloudflare portfolio, right?

I mean, we've sort of attacked this from a couple of different angles.

Do you want to kind of talk us through that a little bit? Sure. So there's generally considered two halves of DNS.

There's actually a third one, which I'll talk to in a little bit, but there's authoritative and recursive, right?

So recursive is what clients would...

So if you were the... We go back to the you are the user who's trying to dial a phone number, right?

So I want to know where something is, and that's recursive DNS.

That is our resolver that we launched 2018, if my memory serves correctly.

And that was launched, so eight years after the rest of our DNS services, which are on the authoritative side.

And authoritative is the answering part.

So it's the book itself. And we originally launched Cloudflare with an authoritative DNS product that was part of the whole system.

It's how you got traffic to Cloudflare.

And we've enhanced upon that over the years.

We've rewritten it at least a couple of times. And then we built additional versions of it to solve customer problems.

So about halfway through Cloudflare's we launched what we now call DNS Firewall.

And that is an authoritative DNS product that it's served for our infrastructure partners who have DNS problems where they would prefer that someone else presents their DNS at the edge for them, but they can't get their individual end users to change their name server record.

So it's kind of an interim step to getting any cast DNS, make it faster, get it closer to the user, make it more robust, but not require your customers to really have to change anything.

So that's been fairly successful, but for a very specific type of customer.

And then also in 2018, we launched our secondary DNS service, which is a specific enterprise service where if you don't want to use Cloudflare as your edge network for your web traffic, but you do want to use our edge network to answer DNS, we've given you a standard way of transferring records to Cloudflare and allow us to present those for you.

We've built a lot onto that product over the last couple of years.

I'm pretty proud and happy with what the team's done and the interesting features we've built on top of it to make it more Cloudflare like.

But those are our authoritative products. And then we have the resolver for both teams, for enterprise and for consumers, which is 1.1.1.1.

Let's go deep on that secondary for a second. So what does the problem look like when a customer, why does the customer first come to us?

Why do they even type secondary DNS into the browser?

Is that a standard? Is that a well understood way?

What is the problem that the standard was trying to address? And then what's unique about how Cloudflare approached that problem as we made that service available to our customers?

Sure. So secondary DNS, I don't think I would call it a standard.

The standard is how you, it's a zone transfer. So that's the standard.

Secondary DNS kind of encompasses that. And what it means is that you want to manage your records in a system that is not the DNS provider usually.

So if you think about your 15, 20 year old company and you've built all these processes and probably some software that updates your DNS records, and that's what you're used to.

And sometimes changes there can be very scary because DNS, as I mentioned at the beginning, it's how you get traffic to you, right?

So anything that dramatically changes DNS is a little concerning.

But they would like to use either another DNS, authoritative DNS provider for redundancy's sake.

So having multiple providers in case one goes down.

And obviously we have a very strong track record of being up.

So we're usually on a enterprise's shortlist for that. Or you just want to use our edge.

So we're the fastest DNS network on the planet. There is no way of getting your DNS faster to a user than through using our edge.

But if you wanted to do that and keep your internal processes for updating records, that was very difficult until we supported zone transfers.

And so that's kind of what secondary DNS is.

It's how you get records to Cloudflare or another provider. Okay. So here's my question, Sergey.

I stole this question from you. So I'm going to throw it right back at you.

We just rattled off a huge inventory of products. Clearly we've invested hundreds, if not thousands of hours in DNS.

We're the fastest global authoritative DNS in the world.

Why have we invested so much money in DNS, yet we don't really drive a huge amount of revenue from it?

So we don't drive a huge amount of direct revenue from it, right?

So DNS is a small part of our entire portfolio.

But up until we launched Magic Transit, DNS was how you got traffic to Cloudflare, period.

So if DNS was not fast, it was not reliable, then the rest of our stuff doesn't work.

You can't even, the user can't get to us and then can't get to our customer sites.

So I think when we first started doing Cloudflare, reliability was key.

And then it became pretty clear that we could shave a lot of time off from eyeballs connecting by being really fast at DNS.

So we just got laser focused on that.

So many, if you can shave a handful of milliseconds here and there, it really adds up, especially now when we're talking about our kind of global average being in the low teens, it's 10, 20% faster if you shave that one or two milliseconds.

So we're there's not a lot of room left to go any faster, but it's not to say we won't try.

Well, and it's interesting. We were talking before we got on this call about the fact that there's a lot of heterogeneity in the Internet and how it works, but DNS is kind of one of the few constants.

And one of the things that Cloudflare has been really sort of maniacally focused on is making sure that it's consistently fast across the globe.

But one of the places where that has been difficult has been in China.

Can you tell us a little bit about like why that is and kind of how we've approached that problem?

Sure. And it's not just China, I would say.

So we send traffic to Cloudflare both to our name servers and to the rest of our network through something called anycast, which means that for any given IP address on the Cloudflare network, we broadcast it over each one of our locations.

So 200 plus locations around the world.

And we rely on the Internet to get to us. So we say, we're close to you, user.

You should get to the closest and fastest location to you.

And that works. I have to intercept with a quick story. One of my favorite Hacker News comments of all time is after we launched our recursive resolver, 1.1.1, and people started picking measurements, one of the top comments was, this doesn't make sense.

How could it be this close to me and you at the same time? I'm in Sydney, Australia.

You're in London. How can we both have 17 millisecond pin times? It's not possible for light to travel this fast.

And it was just, it just, I had this big smile on my face because I knew what was next was like this huge thread explaining this is anycast, like the same, more than like, yes, 1.1.1.1 is an IP address.

No, it is not one computer that is responding to that IP address. Multiple computers around the world are answering it.

But like, as you're about to get into that, that is not always true.

You can't always rely on any cast working that way. And so then enter other interesting techniques to go.

So sorry, I just had to, I had to tell that story.

I remember, I remember that comment, this can't be right.

Like one of us has got to have got the ruler out wrong that we're both showing such extraordinary fast times from different parts of the world on this.

So I didn't know about that comment.

And I'll digress a little bit. I remember during my interview process at Cloudflare, one of the very first people I talked to, I didn't know what anycast was.

And so it was explained to me like, oh, well, that's better.

I was going into the old dynacast DNS world, which is kind of funny now. And I guess it was 2019, we decided to start on this project.

Well, what can we do when anycast doesn't send you to the right place?

Yes. And that's pretty common. And as Jen mentioned, in China, anycast does not send you to the right place because the network there is fairly fragmented and kind of difficult.

But it's also true in other parts of the world where if for whatever reason, you know, if you're close to if you're in India, sometimes anycast says you should be going to Singapore, and that's slow, and you don't want that.

And so one of the things that the company asked us to do a few years ago was, can we take away the assumption that anycast will route you to the correct place, and now selectively respond with an IP address of if we know you're coming from, say, India, and we have a colo that is perfect for your ISP, send you there directly, don't rely on the Internet to route you.

So we built out the infrastructure to do that.

And we turn on different locations and different routing policies when we see the need for that in that region.

And it's really made a massive difference.

It's kind of a flip the switch, and then the customer says, okay, that's better.

And so that's pretty satisfying. And it's key for us in China as well, how we get traffic to our partner locations in China.

I think it's a pattern you see in engineering at scale all the time, which is a great architecture gets you an incredible bump, and then you start to get to the long tail in the corner cases and other places.

And then you add on smarts that hint that. We see this in traffic management.

We see this in attack mitigation. We see this across the board, wherever we're doing things at scale, like, yep, now the system's got to be smarter.

It's got to be able to override the base heuristics. It's got to have another control plane that's thinking smart about where to move things around, how to manage things, how to handle things.

So in some ways, it's not surprising at all, but it is awesome to see that we sort of, okay, this is the next step of the evolution.

And like you said, in some ways, we've been working on DNS for so long, it became inevitable that as our expertise grew, that we were going to find new applications for this.

I want to go back, Sergey, to something you said at the beginning, which is really well put, which is for a while, this is the only way to get traffic to Cloudflare.

There is no other way to get traffic to Cloudflare.

And when we first started offering Cloudflare to customers who said, I don't want you to be my authoritative, I'm not ready to put the whole phone book on Cloudflare, but gosh, I really want you to be the front door of the traffic.

So we sort of split the difference with things like what we call partial setups.

Can you talk a little bit about that and what is the evolution of that feature set?

Sure. So to your point, there's lots of good reasons why you weren't ready to have Cloudflare be your authoritative.

You're used to managing your records elsewhere. Usually the folks that are focused on say website performance and web presence performance are not the same as the DNS people.

So there can be some political concerns within a company, but we allow our customers to name to Cloudflare.

So internally, we call it a partial setup, but rather than changing your full authoritative name service for your entire domain over, you can say, I would like maybe my Apex or www to go to Cloudflare and get the settings for just that particular part of my domain.

And it's something that we have offered to our enterprise customers and are now putting that down into our paid plans as well.

But it's just another way of getting traffic to us without forcing you to be fully on our authoritative DNS, even though that's what we would prefer.

We want to be more flexible about that.

What's been interesting over the years is as we've added things like secondary and things like our roles-based access control and all of the stuff that have made it less appealing for an enterprise to go full DNS is now they're ready.

And we're getting lots of customers say, okay, great. This domain that I've had on Cloudflare in a partial setup for five or six years, you now have secondary, you have all of the things I want to do.

Can I turn that into a secondary zone?

Which is a great question that we were going, okay, yes, we would like to allow you to do that.

So we built a button that does all of the things in the backend.

So if any of our customers who are in partial setups say, I would like to use Cloudflare as secondary for everything, they press a button, we do a conversion, they do a change, and then everything's up and running happily for them.

It always makes me happy when you talk about complicated things and you're like, and then we just built a button.

Like that was a good button. There's a grand history of that.

DNS is one of the oldest foundational parts of the Internet. And the term zone comes from the thing called the zone file, which is this file that all Linux, actually forget Linux, all Unix sysadmins learned how in the old days, this is like, I'm really dating myself.

And there's a big fat O'Reilly book on how to do it.

And one of the first things that Cloudflare taught a lot of people was that zone editor, the key thing where you was one of the first best web interfaces for making it really easy to add your authoritative records.

And with this cool button on the side that says, and orange cloud or gray cloud, which is let the world see it or not see it, but make it go through Cloudflare.

So I think it's a great tradition, Jen, of us putting the complexity behind a single click.

As part of the other stuff that you've built in secondary, Sergey, you and the team have also, there's a whole bunch of other features like this analytics and self -service and an API for export.

What is all this? Where's this universe of features that are, this constellation of features that are around secondary?

Talk to us about what's motivating some of those.

Why are we building all that stuff? So when we launched it, we knew there were things that we had to do and to make it Cloudflare-like.

So analytics is one of those. We had to hook it up to our DNS analytics and make that available via the API and all the things that our customers are used to, because those are awesome.

They're really, really strong features. And to have a DNS product that didn't hook up into those was, it just didn't make sense.

So those were always on our roadmap. We don't ship things when they're a hundred percent here at Cloudflare.

We say, okay, we got to get it. You want your records in Cloudflare.

Let you do that. And then we'll add the rest of the stuff as we move along.

There are also a bunch of things where a customer said, this is great, but I want to do this thing that we never thought of.

So an example of that is overrides.

So I'll take a step back.

One of the things that in the very, very first DNS RFC is that you can't put a CNAME, which is another host name, at the apex of a zone, which is the bear.

Old, weird rules. I think on Stack Overflow, it always gets more comments and attitude like, why can't I do this again?

And there are actually very good reasons for it.

And the rule is not actually that you can't put it at apex.

The rule is that if you have a CNAME at a label, you should not have any other record.

The rule is even more subtle than it sounds, right? Yeah. And that's a good rule because a resolver can't be expected to understand that you meant a CNAME for this thing, but not anything else.

Yeah. And unfortunately, to prevent users from making that mistake, a lot of DNS providers just say, well, then you can't put a CNAME at apex because we know you're going to want other things there, right?

Like your mail, right? That's going to be important.

Yeah. Or your MX record, I should say. And so we, at the very beginning, I don't know how this decision is before my time, but we said, okay, if you want a CNAME at apex, that's fine.

We'll just, we will respond. We will flatten it for you.

We will respond with the IP address instead and solve that issue for you.

Lots of other providers don't do that. They just prevent you from doing that very thing.

And if you're a store and you want to buy something from, and you want to use a SaaS as a storefront, then they're probably going to give you a CNAME and you want that to be at store.com and not shop.store.com.

And so this has always been kind of a funny thing.

And what our customers wanted to do for our secondary DNS, because we do flatten for the rest of our services, my master, where I manage my records, won't let me put a CNAME, but can you just let me put one in your system?

So I can't do that. So we built the override for that feature.

And then the very interesting question came next is, well, can you, would you let me proxy that record?

Because that proxy is not a thing that lives in DNS either.

That's a very Cloudflare specific thing. So we built that too.

And so now it's actually been very, very successful. We have customers who manage their records again in the system that they are very comfortable with and do not want to change.

But as the records get to Cloudflare, they can then go into our UI or API and press the button, Jen, and just say, for these records, which are expecting web traffic, those should go through Cloudflare and get all of the rest of the Cloudflare features.

So we built that secondary override, and that's been a really fun feature to watch, actually to talk about, talk to customers and now tell them that they can do that.

They say, oh, well, that's really, really cool.

It's just another way for them to use the rest of our services.

Especially since they probably started the conversation as, you know, especially for those who came to us as, hey, we need secondary as a backup, as resilience, we're looking for priority.

And then it's like, wait a minute, why is the backup?

It has more features than the main thing, right? And so like, that's a great story to be able to tell and double click on.

Yeah, I think that might be my favorite DNS feature we've built in the three years that I've been working on it.

Yeah.

Great. You guys have covered a lot of ground. Talk to us about CDN and DNS separation.

CDN is, of course, the business of content distribution networks.

So like, we can cache, we can respond as fast as possible.

But DNS, as we said, it's a separate configuration.

You don't need to have authoritative on us. So the evolution of how those sort of interplay with each other, how do we think of that from product perspective?

So it goes back to, I kind of briefly touched on it, which was usually the web security or the web performance people are very different people in a large organization than the DNS people, right?

And neither side wants other side touching their things.

Don't touch my toys. Hands off my toys. It works out.

And so we've had in the majority of our, especially in our large, let's say, we'll talk about if you're a large bank, right?

You're either a web security and performance customer of Cloudflare or you're a DNS customer.

But over the years, as we've really proven ourselves with these larger organizations, what about this other half of the thing?

I'd like to bring that on to Cloudflare as well. But I know those guys are on there and they can't touch it.

And so we've been working for the last half year or so on allowing for that.

So you can have a DNS only zone that actually has C names to your security and performance zones.

And they both live in Cloudflare and you have a different set of group that manages one versus the other.

And everything just kind of functions happily there. And our customers are, again, comfortable with the setup and the hierarchy that they put together for themselves and they can replicate that in our network.

It's what we strive for, that our tool matches the way our customers want to work.

So that's excellent. In just the last couple of minutes here, Sergey, I know another thing that the DNS team has worked hard on is Domain Connect.

So just in two minutes or less, what is Domain Connect?

Why does Cloudflare care? What kind of new use cases does that unlock for our customers?

So it makes signing up for and implementing a SaaS of some sort foolproof.

So you subscribe to, you want to move to your productivity services and your mail to Office 365.

Your options are to go and sign up for Office 365, copy and paste a few things and put them into Cloudflare.

And maybe you get that right or hopefully you do, but you might've made a mistake here and there.

Maybe you missed a letter or you fat figured something and all of a sudden your email doesn't work.

That's not a good experience.

Frustrating. This is why DNS is so scary because you can really mess up the IT of your organization.

And it's frustrating from Cloudflare's point of view because we have given you the tools to do this, but it still is a user beware kind of aspect here, which is you have to make sure you enter those values incorrectly because we're going to serve those values.

That's right.

And we will serve them very fast, right? So domain connect is a agreed upon standard.

The standard is probably an overstatement. It's agreed upon way of communicating between us and Microsoft or us and Google or any other of these SaaS providers.

And the customer in the middle of their flow from onboarding is they say the SaaS provider says, oh, we see your records are on Cloudflare.

Would you like to sign into Cloudflare and automatically, systematically include the records?

And that takes away the human error element of it.

You still have to authenticate. So it's very secure, but you don't longer have to go and find the right spot, press the right buttons and hopefully transfer the record.

It just goes directly from them to us and we start serving it and you get your mail.

And for us to build that, that meant that we have a first class integration with these kinds of things that are providing it so that we can consume it and then make those settings for you.

Yes. So we use the open connect standard or domain connect center, I should say.

And then we have a direct integration with the SaaS providers. So anyone who supports domain connect and has a SaaS provider, they can contact us and we can onboard them as a service.

And we were convinced to do this by one of those providers who said, every time we do this, our support volume goes down by 10%.

Yes.

We will happily help you with that because we don't want those issues either. Humans, get humans out of the middle of everything.

So just in our last couple of minutes here, I just want to let people know who are watching, if they found this conversation interesting, in particular, if you think that Sergi and I did an interesting job with the conversation, Sergi is actually in the process of hiring another product manager for DNS.

So if you're interested, please, please reach out.

The job rack is up on the Cloudflare recruiting site. But we are always interested in having smart, curious people join our product team.

Yep, absolutely. And the DNS team is a great team to work with.

Oh, awesome. Yeah. Sergi, thank you so much for joining us.

It's always fun to chat with you about all the amazing stuff you and team's working on.

You guys too. Thanks. Thank you. Bye. Thanks everyone.

See you next week.