How Cloudflare Built This

And we're live with this week's episode of how Cloudflare built this. I'm your host, Rustam Lalkaka. I'm a director of product here at Cloudflare. And I'm really excited to be joined today by Dina Kozlov. Dina, you want to introduce yourself quickly? Sure. I'm Dina Kozlov. I'm the product manager for managed DNS at Cloudflare. So that's authoritative DNS, secondary DNS, and DNS firewall. I'm super excited to be here. Well, great to have you. Super excited. Don't be nervous. I'm not gonna, there's no wrong answers to any of the questions I ask, despite what people might tell you. So, you know, I guess, you know, in general, we've started these episodes with just talking about like how folks ended up at Cloudflare and what their sort of journey was before joining the company. I know Cloudflare is your first full-time job out of school. I'm curious to hear more about like why you chose to join Cloudflare, how you ended up being a PM. Like, always curious to hear those sort of like initial steps and stories that led you here. Yeah. So I went to Georgia Tech and there I studied computer science. And so from there, you know, as every CS student, you're looking for your internship. And so I was really lucky in that I was able to get an internship at Microsoft at their Explorer program. And so it's a really cool program because you get to try out, you do like the dev work for four weeks, you do PM for four weeks, and then you do like testing for four weeks. So it's really cool because you get to get that rotation. And so I guess after my first summer there, I really enjoyed the PM rotation. And so I decided to stick with that for my second summer. And so then I just did a PM internship. And so during those internships, I was also on the security team at Microsoft. So that's how I kind of also knew I wanted to gear my career. So what about that first PM rotation was interesting and prompted you to sort of pursue the second internship? Or like, yeah, what was cool about it? Yeah. So it was actually cool because when you do the rotations, you're in a group of three. And so I did the PM one the first four weeks. And so that was when I really got to kind of spec out what is the problem we're trying to solve, talking to different kind of stakeholders and understanding how is the problem being solved today? What is the main pain point? And kind of from there, just kind of scoping out the rest of the project. And so I just really liked that part. I don't know, I really enjoyed meetings, things like that. You enjoy meetings? Yeah. First time I've ever heard someone say that. No, I'm joking. I kind of enjoy meetings too, especially when, yeah. Okay. So you did, it sounds like you did two internships at Microsoft, PM was interesting to you. How did you, and I'm presuming, did you want to go back to Microsoft after, as a sort of full-time employee after that? Or what led you to Cloudflare instead of Microsoft? Yeah, so I was actually completely planning on going back to Microsoft. I enjoyed it there. But then I went to Grace Hopper, which is this incredible women in computing conference. And so there I saw Booth Cloudflare and I was like, oh, my sister works there. So let me go see if I know anyone. I didn't know anyone there, but there I met Dane and Dane was like, let me look at your resume. And he was like, oh, you have some security background. Or I talked him through what I'd done before. And he was like, what's the interview? And so that night I was like, Rita, I actually don't know anything about your company or what you do. Can you tell me about it? And she was like, oh, just read our blog. And so that's where I just started reading the blog. And I learned all about how the Internet works. I just literally read every single blog post that I could find. And I thought it was super fascinating. And then I got really excited and I completely shifted my focus from Microsoft to Cloudflare. And was like, it would be an incredible opportunity to get to work at a smaller company and have kind of more control over what I work on. Yeah. It's super interesting because, you know, I think a lot of people assume that, you know, because your sister also works at Cloudflare, also is a PM at Cloudflare, has also actually been a guest on this TV show, that that was somehow how you ended up here. And in reality, it's like, basically a total coincidence, right? Which is- Yeah, no, it was actually funny because on the way to the first interview, I was like, Rita, I'm actually so excited for this. Like, this company is so cool. And Rita was like, I'm just going to be really honest with you. Like, there's a very low chance you're going to get this. You're totally a hire, new grad PMs. And I was like, are you serious? And I was like, I'm still going to do my best. You never know. And here we are. Here we are. Yeah. So talk to me about that experience of it, you know, coming straight out of school into a, you know, smaller software company. What was that like? And, you know, Microsoft is interesting because they're so big and they have all these programs and like, they basically have like a, what's it called? Chaperone for you when you first show up and they like, make sure you have this awesome experience. How did that compare to starting at Cloudflare, which is, you know, a little less mature in its processes around all that sort of stuff? Yeah, it was completely different. So I felt like my project in Microsoft was very kind of like small scope, like this is where you're going to be doing. And for all the things that I needed help with, there was someone that, there was a person for every single thing in a process for it. And then I came to Cloudflare and actually wasn't on the DNS team at first. I was on the crypto team for my first few months. And so that was a completely different shift. The projects were very kind of broad, especially for a crypto week. There were, I think, five or six products that we were launching in a week and each of them were completely different from the other. And so it was a huge learning curve of each product starting, like figuring out what is this? Like, what is the network time protocol? Why are we building a time service? And then figuring out what is the randomness beacon? Why are we building that? And so on and so on. What is the distributed web? So it was a huge learning curve. And another thing was that at Microsoft, so there's like a person for every single thing, but here it was just kind of like, figure it out, go do it. Start emailing people, get some feedback. It was like, okay. Yeah, I just think I remember one of those early conversations with you where you were digging in on like network time protocols and all that. This is crazy. Like clocks get set by explaining all the nuance to me. And I was like, yeah, this is cool. Yeah, and all of a sudden someone can hack your time and mess up your system. And it's just things that I guess you never think about, but it's also like, where does the time come from in every single machine? Super cool. Yeah, it sounds very like existential, like what is time? Fun makes me think about it. Okay, so it sounds like you started on crypto related projects and some timekeeping related projects and then moved over to DNS. I guess before we go deeper on that, what is DNS? Like for folks who aren't familiar with the acronym or what it does, how would you explain it to them? Yeah, so DNS is the domain name system. So every time you go on your browser and you type in a website, say Cloudflare.com, it's what goes and actually fetches the IP address that you're gonna go to. And so this is because humans are not good at remembering IP addresses. I can't even remember like a phone number, but we can remember things like google.com or Cloudflare.com. And so it's kind of what works in the backend to make sure you get to where you need to go without from like the human perspective without having to remember an IP address. Got it. So is it fair to say that everyone using the Internet uses DNS every day? Yes. Unless you're directly typing in IP addresses. That's what I use. It's actually funny. I think I remember the first IP address that was assigned to like our house's DSL line, but I don't remember any of it. And I remember one, no, one, no, one. I can remember probably five IP addresses. Cool. So DNS sounds incredibly important and sort of critical path for every request that or piece of traffic that sort of interacts with Cloudflare. When did Cloudflare start building DNS related products? Yeah, so I wasn't here for it, but before even Cloudflare launch, they started working on DNS because they knew right away that it was gonna be really important for us to have a really solid kind of DNS foundation before we built out our CDN product. And so when Cloudflare was born, so was Cloudflare's DNS service. Interesting. And does that mean that the product's done? We've been working on it for 10 years and all the features are shipped, time to go home? Is that why we have to do TV episodes instead of working on real stuff? Yeah, I'm actually not sure why I'm employed here. Like DNS is ready to go yet. No, DNS is not done. There's still a lot of work to do. We have to keep making it faster. We are the fastest right now, but we need to keep it up. We need to make it more secure. We need to make it more reliable. And yeah, so there's still a lot of work to do. And our customers, their infrastructures are going and growing and developing. And so we need to make sure that we keep up with their needs and expectations. Got it. When you say our DNS is the fastest and we need to keep it that way, why? Why does that matter? Why does it matter? Yeah, I mean, like if DNS is slow, who cares? Yeah, because every time you go to your google.com or facebook.com, the first thing that happens is the DNS query. And so imagine if that's being slow, like after this, you still have to go through your HTTP request, which is probably gonna be even slower. And so do you want that first step to be as fast as possible? Especially since every single, we're gonna assume, person that goes to access the site has to make this DNS query. We wanna make it like milliseconds fast. Got it. And so not only is everyone using the Internet using DNS every day, they're potentially using it thousands of times a day or millions of times a day or some large number of times a day. And if we can make each of those interactions like a millisecond faster, that's a lot of time. Is that right? Yeah. And just to put it in perspective, Cloudflare sees 6 million DNS queries a second. So that's a lot of queries. And if each one of those is slow, then the rest of the Internet suffers as well. Got it. Yeah, that sounds like you've got a really important job. So we've talked a little bit about what DNS is in the sort of abstract sense. I know there's a couple of different kinds of DNS. Can we talk through what flavors are? So I guess before we get into the flavors of DNS, it makes more sense to kind of go through DNS as like the hierarchy that it is. So every DNS request, so say I'm looking for dina.com, what actually happens is there's a chain of servers that you have to go through to finally fetch the IP address of dina.com. And so there's a server called the resolver. And so that resolver is like the messenger. And so the resolver goes back and forth between the other servers that I'm about to go through and fetches answers for those. And so the first step that the resolver makes is it goes to root, the root server. And so the root server is run by 13 organizations. And so actually at the end of dina.com, there's like a dot there and that symbolizes the root. And so it goes to that server first. And so the root server tells the resolver where's.com or for example, where's.net or org, or if you have like a country top level domain, like .se for Sweden. So it tells you the IP address of that server. And so then the resolver knows, okay, I need to now go ask .com where dina.com is. Then it goes in and talks to the.com server. And then .com says dina.com is at B2 Cloudflare name server. And then the resolver goes to the two Cloudflare name servers or to just one of them and asks, what is the IP address of dina.com? And Cloudflare being an authoritative DNS provider returns this response, either as an A record for IPv4 or quad A record for IPv6. Got it. And does, I know there's two different kinds of IP addresses there's IPv4 and IPv6. Is there any difference between how DNS works on the different IP versions? There's, I guess IPv4 addresses are starting to run out or they are, which is why everyone is moving over to IPv6. IPv6 has a lot wider range of IPs and you can actually get more granular with like if you're doing traffic routing, things like that, it's easier to do with IPv6 than before. But not every client server supports both. We're still migrating. Got it. So in the example, you just walked through with dina.com, there were authoritative and recursive DNS servers involved, right? And so does Cloudflare, is Cloudflare mostly focused on the authoritative stuff or the recursive stuff or like, how does that work? Yeah, so we're actually in all parts of the DNS hierarchy. We provide any cash services for one of the root servers. We also have our own recursive resolver, 1111, quad one. And it is also the fastest DNS resolver. And then we also control the authoritative DNS part of the hierarchy. Got it. And why is it important? So I think I understand why offering authoritative DNS services makes sense. It helps our customers actually get traffic to Cloudflare so that we can proxy it and do all these security and performance things to it. Why the focus on recursive resolution services as well? Yeah, so if you think about it, recursive and authoritative actually helps one another out. So when a recursive DNS query is super fast, then our customers who control like dna.com are happy because their eyeballs are getting to their domain super quickly. And then if you think about it, if our authoritative DNS servers is super fast, then anyone querying the resolver will also get their response much quicker because they're both stored in the same data center. Got it. The services. So the more properties that are using Cloudflare for authoritative DNS, the faster our recursive resolver can return answers and vice versa, right? Like the more people using our recursive resolver, the faster they'll be able to access the actual sites on Cloudflare, is that right? Yep. That's a really good feedback loop that makes the Internet and better for everyone. Exactly. So going back to the, is DNS done yet? Sort of a question. It sounds like we can always make it faster and the faster we make it, the better it is for everyone. You also mentioned things like security. How does, what's the sort of threat model around DNS? Like if someone wanted to do bad things with DNS, what sort of attacks are we worried about there? Yeah, so DNS when it was built in the 80s, security wasn't one of the things they kept in mind. And so here's a scenario that's completely possible is that I am an authoritative DNS provider for dna.com. And so usually I return the IP address, 1234. But now say a hijacker wants to kind of poison this query and send my customers to a site that looks just like mine, where they will then type in all of their credit card information, thinking that they're buying my product, but they're actually in a completely different site. And so then when I'm returning the DNS query for dna .com, it actually intercepts and returns one that says, actually don't go to 1234, go to 6666. And so that way that site visitor ends up on a different domain. And so that's because DNS is public, it's insecure. And so out of this came out DNSSEC. And DNSSEC is a security extension to the DNS protocol. And so what it does is it adds authenticity to the DNS response. And so the way it works is I, the authoritative DNS provider for dna .com, will sign the response, which means that as a resolver, when I'm then asking, what is the IP of dna.com, I get a signature back with it so that I know that this response came from the actual authoritative DNS provider. And it wasn't an attacker that interjected a different IP address. I see. Are there any downsides to using DNSSEC? Like why doesn't everyone use it? So some say that it makes things slower. We've done a lot of things to make it super fast. But also it's not really supported in all forms. And there are a bit of difficulties. We're really trying to drive DNSSEC adoption. And so a few years ago, we added support for CDS and CDS key records. What this is, is when you sign your zone, there's kind of like the stamp, the hash of your record that you need to go and then upload to your registrar, the DS record. And so sometimes customers forget to do it, or it's not, it's kind of like a multi-step process. And so what CDS does is if I'm a registrar and I support CDS, then as an authoritative DNS provider, I can just upload this record on my customer's behalf. And so this is really cool because, for example, Cloudflare registrar customers, Cloudflare registrar supports CDS. And so anytime a customer adds a zone to Cloudflare and then they click enable DNSSEC, that's actually it. Like it is literally one click because then we will upload the record on their behalf. But unfortunately, not all registrars support it. There's also, for example, secondary DNS and DNSSEC. It is kind of an industry -wide problem to implement. And because of this, recently has come out this new model called multi-signer, which allows two DNS providers to kind of sign one another's public key so that you can upload both signatures. And so that way as a resolver, I can always validate the response no matter which name server I query. And so that's something that's on our roadmap to implement. And that's kind of, it's in its final draft stages of being an RFC, but it's kind of the way forward for multiple DNS providers in DNSSEC. I think, yeah, secondary DNS is super interesting and we should spend some time talking in more depth about that. But talking about other sort of innovations we see in DNS and the fact that it's not done yet, you mentioned CDS records. I know there's all sorts of interesting new records being created all the time. I know folks familiar with DNS might know about the A record and the C name record, and then like the quad A record and the MX record. There's some common work, right? But then what are some of the new record types that are trying to solve new problems and might be interesting to folks who are trying to learn more about? Yeah. Yeah, so there's currently a draft for a new record type called HTTPS SBC service record. And so what it does is in a DNS query, gives additional information to the client. So like a browser over how it can then connect to that origin server. And so, for example, right now, if you wanted to connect over HTTP3 or QUIC, you have to make that HTTP connection at first to know that you can upgrade to that protocol. Or for example, if you type in HTTP, but the origin can actually support HTTPS, that's another kind of HTTP trip that you have to make. But DNS is super simple, it's super fast. And so this new record type allows us to provide information to the browser or client over the kinds of protocols that the origin server can connect over. So that instead of making an additional round trip, it's just all in a DNS query. Got it. Yeah. So how does this relate to SRV records? So it's like a service record that is specific for HTTPS. Got it, got it. And so this would presumably require some client support to actually get working as well, right? Are we likely to see clients in the near future support? Cool. From what we've heard, there are some major, some of the main browsers are looking to support it. And so I think that will really help drive the adoption. And if you think about it, that really speeds up performance for everyone, like for that full request. So I'm really excited to see the results of this once we start testing it end to end. Interesting, very interesting. What are some, okay, and then shifting back to secondary DNS and all that. So I know this is something we're seeing more and more demand for. Is secondary DNS the only way to make DNS more reliable or are there other techniques that folks are using as well to harden your infrastructure? Yeah, so there are other techniques. For example, we have Anycast DNS, which makes us a lot more kind of stealthier. We have 200 points of presence and we run our DNS software in all of them, which is a lot more reliable than if you only have two DNS servers and if those go down, that's pretty bad. Another thing is like IP diversity of your name servers, things like that. Got it. So focusing on secondary for a second, what exactly is secondary DNS? I think we alluded to it earlier, but like what's an example deployment of secondary DNS and who might use it? Yeah, so secondary DNS is for when you want to have more than one DNS provider. So there's kind of a few different use cases for this. One is, for example, you want redundancy. So you want to use Cloudflare, but you also want to use another DNS provider. And so this is so that if one provider has an outage, the other one continues to serve your DNS records, which means that your customers, the eyeballs won't see any downtime because they'll always know what IP address to connect to. Because if you think about it, if you only have one DNS server or provider and they're down, then customers can't reach your site. It's pretty horrible. Another use case is there might be some legacy on-premise infrastructure that our customers maintain. And so they have their own DNS infrastructure. And so for them, it's pretty difficult to move away from that. They might have like some dynamic records that get added. And so they don't really want to change that process or they might have certain controls in place and it's hard for them to move away from that. So they might use Cloudflare as their secondary DNS provider and what is called a hidden primary, which is where they transfer their DNS records from their own DNS servers over to Cloudflare, but they actually only put Cloudflare's name servers at the registrar and their own DNS servers are hidden from the rest of the world. But it's almost as if Cloudflare is their authoritative DNS provider, but with the added benefits of not having to move away their infrastructure that's already set up. Got it. What's hard about this? What's hard? Yeah, I mean, that all sounds awesome. Like sign me up, but like, I guess two questions, right? From a customer perspective, what's hard about implementing some of our secondary DNS and then from a provider perspective, what's difficult about this? Yeah, so from a customer perspective, not all providers support secondary DNS and there's different ways to support it. So currently Cloudflare supports us as a secondary DNS provider, but we do not have support right now to export records to other providers. And so just to honestly just take a step back, the way secondary DNS works is that if you will have multiple DNS providers, you don't want to be adding the same record to each provider every time. What you wanna do is you wanna manage all of your DNS records in one place and then you want that copied over to the rest of your DNS fleet. And so the way it works is there's this zone transfer protocol. And so you have your primary server and you add all of your DNS records there. And then anytime you make a change, that primary server notifies the secondary, it is literally called the notify query, and says, hey, there's a new copy of the zone that's ready, you should go pull it. Or additionally, notifies are not always supported and secondary just knows to go check for a new copy of the zone every so often. And so this way, the two providers stay in sync at all times and always have the same copy of the DNS records. And so, yeah, from a customer perspective, you're not always gonna find a DNS provider that supports the kind of setup that you want. Another thing is, for example, DNS tech support, that is something that kind of, again, it's an industry wide problem. And so, especially for example, banks, because of compliance, they have to have multiple DNS providers. But again, because of compliance, they also need to have DNS tech. I think adding DNS tech support would really benefit them. And another thing from like the provider side is, so we've built out all these little hacks with DNS. So for example, you're not allowed to have a CNAME record at the apex. So we implemented what is called CNAME flattening. And so that's where we go out and we get the IP address and we cache it and we return it. And so it's not like we have a CNAME record at the apex, but it's actually like an A record. And so other providers have also implemented their own hacks. And so they have things like alias or A name records. And so that's their way of doing like CNAME at the apex. And so things like that, those don't get transferred out. And so for that, it's kind of finding that compatibility. And so for example, for secondary DNS, since alias or A name records don't get transferred out, it's possible that our customer has this record at the apex, but we don't even know because we never receive it. So something that we've added is what we call override functionality. And so it allows you to create a CNAME record at the apex on Cloudflare when we're your secondary provider, so that whether you're a primary support alias or A name, we can both kind of serve, we can both point to a host name at the apex. There's kind of an interesting tension here, right? Between like providers pursuing these sort of hacks, as you've described, like CNAME flattening and all that. And then, you know, also working within the standards process to ensure interoperability. How do you sort of, how do we balance those things, right? How do we make sure that we're able to move quickly and deliver the stuff that our customers actually need while also making sure that we interoperate well with other implementations out there? Yeah, that's such a hard question. From the RFC side, we thankfully have Oliver, who's up to date and helps us kind of from that end. But it's kind of, you have to, not like you have to create these hacks, but if you think about it as load balancing, that's kind of like an add on to DNS. And it's something that each provider has implemented on their own. And it's something that we're gonna keep, have to, we're gonna have to keep making these improvements that aren't always kind of standard. And so that's another thing of like with two DNS providers, if the query hits us, we might have some logic to load balance it. But you wanna make sure that your other provider also has something there that's able to have the same kind of logic. Yeah, makes sense. Sorry, go ahead. I was gonna say, it's actually cool with secondary DNS. The way it works is the resolver is actually responsible for choosing which name server to send queries to. And most times it sends it to the fastest one, Cloudflare. But it's really cool because the resolver is also smart. And if it sees that one DNS name server is down, it will actually send all the queries to the other one to kind of minimize. Because if you think about it, you don't wanna be in a scenario where you have your DNS provider and then it goes down. And then you have to update your name server at the registrar, that might take a long time to propagate. And the name server TTLs are usually really long and they're usually cached. And so by the time you've added the second DNS provider, your customers have seen. I feel like this is one of the things that was like first really frustrating to me, being a nerd on the Internet, was like trying to set up a website when I was like 12 or whatever. And it's like, could not understand why it took like an hour or whatever for the name servers to change. To this day, some registrars. Yeah, right. It's 12 to 24 hours and I'm like, yeah, I was gonna set up this test to me today, but I guess I'll wait till tomorrow. And you never did. Got it. So it sounds like there's a ton of functionality to be built out around secondary DNS. So getting away from like the specifics of how DNS works, how did the product management around this work? Like how did you figure out what features needed to get built in what order? Like why they mattered? What was the sort of like art behind how you made sort of that discovery process happen? Yeah, so I actually came into the team when authoritative DNS was like in a really good place. It was really solid. And so the focus started shifting to secondary DNS. And so when I joined, we just created a UI for secondary DNS that allows you to view your records, but that was kind of it. It was kind of a standalone product. It was either you're using Cloudflare as a secondary DNS provider, and you can't do anything else, or you're using it as authoritative or partial. And so we really, that's when I started getting on customer calls and I started noticing a lot of these customers wanna use our services, but they also wanna use us as a secondary DNS provider. And so it was like, how do we make this work? And so that's when we started building out this override functionality. And so Cloudflare, all of our services are built on top of our DNS. It is how you enable things. You toggle the orange cloud, and that means that we will now proxy requests to that hosting. And so we started thinking about how do we build this out for secondary DNS? And so one of the main kind of caveats there is we did build out this functionality where Cloudflare can be your secondary DNS provider and also proxy traffic. But a huge caveat there is we want our customers to use that only in a hidden primary configuration, which means that your primary name servers will always be hidden. And the reason for that is if you query your primary DNS provider, they're gonna return your origin IPs. And then if you query Cloudflare, you're gonna return Cloudflare IPs, and that request is gonna go through our edge. And so that was like one of the interesting things of how do we make sure that customers are aware that they need to be in the setup? We literally have a giant box in the UI that says, like, you are in override mode. Keep in mind, this is what will happen. And so- How did you know that adding that box solved the problem though? Like what was the sort of process there to figure out? It sounds like we figured out through sort of painful experience that we needed more warnings and all that. And how do you know that that worked? So it's more of before we built it out, we kind of looked towards the future and we were like, we see ourselves debugging a lot of issues where this is gonna come up where two different name servers are returning different IP addresses. It's gonna be really difficult for our team and like their team to debug. And so we wanted to make sure that customers were really aware of this caveat. One of the things that we were considering doing is if you had Cloudflare and other name servers at the registrar, then we would show this kind of like warning of, hey, by the way, this is what's gonna happen. But one, that would have been kind of hard to implement. And two, we also didn't want our customers to set it up in one way. And then we wanted them before they added like their own name servers to keep in mind that this is what's going to happen. So kind of preventing this problem from happening before it were to happen. But that's actually also where we're talking to customers, I would tell them like, but by the way, keep in mind, you have to do this in a primary configuration. And actually the feedback I always get is like, oh, yeah, of course, like we need that. And I'm like, great. So it sounds like there's a sort of like bias toward doing simple things that make sense versus like fancy things that might solve the problem sometimes, but it might make the problem worse in other ways. Is that sort of like something you explicitly think about as you're going through the design process or that's just sort of intuitive or like, how does that actually work? Yeah, that's a loaded question-ish. But I guess, yeah, it's something that we think about. For example, right now we're building out kind of this conversion UI between different zone types. And so one of the things is we call it partial, that means you're not using DNS. But if you're not familiar with Koffler, that doesn't really make sense. Like what is partial? Why does full mean that I'm using you for authoritative? So we're really thinking about how do we make this clear to the customer? How do we make actions clear to them? Looking at support tickets, I'll do that and see kind of what are the main questions that customers are asking and kind of what is not straightforward to them? Because we really do have like our enterprise customers that might be experts on DNS, but they might not be at all. And so we wanna make for those customers as well, we wanna make it as easy as possible. And so like, for example, our DNS scan is really helpful if you don't know anything about DNS, because then we'll just find here are the DNS records that we found from your other provider. And so I remember the first time I was setting up a zone on Koffler, I had no clue what DNS was and I was just clicking through it and I was like, great, something works. And so it's really cool that we've built out tools for all level customers. Got it, that makes sense. How do you actually get feedback from customers though? Are you just like sending them emails being like, yo, what do you think? Or like, yeah, what does that look like? Yeah, you don't send an email to all 20 million. Just kidding. There's different ways. So for UI updates, we will post a survey in a banner. When we posted our last survey, we got thousands of responses, which was really great. I think it was like 5,000 responses. It's pretty incredible. That's like a lot of data to help us understand like, is this better? Is it worse? What are the main kind of things you're struggling with? Another thing is I would literally get, I have at least like a few calls a week with different customers. And I always love to hear about their current setup. What are their pain points they're experiencing right now? What are kind of, what are they looking to solve and do? What is their infrastructure gonna look like five years from now? And kind of making sure that whatever we provide to them actually solves the problems that they're facing. And then also, yes, go ahead. I was also gonna say like everyone at Cloudflare or like lots of SVs, they use the dashboard, DNS, things like that every single day. They debug things. So getting internal feedback is also a great way, yeah. How do you actually then take that feedback and turn it into like changes, right? Like I'm always curious to hear how that goes in. Like, how do you share that feedback with your customers? With folks like your engineering team and the rest of the company in a way that makes sense? Yeah, so it depends. Our team, we have standup every week. We literally have a part of standup that's called customer updates, where I'll tell them about problems that customers are facing or what they're wanting us to do. There are feature requests that you can file and that's an easy way to look through things. When kind of figuring out what features to tackle, we think a lot about impact. Things like adding DNSSEC to secondary DNS that is huge, that will have a lot of impact. Yeah. Got it. And then what about, like, does the process change for things that don't involve a lot of UI, right? Like showing a UI to a customer and then they say like, oh, I want that button to be green instead of red or whatever, like that stuff seems straightforward. But like, what if it's a very abstract, like how zone transfer should work or whatever? Like, is the process the same for that stuff? I guess the more general question is like, you work on a lot of really technical things at Cloudflare sometimes, like do you think that the PMing for that is very different from the PMing on the more sort of like UI or less nerdy things you work on? Yeah, for sure. I don't know, like a lot of times I literally just have to go through, read RFCs, Google things. What's an RFC? An RFC is kind of like a standards document for how, it's kind of like, this is a standard for how we should do this. You can tell it's a document for nerds because it has a very bizarre definition. RFC stands for request for comment, but somehow that's a standards document, the very passive way. Yo, feedback on this? I want it to be official. Yeah, I also think it's super cool that ITS stands for Internet Engineering Task Force. That is like nerd super squad name. But I forgot, what were we? Oh yeah. I was gonna say like, yeah, if you're on the nerd super squad side of the spectrum versus like, here's a really nice, like easy to explain UI. Yeah. Does being on one side of that equation or the other change the way you approach your PM job? Yeah, it's very different. For a lot of backend changes, I'll work with Vicky, our engineering manager, and kind of work with him to make sure I understand, like, why are we making this change? What is this gonna do? Because a lot of the backend changes, they're not really customer facing, but it's still really good to be in the loop. Certain things, they might then turn into products eventually. So it's also good to understand that. I actually learned a lot about DNS from debugging, or not debugging, but kind of following the threads of, we have the support ticket come in, this customer has a weird configuration, and kind of diving in there and figuring out what are they doing. That has really helped me figure out how DNS works. And yeah, no, but I think it's really cool and fun to kind of dive into the technical side of things. Yeah. What sort of, you've taken all this feedback from customers and you've built out a roadmap and all that. I guess first question is like, how do you approach building out a roadmap as a PM? How do you figure out which features should get built next versus like in six months versus in a year? Is there a plan or is that, is that a feel or both? Yeah, there's a equation in Google Sheets. No, I'm kidding. No, it's like- Well, I think there is an equation, right? Then you have to let them down. I always forget this. No, it's actually like one of the things that I feel like I'm still learning of like, how do you prioritize things? Every quarter we have our quarterly planning. And so we kind of think of, here are like the projects we wanna pursue. Here's why we wanna pursue them. Here's the type of impact it will have. Sometimes we look at the revenue. Sometimes we look at kind of, our mission is to help build a better Internet. So like things like DNS, like adoption, we really wanna help kind of drive that. And so it's also looking at the team's resources. What all can we actually tackle? What dependencies do we have? Do we have dependencies on other teams? Is this gonna have to wait? Yeah, there's like a lot of things that go into it. So getting away from the abstract, what does the roadmap for DNS look like over the next six months? What are you really excited about building? Yeah, so in the next year, I'm super excited. We have a lot of things planned out. As we said, we're kind of focusing a lot on secondary DNS because we're really seeing that customers are moving their infrastructure to kind of a multi-DNS system. So something like I said, DNS tech support for secondary DNS. We also wanna support Cloudflare as a primary provider that is able to transfer records out to other DNS providers. That's gonna be really exciting. DNS in China, that's gonna be super great once we have it and really kind of decrease that latency for our China customers. We recently came out with a secondary DNS analytics and we also added Domain Connect support. Domain Connect support allows you to, if I am a WP Engine, for example, and I need to add DNS records on behalf of my customer, by having that supported, the customer can just literally, it's like a one-click, they're directed to the Cloudflare dashboard and WP Engine adds the records on the customer's behalf. So in the next few months, we're gonna add support for a few other service providers. For Domain Connect, we're also gonna have some UI changes. We recently introduced confirmations to the UI. We wanna add bulk edits so that customers can make multiple changes at once. Yeah, things like that. DNS secondary, DNS error notifications. So if your zone transfers, if your zone transfer fails for some reason, then we can send you an email, tell you what error we're seeing, help you debug it on your own so that your two providers can always stay in sync. Sounds like a lot of stuff coming up. Good luck keeping all that on time. I know you won't need it, but. I mean, and actually on that note, like how do you actually, like how much responsibility do product managers at Cloudflare have for like the actual schedule, right? Like it sounds like we help figure out like what order things should get built in, but like how much influence and control do we have over like this thing's gonna ship in two weeks versus four weeks versus 10 weeks? Yeah, I feel like PMs are kind of like the glue of everything that goes into building a product. And so we're kind of like on top of the ship saying, does this team have this ready? Are kind of all the dependencies ready to go? Have we kind of thought through everything? Do we just wanna ship it? And then kind of like have a small launch where a few customers kind of use it on test domains and figure out what is missing, things like that. Like is engineering ready? Is marketing ready? Do we have a blog post ready? Things like that. Got it. Who writes the blog posts? Sometimes the PM, sometimes the engineer. We actually have two coming out this week on secondary DNS. Did you write them? No, Alex on our team did. I'm actually kind of curious, do you enjoy writing the product launch blog posts or do you do that as a chore? No, it's really cool. I don't know, it's kind of like a reflecting back on the whole journey. And then when you start writing it and you're like, wow, I remember when this was just an idea or like just starting out the PRD for it and now we're launching it. Yeah, I find that kind of complicated or like hard sometimes because like, you know so much about the backstory and then you're writing this blog post and you realize like, no one actually cares. Like the user cares about the feature but they don't care about all the like, you know, weird hoops you had to jump through to get. Exactly, and sometimes the project changes direction multiple times but no one's ever gonna know. Yeah, nor should they really, right? Yeah, exactly. Yeah, it's super interesting. I've coached some folks through writing these posts and like, I think a common problem with the early drafts is like it reads like a history lesson and not like a, here's what's actually special and cool about the feature, right? Like it's easy to get lost in that. I've also been told that like basically all of the product announcement posts on the Cloudflare blog start with today, we're excited to announce X feature that allows you to do Y thing. And the person who was complaining or not complaining, remarking on this was like, this seems like a bug. Like why does every post start exactly the same way? And I was like, I don't know. How else would you start a product? How else do you announce things? I've been thinking about this though now, like how, next time I write one of these posts. Well, I'm gonna look at every one of your blog posts from now and make sure you don't use any time references. Yeah, right. In a galaxy far, far away, secondary DNS has launched and a lot... No, I don't get to write these posts anymore. I just, I have to proofread them. Actually, I remember one of the posts that I read when I was interviewing was the one about cash and it had the Arrested Development cash references. There's always cash in the banana stand. Oh yeah, I was actually pretty proud of that one, yeah. I was like, this is so clever. And then during my interview, you were one of the people I had lunch with and you were like, what blog posts did you read? And I was like, oh, there's this one that I thought was really funny. And you're like, that was mine. And I was like, oh, cool. Yeah, actually, there's a little Easter egg in that post. There's an illustration of a banana stand. And I think Matthew is actually in the banana stand. Yeah, I saw it, yeah. He was at least, he was in one of those, he was in one of the early drafts of that drawing. I don't know if that made it to the final. So, I mean, just switching gears a little bit. You've been at Cloudflare for what? Almost a year now, right? Is that right? Like a year and a half, almost. Wow, time flies when you're having fun. What are some of the things you've learned about software development and being a PM and all that? And yeah, in that year and a half, what are the sort of big takeaways so far? There's a lot. I don't know, shipping a product is hard. Even if it's something minute, there's a lot of work that goes into it. From like when it's an idea to when you write a PRD. What's a PRD? Product requirements document. And so that's when everyone starts commenting and a lot of people have like opposing decisions or opposing opinions about it. And you're like, who do I, what? Yeah, let's dig in on that. So, you and I have both seen these. So, at Cloudflare, we write these product requirements documents and then the PM posts it on our internal wiki. And this was one of the shocking things to me starting here. Like you post a thing, random thing on the wiki and like it's like a moths to a flame kind of thing. Like people just show up and start commenting on it. And then sometimes like two people will disagree about something and then you end up with a comment thread that's like 40 comments long. How do you go about like navigating that? Like how do you figure out what the right thing to do is? Which side to pick or are they both wrong? Like how do you approach that? Yeah, so I think kind of like going back and forth in the comments is really helpful because it kind of like, I feel like helps develop the conversation in a few different directions. What I've found to be most helpful is to then kind of book a meeting and put those people in a room and be like, okay, by the end of this meeting, we will figure out the path forward. And kind of just going through like, what are the pros and cons of doing each thing? What is feasible? What is not? But then you also have to think about like in the long term by choosing the easier route, like are we shooting ourselves in the foot? Is this gonna have to be delayed if we wanna do it the proper way? Do we wanna do that the proper way because customers also have deadlines or like if they have a migration or something? So just kind of trying to, I always come into a meeting with questions that is always kind of helpful to help drive the conversation and then try to come up with a conclusion. And the conclusion might be like, here are the kind of three solutions and here's like the one I think we're gonna go with and kind of going from there. It sounds like having an opinion is part of that, right? Like, yeah, scheduling a meeting, but it's not like you're a completely passive observer. No, yeah, you kind of, and sometimes like your opinion completely shifts and you're like, oh yeah. Yeah, it's funny. Alex, who's an engineer on the DNS team, we always have arguments about secondary DNS and I'm always like, I think it should be this way. And he's like, I think we should do it this way. And actually multiple times at the end of it, we both switch our opinions. And I'm like, actually, I think we should do it your way. He's like, no, actually your way makes more sense now. So how do you break that tie if you're both arguing both sides? So it's, I mean, it's all like depends on it, but it's kind of like, I don't know, you have to think from the customer's perspective and sometimes like I'm definitely the person that does that more than he does. And he's thinking about it from the engineering perspective and that's kind of where you have to converge on like, what can we do that's good enough for the customer, but also like we can literally do like engineering wise. Yeah. Yeah, that's sort of like magic equation to solve is super funny. Let's actually talk about that more. I'm sure you had no exposure to that conversation in school or prior to becoming a product manager. How did you learn how to balance that and converge on the solution that balances like what customers actually want and what's actually possible to build? I think it was like a lot of trial and error. I think honestly my first few meetings were probably really bad and awkward, but you know, you fail fast, learn. So I started coming into like a meeting with questions and being like, you really have to shift the engineers or else they can like diverge forever. And yeah, I guess it's kind of figuring out, okay, like what are the main pieces we need to figure out and kind of also not focusing on the minor details is also important. Yeah. I don't know. So you're just basically figuring out what matters and what's nice to have and then being willing to compromise on the nice to have and not so much on the must have, right? Exactly, exactly. And that's one of the things that I love about Cloudflare is we ship products really quickly. And so it's really easy to kind of like iterate on them and kind of get feedback quickly of, we released this and then three months later, it's gonna like completely improve, but it's like, we don't wait to have every single detail figured out because a lot of them are kind of minor. Yeah, no, that makes sense. What else have you learned that you wouldn't have been able to articulate prior to starting? I guess like all the stakeholders, like for example, it's not just like customers and engineers, there's also like marketing, there's also the sales team. You have to incentivize them to like go and sell and like kind of make sure that they understand what they're selling and make sure that they are not pitching things to the customer that they think the customer needs this thing, but that actually won't solve their problem. Things like analysts, that's completely new. I don't know, having also meetings with executives and kind of seeing kind of them bring their opinion in and yeah, no, I feel like there's PM and then there's like a million different pieces to kind of hold. Yeah, and so how do you go about managing that? Is it truly just trial and error? You like go into these meetings and you ask a lot of questions and listen and see what comes out of them or is there some trick to navigating all these opinions? It's kind of like that for me, like I have like here all the projects we're working on and kind of like, what is the next step? And so that way I can always keep track of like, is there something that's blocking it? Is there something I need to do to push it forward? Do we need this? And kind of making sure that like, I feel like my job is always like pushing it forward and just keeping track of, yeah. So if your job is always pushing things forward, it sounds like a good day at work is when you feel like you've made a lot of progress. Is it just when things feel stuck or like what leads to that feeling of like, oh, like today wasn't productive? I feel like there's so many mix of days at work. There are days when like we'll spend all day at debugging customer issues and in a way that's satisfying because sometimes we figure it out, but in a way it feels like I didn't do anything that day. So I was like, oh, I'll write the CRD today and then it's like, nope, didn't write it today. And then the next day something comes up where there's meetings all day and then you keep postponing it. And then it's like, it's been a week and I didn't write this thing I wanted to. So that always kind of bums me out. But yeah, like today was a no meeting day. So I just wrote. And so that felt good to move things ahead. But at the same time, like also customer calls, it's like every single thing you do kind of shift it in some way. Like every meeting there's a purpose to it. Yeah, I think just speaking from my perspective, that feeling of like being really excited to write a PRD. I think if I explained that to like high school Rustam, he'd be like, wait, you're excited about writing this like really complicated, long document describing some random thing in minute detail. And I'm like, yeah, I've like done all this research and I like really understand it. And now I want to explain this to everyone else. Yeah, and the best thing about writing is down as you kind of organize your thoughts of like, I feel like there's like, I'll do my research. And then I have like all these thoughts and I'm always thinking about this project, but it's when I actually write it down that I can break down of like, okay, here are actually the three main problems and kind of like stepping away from like, you wanna keep going to the solution, but like you shouldn't, yeah. Yeah, no, it's a super good point actually that the PRD and all the documentation we produce is partially or in large part for the people reading it, but there's a huge amount of value and for the person writing it, right? Like just writing helps or forces you to organize your thoughts in a way that just yammering in a meeting or any other form of communication doesn't require. Yeah, and it's interesting. I think Cloudflare is a pretty writing focused company. Like we value written communication. And I think that's one of the reasons we've been able to pivot successfully to more remote work, right? Like we were already super diligent about putting things on the wiki and driving things through written communications versus meetings and stuff. So I think that's one of the reasons we've kept the momentum up during all the COVID business. And that's typical, yep. What other tips do you have for folks that are in college thinking about being a PM or thinking about being a software engineer? Like what, is there some like magic trick to being a PM or is there some, what recipe, if any, is there to follow to get to where you are? Honestly, I feel like people skills. That's like super important. Like as a PM, you have to be ready to kind of communicate with everyone and you're gonna interact with lots of types of people. And so being able to kind of, you might get in a room with engineers and their silence and you have to be able to kind of get them to talk and drive the conversation and kind of like shifting gears constantly. So I think, yeah, clear communication is important. And I'll just kind of, also like empathy is like super important. Like we have to think about like the customer, like what are their pain points? Like they're struggling with this and that's more time that they're gonna spend like figuring something out. Yeah, I think even like in college, though, there's lots of opportunities for you to kind of like PM a project. And I think I got to do that multiple times and that's kind of what helps me kind of decide I wanted to do that more than the actual implementation and the coding part of kind of like doing the research. That's super fun. Talking to different people, getting their opinions and stakeholders and kind of what are the different use cases and yeah. Yeah. Makes sense. So yeah, like try and PM a project yourself. Like right, people work on side projects as engineers. There's no reason you can't do the same as a PM. Yeah. Cool. Well, we have just a minute left. Thank you so much for joining Dina and explaining DNS and all its different forms and all the cool stuff in the hopper that's coming up in the next couple of months. And then shedding a little light on like what being a PM at a conference room. Thank you. Yeah. This was super fun. Yeah, I hope these are fun and not torture. I was a little worried that the wrong reputation was spreading. Yeah, cool. All right. Thanks, Dina. Bye.