Dial Up Motive

🎵 Upbeat Music 🎵 Good morning, everyone. Welcome to the third episode of Dial Up Motive. This is when we explore early Internet stories and how they formed a career story for our employees here at Cloudflare. So today I am joined by Ammar Zuberi, a systems reliability engineer at Cloudflare. And ideally, we'll go through a fun and riveting tale about his adventures on the early Internet. With that, Ammar, can you provide a quick introduction of yourself? Hello, everyone. My name is Ammar Zuberi. I am a systems reliability engineer at Cloudflare. I am actually pretty new. I've been here for about two months. And my core job is about making sure that Cloudflare systems in our core data centers stay online and stay healthy. And in your two months, what's been your biggest surprise so far? I was expecting it to be less hectic, I'd say. But I've been trying to describe how my first two months have been. And I think the easiest way to explain it is exponential growth. I think I've learned probably twice as much as I did the previous day, the next day, pretty much every day for the last two months. So that's been interesting. So yeah, we keep everyone on a sharp learning curve. Yeah, the sharp learning curve is definitely real. And just for those that are unfamiliar with the role of a systems reliability engineer, could you go into a bit more depth just generally about what that role does? Right. So I touched on how it's very simply your job is to make sure nothing breaks. And when something does break, you have to go deal with it. So naturally, what that makes you do is want to automate as much as you can so that you have to do as little work as possible. So that's pretty much what systems reliability engineers do. We're, I guess, software engineers that we also have to deal with production. I think that's an interesting way to put it. So if anything, when you deploy code, it's to keep all the packets flowing. And when there's ever an issue, like discovering the issue and then triaging it appropriately. Exactly. And we also do on-call rotations. So we do releases where we release code for other teams every 10 days or so. And then we do like an alerts rotation, which I actually haven't done yet. So I probably can't speak with too much, I guess, hard with that one because I'm not as scared of it as I should be. But alerts will be an interesting time. I'm sure it's not that scary. But I've always been a few layers removed on working on the customer side. So I always had the SRE team to kind of take the brunt of network issues. So diving into kind of what brought you to Cloudflare, what was your early career journey or what was your path like there? So I guess like going with the theme of the show, I kind of remember dialing up on a dial-up modem like really early in my life, but not really. Like I remember it happened, but I remember what provider it was or where I was. So I guess the point of me saying that is I have done that before. I'm not that much of a baby. But anyway, so my early journey was, I would say, probably around 2009 or so, 2010, 2009. Around the time Cloudflare was started as a company, I was like, I don't know, like 12, 13 years old. And I played a lot of Minecraft, RuneScape. I played a lot of RuneScape. And one of the things I did around then is I started my own RuneScape server. It turns out that people in the gaming community aren't the nicest, and they really enjoy DDoSing and all sorts of like hacks that they could do against your game servers. And this was pretty much widespread across every single person who ever ran a game server in the RuneScape community, pretty much full stop. So that's pretty much where I got my earliest intro into everything that I do today. I started RuneScape server, which actually started on a laptop that sat under my bed. And then as I got more players and I was starting to handle more bandwidth, my home connection started getting weird. My parents were like, you need to stop doing whatever you're doing. Our Internet's too slow. So that's when I started, I got a virtual server somewhere in the UK, if I remember correctly. And that's pretty much where it all started. I learned how to use Linux from then. I learned very basic network administration stuff. And yeah, so that's pretty much my earliest taste into, I guess, the hard technical work that I do today. And did you ever dip your toe into the black hat side of things and actively DDoS or try to hack any of your peers in the RuneScape community? I have no answer for that. All right. You plead the fifth. I will plead the fifth. Yes. And I guess what intrigued you about RuneScape, about that type of game or about the strategy involved? The kind of community that you got out of playing those games back then. I don't think you get that as much if you do that today. I think a lot of these games are like highly commercialized now. I mean, RuneScape isn't as popular as it was in 2009. I mean, they just got like a big bump in players because of the COVID pandemic. But because people just want that nostalgic feeling these days. But it's just not as big as it was back then. Now everyone's going to sell things. There's other websites that surround the whole community where you can buy money and things like that. So it's not as authentic as it used to be. So it was interesting to go on there and meet all these people who were just trying to just do whatever they could. And they were all around my age. So it was interesting to do. But now everyone is older because it's a lot of people who are playing for nostalgic purposes as opposed to actually playing the game. But I'm not talking from an informed point of view because I haven't spent that much time in the game in the last probably five years at least. Yeah, but definitely at that time, it was very formative for, you know, it was a group of peers. Many that you'd probably never met in real life, you know, all joining around this common experience of with, you know, varying quality because online games were just figuring out the world. There was always lag and latency. And of course, in your experience, DDoS attacks across the board. Yeah. And I grew up in Dubai. So latency there was worse than it was here, too, because they didn't have any POPs nearby. So I was always at a disadvantage playing pretty much every online game growing up as well, which was an interesting perspective that I guess you don't have if you live in a country where there are POPs for pretty much every game. Oh, no, no, I understand completely. I grew up on a farm. So for a while, one of our means to get the Internet was actually essentially a cantenna that we put up on a tower and pointed at another tower. And so, as you can imagine, the packet loss was not exactly the way I wanted it to be as someone diving into online games. Yep, 80% packet loss. Sounds great. Yep. So definitely can feel that, you know, you're not at the peak compared to some of your peers because you're dialing, you're getting online in a very different way or from a very different location. But a lot of the community aspects are still there. You can still enjoy that time, despite it not being, you know, a high bandwidth experience that you have today. And there were people who were in much, I guess, worse Internet situations than I was. So it all kind of balances out. And the people at the top, you can kind of tell are the people who have the best latency because they can win the games fastest because their games don't lag out. So it was interesting to see. And what do you think was that pivot point from, you know, I'm playing this game, I'm having a great time as a consumer to deciding, oh, I want to open up my own server. I want to play around with this at a deeper level. What was that mindset for you? So I remember, not that vividly, but I somewhat remember. I don't remember the year, but I was just playing RuneScape one day and I saw some guy who came on and was like, hey, join my RuneScape server. And I was like, OK, I'm like 12 years old and was like, OK, what is that? Let me go click this random link that this guy just sent me in this game. So I did exactly that. And I found myself playing on this other guy's RuneScape server where you could get whatever armor you wanted. You could play, you could do whatever you wanted. And the guy seemed pretty cool and he was giving out whatever people were asking for. So I was like, oh, maybe I'll start doing this. So I started playing that. And I guess that in an interesting way, in a way that I haven't realized before, is that might have been the spark that was like, oh, I want to do that. That sounds pretty cool. Like, I want to be that guy. So that's pretty much where I started to learn how I would run my own RuneScape server. Back then, there were people who had pretty much reverse engineered the entire protocol for RuneScape who had already released their source code. So you could use that to build off. Back then, I didn't really know how to code. So I just ripped someone else's code directly off one of the forums that were around back then. And yeah, and I started building on that. I think everyone starts by copying and pasting. Like, there is no getting past that phase of, yep, I don't understand how this works. But I know if I put it here and run it in this way, the output will be a RuneScape server. Right. I've been writing code for probably 10 years and I probably still do that. I mean, the way Stack Overflow works, I think the majority of engineers are still doing that to some degree. Exactly. Hopefully they understand over time what they're pasting. But still, that's a starting point. I'm finally getting there now. So you opened up, you essentially had this experience where God mode was opened up to you in this world. Great. I want my own server. I want to be able to dive into this myself. How did that continue to grow and how did that curiosity continue to develop? I think from RuneScape, I pretty much branched out into two sides. One of the things I realized at that point was a lot of people also wanted to do this. It wasn't the simplest thing to do. These days there's SaaS products and all where you can just pick up or subscribe to some service and they'll just set up a game server for you that you just hand out to all of your friends or whoever and they'll join and they'll handle pretty much everything for you and you never have to touch the line of code. Which is, I guess, you lose out on that kind of experience that I had if you're doing that these days. But back then that wasn't really a thing. So what I realized is a lot of people were in the same boat I was and wanted to start their own game servers where they could do all of these things except a lot of people didn't have that technical knowledge. So that's pretty much where I started out. I created a small hosting company where I would basically set up RuneScape servers for people. As RuneScape got less popular, Minecraft got more popular. That was around the time Minecraft got really popular. And I started doing that too. So I would set up Minecraft servers for people. And it turns out that the Minecraft community is actually more toxic than the RuneScape community. You get hit with DDoS attacks probably every day. I know this is a problem still. One of the largest Minecraft servers is a Cloudflare customer. Which you'd be surprised by because it's like, why would they need that? But they actually do. And the kind of money that people are making in the Minecraft community is actually a lot of money. So there is value to protect in that space. And do you feel like... So it's a very similar skill set as you start to learn how to set up a server. You also learn how to tear one down. And so you almost feel like some people, they get their thrill from... Great, I want to spin these up. I'm going to start my own little business around helping others set them up. And others are like, yeah, I'm going to just go down the path of darkness essentially. I'm having fun DDoSing other people or taking these servers down. And you get this kind of cat and mouse game. And in 2012, it was pretty harmless. Nobody was launching 300 gigabit per second DDoS attacks against actual businesses. It was people launching DDoS attacks against Minecraft servers owned by people that they knew. So it was a very different... The Internet, I think, as everyone knows, was a very different place a decade ago. And I think that's relatively true for the network attack space as well. And that's where Cloudflare started. Cloudflare started, I guess, pretty much out of those smaller companies that could not offer their own DDoS protection. Which is why a lot of people jumped to Cloudflare. Which, I mean, I did. When I started at Cloudflare, I sent Matthew Prince an email. Before my first day, where I pretty much laid out the story in summary form. Saying, I literally came to Cloudflare because it was a big part of my childhood. So it's interesting to see myself here at Cloudflare now. Given that Cloudflare was a company that I pretty much relied on for a good part of 3 -4 years. When I was 12. I mean, that had to have been a very nice letter both to send and receive. Or a nice email like, hey, your company was formative to my Internet experience. And now I get to work here. Like, this is awesome. It feels exactly like that. And you had an interesting story when we synced earlier. About you essentially making a micro -Cloudflare. Right. And I assume this grew out of your kind of Minecraft as a service business. Can you go into more detail about that? So I talked about how I would host game servers. I slowly started branching out to all sorts of stuff. I would pretty much host whatever you want. As long as it was legal, obviously. Turns out that's actually something you have to deal with. There's people who want to host illegal stuff. And you have to go find them. And you have to remove their accounts. And there's a whole process that goes into that. Which, if you're someone who's also at high school, is kind of hard to do. But to answer your question. Going further up the stack, like I said. I started playing RuneScape servers. Like, I want to do this myself. I started offering DDoS protection to other providers. Like, I want to do this myself. So, probably around this time, I'm like 16 or so. And I decided I would do that. So, like I said, I lived in Dubai. Which was a good central point in the world. So, I could pretty much travel to wherever I wanted. I guess back in those days, you can't travel anymore. But, so. I did that. I bought some servers off eBay. I had them shipped to the data centers that I wanted them in. I flew out there. I put them in. I signed the contracts. I got lots of looks. Every time I signed a contract in person. They were like, are you sure you're supposed to be here? Which was an interesting experience as well. But, yeah. And that's pretty much where I started. So, I built a mini cloud player. And I offered DDoS protection on my own. Because I didn't want to pay anyone else to do it anymore. So, out of curiosity. How many servers and how much network capacity did you have at that time? I got up to. So, I handled multiple hundred gigabit DDoS attacks all the time. So, I did that. Clean network capacity. I don't think I ever broke 25 gigs per second. But it was still good. I probably had two or three hundred cloud servers running at any one moment across the world. I had a rack of servers in four different locations. And so, this was very similar to Cloudflare's architecture. Where you were running any cast IP across those locations. You were proxying the traffic on behalf of the server sitting behind it. And just handling all of the flood and volumetric attacks. Right. A lot of volumetric attacks. A lot of layer 7 attacks towards Minecraft servers. Which was something that someone I worked with at that time was also in the Minecraft community. He wrote a layer 7 module that would pretty much mitigate all of the attacks that I saw at that time. I know he's still around in the community. And he's still working on that kind of stuff. I've recently been reconnecting with all the people I knew from a decade ago. Because of the COVID situation, I'm finding myself sitting at home bored. Where are you now? Yeah, exactly. Let's spin up a RuneScape server. Yeah. It's interesting to see all these people who are still doing the same kind of stuff. So far on after I left the community. And I went to go work for a large company. I went to college. Then I went to go work for a large company. And then I'm at Cloudflare. And now at Cloudflare, I feel more empowered to go and reach out to these people. Because at the large company, they were not into me, I guess, reconnecting with people who I connected to a long time ago. At Cloudflare, I think Cloudflare is more open to me going back and learning about people who were, I guess, involved in this kind of stuff. And I don't want to go too deeply into it. But you asked me about how much Black Hat stuff I did. These were people who were involved in that kind of stuff. So I went back and I reconnected with them. They're not doing that kind of stuff anymore. They're all still in the Minecraft community, which is interesting. How many of them are in prison? Zero. Okay, that's good. Never got raided by the FBI. And out of curiosity, all these servers, I'm assuming they were just running Apache or something else? So it was a Debian stack. I'm still a Debian person. And yeah, pretty much everything else was made by us. I did ACDP forwarding for a little while. But it turns out that no one wants to pay for that since Cloudflare does that for free. So yeah, I stopped doing that. And that was just an NGINX module. We did the whole I'm under attack mode as well, just like a JavaScript challenge. So we did that for a little while. It wasn't cost effective because nobody wanted to pay for it. I mean, at a certain point where you're taking queues from Cloudflare, so Cloudflare released X feature, and you're like, well, yeah, that's easy enough to implement on my set of servers. I might as well do that too. I mean, I think I beat Cloudflare to that one. Okay. The whole layer seven attack stuff was, yeah, I'm not sure when Cloudflare came out with that. But I think it was probably around the same time, if not a little bit earlier for me. All right. You guys, you can take that up with our network team and decide who owns, you know, the mental patent. Yeah, but it's really interesting to see, I guess, be on the inside now and see all of the things that I like to go back and look through the wiki about like developments that happened around that time. It's like, oh, wow, like I went through the same process and I had the same thought process with a different entirely different set of people. And is it fascinating seeing the scale difference? So just like this is, you know, a grown up network that's evolved over the past 10 years and to be able to see it from both sides. So as a customer, as someone emulating kind of this solution in a smaller scale and a more niche market, and then to coming on this side. Right. And that's, I think, the most interesting part of working at Cloudflare for me. I've been able to, I mean, like I go look through the source code. I go look at how did they do this thing that I did 10 years ago? And it's really cool to see, because like you said, it's a huge scale difference. I never had to like be as optimal as some of the decisions that are made at Cloudflare. And I've been talking to people about who have been here for that long. Like, hey, I saw that you were involved with this project in like 2011. And they're like, why are you talking to me about this? I don't do this anymore. But it's really cool to see and go learn about all the different choices that were made and how some of them were very similar. And I thought I would find myself going to go and went through that same thought process. Whereas a lot of things are very different, which is also very interesting. Yeah, I know the wiki is always a treasure trove. And you can get kind of lost in some of the essentially archives of deprecated information. But it's still fascinating to be able to read through the documentation and be like, see the thought process that, you know, this project led to this, this project led to this. And now this is what we're delivering or about to deliver. And it's just a fascinating kind of history lesson. As long as you're not trying to like legitimately find or search for something, and you're just trying to get lost in the history, it's perfect. It's really easy to get lost, for sure. Yeah, I definitely spent my first week and a half just reading through various wiki pages and going down the wiki click hole where you click one link and you click a different link and you just keep going. When was this page updated? Oh, 2011. Perfect. So would you have any advice for, since now it's almost easier to spin up a server, spin up a virtual server, you know, would you have any advice for kids that are in your shoes that are, you know, if they've found a community gaming or otherwise, they want to learn more about not just using that community, but helping build it? What would you kind of tell that version of yourself? I mean, that's a difficult question, just because I feel like these days, like I said, everything is so commercialized. When you go do something like this, I'm guessing you'll go on whatever gaming forums around these days, and you will start spinning up your own stuff. And you'll find probably a million ads by companies who are willing to do this for you for the small price of like $9.99 a month or whatever. So I think it's easy to do that. Sometimes you have to take the path of more resistance, I guess, because that's their whole business model. They give you the path of least resistance such that you pay them, such that you don't have to do any work. But that's where you don't learn anything. So I guess my only advice is maybe try not to take that path of least resistance and try and set things up yourself if you can. So maybe try be a few layers below layer seven if you can. I mean, if you're interested in it. Some people just aren't interested. But if you're anything like me, I guess that's my advice. Yeah, it's interesting as things become more commercial, they leave that hobby stage where you do have the ability to kind of hack with a mod or pull up your own server and adjust things. And so unless the company, especially on the gaming side, is friendly to that or leaving room open to keep it more of a hobby and less of just a product, you lose that entry point. Yes, and I know that's the case with RuneScape now. It was starting to become like that around the time I left the RuneScape community. The company that runs RuneScape is called Jagex, and they had got really strict about people reusing their gaming clients and stuff to connect to their own servers. So around that time was when they started cracking down on that, and a lot of really large servers, not mine, I wasn't popular enough, got like a cease and desist notice and stuff like that. These days it is definitely harder. I know Minecraft has also made that much harder. I guess in your personal opinion, is this a good thing for these communities and kind of burgeoning SREs, or is this a negative impact that we'll see a decade from now? I guess going in the typical negative SRE fashion, I'm going to say I don't like it, but I also don't have a solution. Maybe the next generation of games will be less profit-focused and maybe more community-focused, but who knows. I think the games that are being played by the masses these days aren't like the games that were being played back then. I'm not someone who plays games anymore at all. I can't remember the last time I played an online game. I have two kids at home. What are games? I look forward to when I can introduce them into the gaming side of things, but until then, I don't remember what those are. With that in mind, I guess we're near closing here. What are you most excited about in the future into the Internet? As someone working here at Cloudflare, you really get to work at that scale and define the future of the Internet and working with the latest protocols, network design theory. What are you excited about? I think, like you said, being at Cloudflare allows me to explore things that are really on the cutting edge. I can really talk to people who are working on stuff that literally goes to the IETF, becomes a new standardized protocol, etc., which I think is super cool. That's something I'm really excited about, like helping to maybe shape the future of the Internet and come up with more secure protocols. I think that's an interesting thing that I'm looking forward to doing here. One of the other things I like about Cloudflare, to go off on a tangent as I do, is Cloudflare really allows smaller people to still thrive in that we'll sit there with our huge 200-some pop CDN in front of you and allow you to operate at the scale that someone who's paying millions of dollars is for pretty much nothing. I would like to build features here maybe that allow that to happen more, if that makes any sense. I'm not sure if that made any sense. From a feature perspective, you still want to provide a level of scale for those that essentially can't build it themselves, and would love to be able to rent it. Right. That's part of it, because now I'm sounding like the commercialized people who I was ragging on a second ago. I guess it's more of how can I enable people to do all those things while still learning, as opposed to having this entirely turnkey solution where you just pay and it's done, as opposed to, let me do a little bit of work. I think even just setting up Cloudflare is a learning experience in itself if you've never done it before. When I first joined Cloudflare, I was actually coming from grad school. It'd been a while. I'd had web dev experience, but changing my DNS again and refreshing myself on all of that. I was like, oh, I know how this works. I'll get this. But it just had been so long. That was, of course, one of the things most of our new hires do, either during the interview process or right after, is go set up your own site on our free plan. But it's good to hear that you have goals of continuing that tradition of providing capability to not just those on the commercial end, but ideally anyone that wants to be that hobbyist or to start building that skill set. I think pretty much everyone who is a hobbyist these days who is running a website is behind Cloudflare, which I think is incredibly powerful. I think that's awesome, really awesome. If anything, we need the next generation of online game being written via workers. The mod community are just additional workers, script writers, that are just modding the heck out of it. Who do I need to go talk to about this? I think we need to make a wiki. It always starts with a wiki. I was thinking about things I would like to write for wiki articles, and I think that might be it. I guess I got to go make a game first. All right, there we go. We're near the end of time. With that, I'll thank Amar for taking us through his early Internet history here on Dial-A-Motive. We look forward to hosting you and all of the watchers next week as we'll continue to dive into the world of floppy disks and bulletin boards and Twitch and live stream and whatever else the kids are playing on the Internet these days. Thanks so much for having me on. All right, thank you very much, and everyone have a great day. Thank you.