Dial Up Motive

Good morning, good afternoon, and good night! Welcome to episode 12 of Dial Up Motive. Here we explore the early Internet, computer, and technological experience of Cloudflare employees. With me today is Sam Marsh, a product manager at Cloudflare. Sam, welcome to the show. Thank you for having me, Dan. All right, to kick things off, would you mind introducing yourself to all of our adoring fans? Yeah, absolutely. So yeah, my name is Sam Marsh. I am the product manager for FL, we call it internally, and I'm based out of out of the London office when when I can actually ever go into the London office. I interviewed there and I've never been back. London adjacent office. Yeah, exactly. Awesome, and could you could you double click a little bit more on that? What is FL? What does it do? What kind of work do you do as a product manager? Yeah, that's that therein lies the question is tricky one is so FL stands for frontline platform. And essentially, and I'm grossly oversimplifying here, we handle every HTTP request that comes in to Cloudflare. And then we kind of pass it off to the products we hand it to access or to workers or just through cash. And we essentially look after that main core component tree. So the number at the moment, I think we kind of see up to I think last update was like 20 million HTTP requests a second kind of burst. So we handle an awful lot of traffic. And we're responsible for not only kind of keeping that alive, keeping that whole whole infrastructure system alive and performing well, but also facilitating all the other products and getting them shipped and getting them out there to customers. And then also obviously trying to re-engineer it, make it faster, make it more optimized, use less memory, you know, those kind of things. And then we're also responsible for things like page rules. And I'm working very closely with the security product teams like Firewall and managed rules. So the WAF teams to basically make the rules engine more faster, more performant, add more rules into in there, such as URL rewriting and header modification. So we're a very, very motley crew of engineers and a product manager, just keeping it alive and making it faster. And what are some of the technologies you're working with? Is it just kind of as obscure Nginx and kind of the kernel level code for the optimizations? What does that look like? Yeah, it's quite a mixed bag. So it's heavily, you know, Nginx historically, and then we've got a lot of Lua code in there. And if you go and find some of our job descriptions for engineering roles we have, you'll see a lot more technology. We're dipping our toes in with Rust and starting to try and get in there with Rust and understand how that can help us or give an advantage given what it's been built for. But yeah, we do use an awful mix, an awful lot, a big mix of coding languages, things like C is involved in there as well. So you really have to be quite a good multifaceted engineer, I think, to work in FL, particularly with all the different teams we work with and all of their products as well, which come in. So now the main question, seeing as FL is the first service that is, you know, inspecting or working with the packet, are you always to blame when something goes wrong? Is yours the first phone that's called? Yeah, the way I always describe FL is it's kind of like a series of sips. It's like Kaplunk, you know, I don't care if you guys have that game in the States, Kaplunk, but everything kind of falls through. The sticks, right, the sticks and the marbles, yep. Yeah, so we're the bottom of that tube, you know, whenever all the kind of debugging's gone through the products, it's gone through whatever it may be, these teams, it always ends up with FL and we always get dragged in. And most of the time we can kind of see where it's going wrong and it's not to do with us, or it's an issue with whatever it may be, esoteric things. But it generally comes to us and we kind of quarterback it back off to the relevant team. So we're heavily involved in a lot of the incidents or a lot of the debugging things, just to kind of help point people in the direction of where the actual problem is. So we do get to see quite a lot of the inner workings. Fascinating, and with that in mind, we can go ahead and take the conversation back a bit. What was some of your first experiences, early experiences, with either a computer technology or kind of the Internet at large? Yeah, so I grew up kind of late 80s, early 90s, in that kind of Windows 3 era. You know, I was about four when I had my first computer. My father was a digital engineer, a deck engineer, so I had my first computer at like four years old, which is scary because my son's four and there's no way I think he should own a computer. You know, I remember typing in Win32 to boot up Windows and just kind of clicking around and seeing that Internet icon and double clicking on it and thinking, why doesn't this work for? Like, this is supposed to do something. And it just, obviously, you're a kid and you know, you don't understand, it has to plug into a piece of copper to go out to some other copper thing. It's just a magical thing. So we never really played with the Internet, did anything with the Internet per se, to like late 90s, I would say, you know, when you kind of go to high school and they say, oh, you've got an email address. I don't even know what email is. And then you start to dip your toes into getting a Yahoo email address, I think it was at the time, and just being like, I have an email address now. I've no idea what I'm supposed to do with this email address. So you email your friends and your friends email you, but you're all sat in the same classroom. So, you know, it's just kind of learning about it. And then kind of taking it from there. These are like slow notes that you have to be on the computer to send, like, why would, I'll just write you a note. I know, it's so cool at the time, like, I sent you a message on the computer, it turned up on your computer, like, whoa, this is so mad. And then you realize that's not even Internetworking. Sometimes you can do that intranetworking and that blows people's brains. But yeah, and the thing I kind of look back on these 20 years now, and you start doing all this is all the embarrassing stuff you did, and you set up on the Internet, things like Lycos and GeoCities, and, you know, that kind of stuff. And you think, God, I'm glad they folded, because I would hate to find my... I was a proud user of GeoCities. I went down the rabbit hole that was Flash and made a Flash website that was way too much Flash. Yeah, I went through that learning experience. Yeah, I think my first website was just iframe hell. It was just a website with the iframe for everything. I was like, well, look, it's cool. It's fixed. It's fixed kind of height. Yeah, but it's just an iframe. So those early days, I guess, did having a computer early, do you think that kind of opened up the wormhole and, you know, started having you ask those questions of, okay, why does this link not do anything? Like, how do I make it do stuff? How do I use the modem? Yeah, I think so. So I was five when I started what we call primary school in the UK, which is kind of like infant school, your first school. And I was the one fixing the computers in the school. So they would kind of come in and say, oh, can you help install Encarta 95? Did you send them the invoice afterwards? I should do it. I went to the statute of limitations on billing stuff. But yeah, I was like a six-year-old fixing these computers for these primary school that I was going to and stuff. So yeah, I never really had a choice but to end up in IT. I was kind of born into it, you know, as always going to be happening. Yeah, the funny thing is everyone talking about like the copper and connections, everyone I know went from, in the UK anyway, went to the kind of 56k Internet connection, then 128 and then that path. And I remember having, my dad for some reason got through work a 64k ISDN connection because we were kind of so far from the Internet exchange and whatnot. So all my friends were on these 256k connections and I'm sat there on this crappy 64k ISDN connection. And I remember when we spoke last week, you know, that's what got me into things like the Internet because you're playing computer games and you see ping times and you think, why does a worse ping time, higher ping time mean it's harder for me to play games, you know? And then you get into the idea that there's a connection somewhere between my computer that I'm clicking these buttons on and there's a server and then that has to, my clicking this mouse press to shoot somebody or to drive a car has to go somewhere to be actioned. So, you know, that's why it's better to have a faster or, you know, a better ping connection to certain servers. So that kind of opened my eyes to the whole thing and cursed me into this life I'm in now. Because then it just begs the question of like, well, how do I make this number smaller? How do I make it faster? And then just kind of the engineer drive of like, okay, now let's take those steps. Yeah, it's more like, how does it work? You know, if this is just a server, then I could have my own server. So let's just get my own server. And you find that it costs money. Well, I'm like a 13 year old kid, I don't have any money. So I'll just find a way to install it at home from spare parts and let my friends play on mine. And then they'll all complain because I've got a 64k ISDN connection. So my server sucks compared to some of these other kids. But yeah, and then I even expose you to the command line, because you have to go in the command line to see these ping times. You say, wow, yeah, I'm a real hacker now. I'm command lining stuff. And you're obviously not, but you feel like you are. And what games were you playing? I know we've on this show, we've discussed early MMORPGs. We've discussed early first person shooters. Gaming was definitely an entry point for a lot of people across the globe as the Internet was taking off as computer gaming was becoming more prevalent. What game did you make a server for? Yeah, mine was Counter -Strike, the original full blown Counter -Strike based on Half-Life. So started playing that in 1999, I think 2000, something like that. And then you kind of get into IRC and QuakeNet. And then you kind of go from there to having guest radio shows on Winamp for gamers and stuff like that. And then you think, how did I get into all this? And nowadays, if you said, if you said what you were doing as a parent, I'd be horrified. Oh, yeah, I'm chatting with random men on the Internet and in chat rooms like, oh, yeah. Yeah, they invited me to their radio show. This is great. Exactly. This was some 14 year old kid and I'm on chat with them, you know, and you look back and you think, oh, yeah, that was a different world. It is interesting how much having conversations like this of a wild, wild west it all was. And so even as a parent, you know, as a parent now, like wondering, sure, there are more controls and a little bit more contained. But then do you lose some of those learning moments where they don't have just the accessibility or things are a bit more locked down than they were before? Or you have to go buy, you know, kits in order to teach them what used to be just you playing with a server and actually building it yourself? Yeah, but on the flip side, though, the good thing about what's happening is the fact that you can get a server for free with pretty much any provider now, a virtual server, obviously, for free with Amazon or Google or Azure, and you can spin it up. You can get any operating system you want on it. I think you can even get Windows on it free now and you can just start playing around. You've got a public IP address, you know. So what was a complete pain in the ass 20 years ago, grab some spare hardware, get it working, you know, get it exposed somehow. All that's been taken away. So you can you can start to play around with it much easier, I would say. So the barriers of entry have shrunk and it's easier just to get into more of the the meat of I'm running a server. I can put code on it. I can deliver a website. I can do what I need. I mean, it's fascinating just how many Minecraft servers are being created these days. That's almost the new Counter-Strike is someone just gets that free server, spins up Minecraft, and plays with their friends. Yeah, it is. All you need is that virtual server for free. You install Minecraft or whatever game is on there and then you're the administrator. So you have to look for the logs. If there's kind of like modifications you want to install, you can kind of put them on there. You need to install them. And before you know it, you're a full-blown server admin, you know, looking after these things. And I think that's kind of like I said how most people get hooked because you just want to make it faster or install this patch or hack around with this. And then you just think, what else can I hack around with? And then, you know, you're in it basically. And how do you think, so going from helping install Encarta as a young kid into running your own server, how did that evolve through university days and kind of your early career? Yeah, it's an interesting one, university. So I would always say to kids, you know, try as hard as you can in school. But I was never someone who bothered trying very hard. I always liked to kind of like play around and learn on my own. So I'd take things to pieces and fix them and put them back together. And then they would give me a book on like object-oriented programming. And I just, I don't care. I'm not interested about this. It's not for me. I like playing with infrastructure, but there was never a kind of computing for kind of people who like to touch or servers or kind of thing. And I went to university and we did our degree there. And one of the things I was lucky enough to do was to do an internship, like it was 13 months, like part of the degree exam, which year they called it. And you'd go to a company, you'd spend a year working there, and then you'd come back and you complete your degree. And I did my internship at Intel, the chip manufacturer, and got to play with some really gnarly, really cool servers there. And, you know, building pre-production stuff, getting exposed to customers, working with customers on technology in that kind of pre-sales role and understanding what they need and how you can help them and things like that. So when I went back to university, I was like, okay, yeah, I kind of get how this clicks together now. I didn't understand when I was working in the books. So like, it doesn't really make sense. And literally from doing that year, my grades, my first year grade was like 47%. And my second year grade was like 51%, which were just abject, you know, and then I went back to university. And in my final year with my dissertation, which is a full job itself and working full-time hours, I got like 84% in my final year of my degree, because you kind of get that focus, like this is why we need to understand how WiMAX, you know, works and stuff like that, because there's a real world application for it. And do you think a lot of that was going from what felt really abstract when you first started to actually applying it or seeing it in a more concrete kind of interaction or concrete environment that allowed you to say, oh, so this is something I should be taking a little bit more seriously, or I'm now more interested in it because I see how these pieces interlock? Exactly. Yeah, I think it's more the fact that you can see how this would have been used or will be used. And having that real world experience, we end up just kind of talking about theoretical things or, you know, Tom's Tire Company, it's like, you know, you kind of know how this is really used. And I kind of worked full-time hours in my final year as well. And having that hand in hand gave me that I learned this on a Monday and on a Wednesday, I can actually apply this because I'm building out a call center for someone who wants to run an open source, for example, and here's why real-time streaming protocols are interesting. And then you go and do an exam on real-time streaming protocols, then you find out you know a lot about it because you've just deployed it like two weeks beforehand. So, yeah, I think having that real world, getting hands-on to break something, it kind of really taught me a lot more. And, you know, we used to spend our spare time buying old hardware on eBay that was broken and we'd buy it for like £10, you know, and you'd get it home and you'd fix it and you'd get it working and it's like a £700 machine. You would never be able to buy it but you could kind of get it for scrap and fix it. And sometimes the fixes were hilarious, you know, you'd buy a broken server in quotes and you'd take the hood off and one of the fans had died and you'd swap the fan out and it would boot and you'd get yourself a brand new server which you can install virtual machines on and stuff. So, yeah, we had like a full rack in our living room just full of spare equipment that we were just practicing VLANs on and you name it basically. So, it's basically nerd and geek paradise in your university home of just servers everywhere. And it's interesting that that idea of taking things that people were disposing of from just lack of knowledge or lack of context and being able to quickly revive them, you know, some percentage of the time and then getting full-blown enterprise-grade hardware at a fraction of the original cost for, what, a few dollars for a fan? Yeah, exactly. And we used to sell quite a lot of it back onto eBay the day after because you could then make some actual money to go and, you know, buy beer and do fun things with as well as learn computing. So, is this how you funded your first startup? Yeah, yeah, yeah. This sounds Michael Dell-esque, you know. No, I'm not that bad. All right, and with that, so once you kind of got out of university, you've had this history of tinkering, of building, of being resourceful, of, you know, connecting the dots to kind of the application. What was your first job like? Yeah, so I came out of university right in the middle of the last recession, which sounds crazy to say now. And, you know, all the graduate schemes were either cancelled or they were literally being kind of, you know, offers withdrawn. And I kind of came out the summer after my degree finished and jobs were just disappearing left, right and centre. And, you know, I'm from and I grew up in Manchester, which is the north of England. And I just wanted to go and do something interesting. And I applied for a job down in where I live now, just on a whim, a storage company, which made storage for supercomputers. And I thought, well, that sounds like what I quite like to do. And took this job, moved like 250 miles on my own, just for a job. And I've been here, met my wife and had my kids here and it's 12 years later. So, you know, it's very fortuitous, just these things that happen. But yeah, I took this job because it was, you know, it's a job for one. Back in those days, it was like, I just need a job, I need to make some money. And it was like level one support. And then it very quickly graduated from level one support working there to going working at Cisco. Obviously, I was a networking guy and Cisco was mecca for networking, probably still is for most people. So I spent two years there working effectively as like a level four engineer on some really hardcore nasty stuff. That exposed me to working with customers and talk to them, what's happening, how can we help you and build that relationship up from a technical side of things. And then after a while, I thought, I'd like to go and be on the side of the conversation, which is where I'm not getting shouted at for breaking stuff. I'd quite like to go in and start selling these things to customers first and making sure that they're buying the right thing. But now you just have sales teams shouting at you. Yeah, exactly. Why can't we do this now? I'm a product manager now, so I'm more used to being told this is too late or it's not quite what they wanted. But yeah, I went and worked on that side of the coin, worked in pre-sales for two years in a Linux software company. And the product manager left and I was kind of quite heavily as pre-sales people do telling engineers what to build and why we need to build this now because this deal and yada yada. And my CEO one day just said to me, look, if you know it all, why don't you just do the product manager job? And I was like, okay, cool. I'll do it part time. I have no idea what it means, but if it gets what I want built, then let's do it kind of thing. And yeah, seven years later and three or four companies later, I'm still in product management, in technical product management. And yeah, it's just a complete mystery how I got here. Awesome. Well, I mean, it sounds like a good fit given kind of your history, tearing things down, building them up, being a connection point between other people, building servers, and especially that stint on the pre-sales side, I think definitely is, I would argue being on the sales side, having that front row seat to customers is ultimately just very helpful to understand what they're asking for, what they actually ultimately need and that difference between the two. And then of course, prioritizing as a product manager. Yeah. My next step can be in marketing or something. Just do the full tour. So what would you say is one of your more difficult roles as a product manager or more difficult tasks? Yeah, it's generally, it's cliche, right? But it's generally just balancing out the tactical with the strategic. You always have a long -term goal, which is inevitably expensive to do, either time expensive or resource expensive. So you generally have that, but you're never left alone, obviously for two years to just build the strategic goal. There's always tactical stuff, whether it's close to steel, it's because we've just found a vulnerability or there's a myriad of things which come out, which kind of drag you off of that path. So I think the hardest thing as a PM is just not going fully into the whole, let's just build what people want, close these deals and move on. Because I've seen firsthand the result of that decision-making and it leaves you in an absolute mess when someone who comes out with actual product leadership comes out as a product. And then all of a sudden you're exposed for what it is, which is just a bag of enhancement requests with the UI basically. So you need to have that one eye on the vision, one eye on the horizon always, and be quite strict with what you're building. And inevitably you're going to fall out with salespeople, you're going to fall out with executives, you're going to fall out with customers because you're not going to build what they want. But while they say you're the kind of voice of the customer, you're ultimately the guardian of the product, right? And you have to make sure that you can sell it today and you can still sell it in 10 years and it'll still solve problems in 10 years, rather than solving the problems of three well-paying customers in three years who can quite easily leave you and you're left holding the baby with this product. So prioritization and making sure that you do what everyone wants a little bit is the trickiest thing I would say. All right. So there is no black and white, I'm going to lock myself in a room for three years and come up with a product that fits the ideal. Meanwhile, I'm not going to say yes to everything that comes down because then you just get that fractional increase or those series of enhancements that don't actually get you where you wanted to be originally. Exactly. There's that balance and that dance. Exactly. Yeah, exactly. It's a tricky one because you kind of want the time to build this kind of CEO you have in your head. But at the same time, I've done it. One of my first things I did as a product manager was build this big shiny thing, which took nine months and you find out you wasted about half that time building something that no one really needed. So the whole like ship regularly, just get bits out there and see is the golden rule, I think. And then the other kind of adage is obviously we spend three years building something that when you ship it, you're three years behind the competition because you built it based upon 2020 needs and in 2023, things have moved on. Interesting. And with that, I know we're nearing the end of time. What are you most excited about for the future of the Internet, either what you're building here at Cloudflare or what other technologies are doing? Yeah, I think the thing which fascinates me is looking forward into the future and kind of predicting things which are going to make me feel very old, you know. And when my grandparents tell me about these things like, oh, we used to have toilets outdoors, or we used to have to go and fetch water, you know, being facetious with a bucket. And then I can imagine telling my kids like, oh, yeah, we used to have these pieces of copper that you have to plug into the wall to get on the Internet. And they're like, what? Why? What did you do that for? I got Neuralink now. What are you talking about? Everything's on 10G. And you know, it's 600 gigasecond. And like, yeah, we had these pieces of like phone, you couldn't hold the phone to your, you know, head, because you just get a sound in your ear. So I look, I kind of look forward and I try and predict these just absolutely ridiculous things. We're going to tell our kids like, yeah, you're going to think this is ridiculous. But we used to have to dig up roads and put pieces of copper in the road to be able to connect to the Internet. And now you just like said, your phone supports 300 gig and can tether like 600 IoT devices kind of thing. So yeah, that's my favorite thing to do in life. In general, it's just trying to think like, what's gonna, what's gonna be that silly thing when I'm 90, if I'm lucky enough to make it to 90, that I can tell my grandkids and they'll be like, what? You lived in a weird world. Fascinating. Yeah, it'll definitely be a very different world. And as always, just given the rate of change that we've seen in our lifetime, from a technology perspective, if you even if that slows down a little, the amount will continue to progress will will of course be almost absurd. By the time you reach that age where you're talking to grandchildren, and I know kids nowadays won't understand what a server is anyway, because you've obviously got virtualization, the cloud and serverless on top of that, as well as all these abstraction platforms. So in five years from now, you know, I dread to think what it will be when you say again, this piece of this piece of melted glass runs all this computing. Well, with that, we're near the end of time. So Sam, I want to thank you for a lovely conversation and I'm glad to have you on the show. Yeah, thank you very much. All right. And with that, thank you everyone for joining, whether you're catching us via the live stream or a recording. Have a lovely morning, afternoon or good night. So you run a successful business through your ecommerce platform, sales are in an all time high, costs are going down, and all your projection charts are moving up into the right. One morning, you wake up and log into your science analytics platform to check on current sales and see that nothing has sold recently. You type in your URL only to find that it is unable to load. Unfortunately, your popularity may have made you a target of a DDoS or distributed denial of service attack, a malicious attempt to disrupt the normal functioning of your service. There are people out there with extensive computer knowledge whose intentions are to breach or bypass Internet security. They want nothing more than to disrupt the normal transactions of businesses like yours. They do this by infecting computers and other electronic hardware with malicious software or malware. Each infected device is called a bot. Each one of these infected bots works together with other bots in order to create a disruptive network called a botnet. Botnets are created for a lot of different reasons, but they all have the same objective, taking web resources like your website offline in order to deny your customers access. Luckily, with Cloudflare, DDoS attacks can be mitigated and your site can stay online no matter the size, duration, and complexity of the attack. When DDoS attacks are aimed at your Internet property, instead of your server becoming deluged with malicious traffic, Cloudflare stands in between you and any attack traffic like a buffer. Instead of allowing the attack to overwhelm your website, we filter and distribute the attack traffic across our global network of data centers using our Anycast network. No matter the size of the attack, Cloudflare Advanced DDoS Protection can guarantee that you stay up and run smoothly. Want to learn about DDoS attacks in more detail? Explore the Cloudflare Learning Center to learn more.