Story Time

Simon, welcome. This is Story Time, which is a silly show where I talk to people from Cloudflare usually about weird stuff that's happened in the past, problems they've solved, how they ended up at Cloudflare, all manner of things and well you and I have worked together for a hell of a long time now. Indeed, yeah. I think this will be seven years coming up in August. Yeah, I'm trying to People don't know Cloudflare history. I was the first person, not in San Francisco, so when I joined Cloudflare there were 24 people and I was at home alone. Funnily enough, actually the desk in front of me is the really rubbish desk that was my Cloudflare desk and I'm using it now for Cloudflare TV. And then finally we got an office and I'm trying to remember, were you the second person we hired in London? No, so I think I was number five or four, so there were, at that time there was Andrew, another engineer, there was James who was the first SRE in London and then there was Sam and Marty who were the first support people. I'm always struck about that is that of that original crew of people we hired in London, only one person left in seven years basically, we're all still hanging in there. Yeah, it's amazing and actually the two people who joined around the same time and after, Marek and Sigis, they're all here too, so you have moved as well. I think one of those folks is actually thinking of moving to Lisbon as well, so we may all get the whole crew back together again. We'll get the band back together, we'll walk down the river in slow motion. Down the river in slow mo, yeah, it's like the right stuff kind of thing like that. Well, I thought it'd be good to get you on this because of that long time at Cloudflare, you and I were there when things were really small and things weren't quite as stable, as efficient or as organized as they are now. We've had a few sort of crazy things happen over time, one of them is that you ended up getting on aircraft and upgrading servers, right? Yeah, so yeah, you're right, in the early days we ran very, very lean, so we had a team of like five or maybe seven of us in London at that time, but around the world we were aiming to do a bunch of upgrades to our data centers and we had a few things that I think we wanted to do, in some places we needed to add new servers, in some places we needed to add RAM to existing servers and TPMs as well, and also some routers and switches and all sorts of things, but basically we had to do this very, very quickly because Cloudflare was growing extremely quickly and so was our traffic, and we had to do it in basically every location on our network, I think over a relatively short period of time. At that time we didn't really make a massive habit of using smart hands, remote hands, and I think for this type of work we decided that it would be better if we did it ourselves, it would save us some money and also we had more people in more locations, but we now have a dedicated team for infrastructure, and I think we had a network engineer or two and a small SRE team and that was it. So at some point Sri, who was the first SRE at Cloudflare, emailed me or messaged me, I think we were using Skype back then, and just said, hey do you want to go to all of these data centers, and there was like London, Paris, Amsterdam, Prague, there was like maybe about seven or eight of them, and I was like, yeah of course, because in my head I was just like, these are all my favorite cities, why would I not go, all I have to do is shove some RAM in some machines, everything will be fine. So you guys, yeah, I was like, you guys are going to pay to buy me to all of these cities, of course I'm going to do that, so yeah, I jumped straight in with both feet, and Marty, who was working with us at the time, was also very excited and muscled his way in. Marty demanded to be included, and also went to some data centers, so yeah, the job was to install some RAM, it was to install some TPMs, some servers, some routers, the pluribus I think we were using. Yeah right, so it's worth talking about some of the gear, right, so when I joined Cloudflare, we actually had a bunch of servers that HP had essentially given to Cloudflare, and they were installed in places around the world. I think HP was hoping that Cloudflare would grow enormously, would buy loads of HP kits, and it'd be a really great strategy. We did grow enormously, we didn't buy HP kit, and we ended up getting our own servers made by ODMs in Taiwan, but there was this intermediate period where instead of doing what we do today, which is we're upgrading by sending new servers and running different generations of hardware, we actually had to go in and literally shove RAM and I think disks in machines, because we were just growing outgoing RAM, all that kind of stuff, and so this was what led to this, you know, European vacation thing where you go around and you go plug stuff in, and the other thing is we were really, really interested in was making sure that the software that was running on those machines was our software, and to do that, where you mentioned this, these TPMs, right, these trusted platform modules which give you a cryptographic way of verifying your boot process or the software you're running, and we hadn't installed those, and so somebody literally, a little square, right, somebody had to go and power off the machine and plug them in, so where did you go on your, you know, your jolly trip around Europe? So I went to, and I've made notes, I went to Stockholm, Prague, Vienna, and Warsaw, and we also had a little warm-up trip to London, so those of us who were prepared, so Marty and myself, but also Sam and James, and I think Sigis and Marek, we all had the opportunity to go to the London data center, because no one had seen a Cloudflare data center, or most of us hadn't been, really been to a data center at all, so we all jumped at the chance to go, but what people don't tell you about data center upgrades, or it all sounds very glamorous, traveling to all these, all these places, what people, when you're running a live network, you can't take London down in the middle of the London day, so what you have to do is go at sort of two in the morning, when traffic has kind of tailed off from the evening, so that you can safely take the location offline without impacting performance and capacity across the whole network, so invariably what you do is you show up at these locations in the dark, they're normally in sort of strange parts of cities as well, they're not always quite where you think they may be, although in London, it was actually in Canary Wharf, so it's kind of, you know, a very popular space, but at 2am pretty, pretty quiet, so you're, you're quite sleep deprived, and then you have to do a bunch of very intricate work with screwdrivers, unracking servers, carefully placing RAM modules inside, putting everything back, remembering where all the cables went, and so it's actually really mentally taxing work, it doesn't sound like it, but in the middle of the night, when you're sleep deprived, it's really difficult. Yeah, and also the one thing you haven't described for anyone who's not been in a data center is the noise, everything has got fans in it, there's air conditioning, it's actually really not a very nice place to be, and most people who work in data centers are not actually in the data center bit, they're like in a little office with a big window, because the noise is tremendous. Yeah, yeah, exactly, you've got all these fans running constantly, and it just kind of drives you a bit crazy, right, and the air is very strange, because depending on where you are, you'll have like hot aisles and cold aisles, and depending on where you are, it's either going to be very warm or very, very cold, and also very, very noisy, so it's quite difficult, and I remember someone on the SRE team, Josh, saying, explaining like, you are basically operating at half your mental capacity when you're in a data center, because of the noise, because of the sleep deprivation, because you're kind of heightened and stressed, your normal sort of logical way of thinking through things, it doesn't operate quite the same, so you have to be really careful, and actually what you normally need is when you're doing anything really important, you're normally on the phone to someone who is not in that environment, who can guide you and help you, and that was something that really stuck with me, because it was, yeah, you're doing it, apart from the London trip, it was done all on my own, and on the phone to someone else, pretty much, who was making sure that I wasn't making mistakes along the way. I found that, you know, having like a checklist, having things that are really basic, just so that you can keep yourself sane in there is helpful. The other thing that's really can be difficult in some data centers is that many data centers have a restriction where you can't take photographs, and because you don't, they don't want you photographing other people's equipment, so they have a blanket, you can't photograph anything, and the first thing you often want to do is take a picture of the rack, just as a sort of ed memoir in terms of like, oh, where were these things plugged in? Did I actually get it absolutely perfectly right? So, it's a very stressful environment, and I seem to remember that Stockholm didn't go so well, right? Yeah, that's true, that's true. So, Stockholm was another classic example. I remember going to Stockholm, it was snowing, which felt very appropriate, and I had trouble getting in. That was always the hard part as well, like even though you'd kind of, the right people at Caltech had contacted the data center, normally you have to book an appointment, right? They have to know that you're coming, you have to be authorized, you have to provide identification, so you have to do all of that stuff, but invariably you show up, and they don't know, then you have to get on the phone to people, and it took, it must have taken like a good 90 minutes to even get into the place, so I stood outside in the snow and cold for about 90 minutes. I got in plenty of time, same deal, right? You don't start work until sort of, I don't know, one, two in the morning, so you don't actually start touching anything. So, you start kind of preparing, you start unboxing the new kit and getting everything ready, so I'd sort of done all of that, and then you kind of need to take a look at the rack, and a rack of servers, it's just a big column, if you've never seen it, it's just a big column of shelves, and the servers kind of just slot into it, and normally they have a door, and the door might be locked, and the door is kind of a mesh grill, so you can't normally really see what's going on inside, all you can see is a bunch of blinking lights from the machines behind, but in order to kind of assess what you were saying, take a photo, that's totally true, that's really what you want to do, you want to see how everything is laid out, and also when you're communicating with people who are coordinating it, you want to be able to send it to them, so they can see what you're seeing, and help you, but you often can't do that. So, the first thing before the maintenance start, you kind of want to just take a look at the rack and see what's going on, so that seems like a safe thing to do, it ordinarily would be, so I open the door, about maybe a couple of minutes later, I'm just looking at things, I'm literally not touching things, a couple of minutes later, I get a phone call, and it's Sri, and Sri is probably one of the calmest people I've ever met, he's got such a soothing, relaxed voice, and Sri says to me, hey Simon, did you do something? And I was like, at that point all the blood like drained from my body, and I was like, oh no, I literally opened the rack door, and he's like, Stockholm is offline, and I was like, oh god, and eventually what it turned out, that just opening the door, that the router hadn't, the power cord in the router was not seated correctly, it hadn't been pushed all the way in, and the router also was screwed in on a small set of rails, a half set of rails, which meant it was kind of balancing, it was kind of hanging down with the weight of the router, and just opening the door caused it to wobble slightly, and when it wobbles, the power fell out, and when the power falls out, the whole communication to all of the servers just completely dies, and the whole of the Stockholm location just disappears off the map, and at that point Sri calls me and says, did you do something? And then we have to kind of explain, and it took a long time, if a router dies unexpectedly, I don't really understand why, because I'm not a network engineer, but it takes a long time for it to come back online, and to get everything back into a safe state, and I'm sure things are much more optimal now, and how we configure things, but back then it was kind of a hard manual task for the SRE and network team, to kind of get everything back, and check everything over, in order to be able to put everything back online, so yeah, that was a chilling moment, I'll never ever forget that. I mean the good thing is, because we used BGP Anycast, when that happened, I mean Stockholm disappeared, but other data centers would have immediately picked up the load, I guess the bad thing was, this was a long time ago, and we had hardly any data centers, so it would have been, you know, suddenly those Swedish, those middle of night Swedish web surfers, were probably being routed to Prague, or something, and the performance wasn't so great for them, yeah, yeah, that's, you know, early on, when you put these things together, you remember that, you know, that famous picture of Google's first server rack, where it's like cardboard between the layers of servers, I think early on in companies, you do a lot of stuff really fast, and then you have to go back, and make it a bit more professional, and thank goodness we have this infrastructure team, that manages what, 200 cities now, where it's a little bit better than send Simon to Stockholm. Yeah, yeah, if you were watching this as a Cloudflare customer, you need to know that they don't let me in data centers anymore, like absolutely not, and yeah, that was very, it was a very interesting, and a massive, massive learning experience, and the Stockholm one in particular, was particularly challenging, yeah, the other thing to mention now, actually, as you mentioned about the whole photograph thing, is that, and this is an example, that I, that is true, and I would literally have a piece of paper, and I would draw the rack, and write things down for myself on paper, I believe now, actually, our infrastructure team, we have actually, there's a piece of software, that essentially inventories all of your racks, so you know exactly where every server is, you know all of the cabling, and it's just, it's just a all -around slick operation, but back then, it was intensely manual. Yeah, it was, it really was all that sort of stuff, and there was a lot of flying machines to countries to get them there, and if you remember in London, we used to have that big pile of servers, there was all those servers that would just turn up in the London office, and be like, yeah, don't worry about those, send those to Lisbon tomorrow, someone will come and get them, and you know, if you haven't seen the size of servers we have, and the boxes, they're absolutely huge, there's like this gigantic box, it's nothing like a PC box, it's huge box, and occasionally, Marty would get one out, and test it in the office, and if you remember in that tiny office, the sound of that, just one of these things, you know, in full blast, was terrifying. Yeah, I wanted to ask you about something else, in 2014, we had a very well-known service provider, I won't name them, because they probably don't want us to talk about it, but they had a vulnerability, they were very worried about, and they knew that there were a whole load of their users who use Cloudflare, and they came to us, and they said, could you put something in your WAF, that would block this particular attack, just while we have time to patch, and get everybody to upgrade, and we're like, sure, no problem, and they said, well, here's what you're going to do, you're going to match this URL, and you have to match this list of IP addresses, and they gave us a bunch of ciders, and I think it was actually, ended up being about 200 individual IP addresses, and we were like, yeah, we'll do that, and it turned out, we didn't have a system for doing that, right, and so we had to hack it, and I remember, I was in Poland, and you were on a bus in London, and we got on the phone, and discussed this, what was the solution? So, yeah, you're right, like, at that time, I think your job title was programmer. It was, it was. Yeah, your job title was programmer, and you had written the first version of Cloudflare's WAF, was that the first one? The second one, the second one, so the first version of Cloudflare's WAF, just full disclosure for everybody who wants to know about, if you think the server thing was a little bit scary, the very first version of Cloudflare's WAF was, you got NGINX, which everyone was talking to, then you got another NGINX, and that NGINX was talking to Apache, and Apache was running mod security, like the classic mod security thing, and the reason there was an NGINX in the middle, was that Apache kept crashing, and it was a way of protecting our main service, and one of the first things that Matthew, when I joined Cloudflare, he gave me his list of things to do, and one of them was, write a new WAF, and so I ended up writing a new WAF, which is still in production, I think the team is slowly now migrating off of it, so we had this thing, so yes, I was kind of the WAF guy, and you had got a reputation for being the other WAF person who would write rules for customers, right? Yeah, that's true, so I had my head in my hands as you explained the technology of the first WAF, because I thought you would, the letters that were going to come out of your mouth, I thought were PHP, and they didn't, so fine, we'll move on. Okay, I mean, listen, do you want to hear some PHP? All of Cloudflare was written in PHP at the beginning. I do remember that, I think just as I joined, I remember the migration from PHP to Lua. Yeah, that was a crazy time, but yeah, okay, but I mean, Apache, ModSecurity, it was, it was, it sort of worked, but you're talking about when, you know, Cloudflare had like 5,000 customers or something, instead of 27 million domains, so it was a different world. For sure, and yeah, so the WAF that you wrote still used the ModSecurity syntax, which meant that we were able to migrate the existing functionality that we had into this newer, faster, more flexible thing, and it meant that customers also could come to us and say, hey, I run ModSecurity on my origin, or my previous provider had these rules, can you, can you create them for us in Cloudflare? And yeah, we would take those in the support team, John, I think you trained us and showed us how to do it and how to write tests for it. Yeah, so it was something we would do relatively regularly, but most of the time we were writing rules just for a specific customer, so kind of the scope was much more narrow, but yeah, when you called me about this one, we were talking about writing something that ultimately we turned on for every single customer on our network, I believe. I think we get that rule away for free. The WAF traditionally is something you get on our pro plan, starts at $20 a month, but in specific cases where there's a vulnerability that has a much wider impact, we have decided to deploy those rules to protect everyone. Yeah, that's a very good point, actually. We, for a very, very long time, ran a set of rules for every customer of Cloudflare because we felt like the threats that we were dealing with were so widespread that it was better just for the Internet, just to say, I know you should be paying for this, but we're going to protect. We didn't tell anybody, we just protected people just to make the Internet better, and there were actually some threat protections that were literally hard -coded, so when Shellshock came out, if you remember the Shellshock vulnerability, I actually did literally hard-coded protection for that, not even in the WAF, it's like literally an if statement, because that was such a scary vulnerability. So yeah, we were doing this, and then this particular one, as you say, if you remember, I made that little thing where you could create a custom rule, it would set up a test harness for you, and you would write positive and negative tests, and then this one came along, and this one was, block this URL pattern if the real IP header has one of these 200 email addresses in it, right? Sorry, 200 IP addresses. Yeah, so it was like a control endpoint for a piece of software, and that endpoint was only designed to be used behind some authentication and by a trusted set of servers, and I think a bug was found in the authentication layer, which meant that if someone could exploit it, then they could take control of this application, so it was a big vulnerability, and the team who wrote this software, I think wrote to someone at Cloudflare, and was like, do you guys have, you know, at that point we had a lot of customers, we have even more now, but if we could potentially protect all of our mutual customers for them. So yeah, what they had was a list of, I don't know, did you say 20 was about 20? No, I think it was, I think the actual number of IP addresses was 200, but we had about 20 CIDRs, right? Yeah, so we have these 20, like what are called CIDR ranges, I guess, and basically that's a shorthand way of expressing a range of addresses from starting from a number and ending at a higher number, but the logic can get quite complicated, and it's really confusing. I've always had problems with CIDR ranges, so I was very nervous for two reasons. One, I don't really understand CIDR ranges, I understand them a bit better now, having been here for seven years. Two, the WAF did not have the ability to analyze IP addresses, but what it did have was the ability to look at request headers and test the values of request headers, so Cloudflare being a proxy would always pass around the IP address of the real visitor, which meant that theoretically we could just write a rule that says look at this header, and the header is called something like X real IP or remote address. Yeah, X real IP. Yeah, something like that, and test that value against it, so that's problem one. We can now, we can actually write functionality to test the IP address by using request headers. But as a string, right? Not as a number, so it's like literally string matching. Yes, so expressing ranges is a challenge I suppose, but what you can do, you can write regular expressions to pass strings, and you can express a regular expression to match a CIDR range, but it gets really really complicated, because there are lots of different variations, and so for example, a CIDR range might be what we call a slash 24, and if you know an IP address, an IPv4 address, it has four segments or octets, and a slash 24 basically means the last octet is the range that you're trying to match, anything from 0 to 255 in the last octet. So you can kind of express that in a regular, I could figure out how to express that in a regular expression, because I'm just going to match the first three parts, and I'm going to ignore the next three, but then you can have like a slash 28, or you can have all these other numbers which are kind of like, they don't match the octets perfectly. So I sort of googled around, I did what any enterprising coder would do, I googled it, and someone has written a tool online that you can plug a CIDR range in, and it'll give you a regular expression back. It was really designed for one CIDR range, but we had like 20, so I just did that like 20 times, and then I just piped them all together into one giant regular expression, and then I wrote, went away and wrote 40 tests, like one for every single, I was so paranoid about this, and so worried about the scale at which we were planning to run this, and also the speed at which we needed to do it, and I wrote the most exhaustive tests I could come up with. And yeah, your WEFT tool had this ability to, you would write your rule, and then you would also have tests, and the tests basically were just text files which had a HTTP request in it. Yeah, HTTP request, and then there was, I remember well, there were two directories, like a 200 directory and a 403 directory, the idea was write test, write stuff that's meant to get through, write stuff that's meant to get blocked, and then we can run it to test the rules work. Yeah, so what you're always really worried about with a WEFT or any kind of security thing is what's called false positives and false negatives, and what I'm really worried about, yeah, is blocking things that we shouldn't be blocking, and not blocking the things that we should, so your tests really allow you to assert that that's not the case. So I write like 20 tests that should be allowed through, and they're all matching those IP addresses that are trusted, and then 20 that don't, and then run those. I've got it right in front of me, actually, you wrote 16 tests that were meant to 403, and 15 that were meant to 200, so yeah, you read a lot of tests to make sure you were... I was just so terrified, because obviously, like I say, I can understand regular expressions mostly, but when it came to this stuff, I was just plugging this into a tool, and who knows whether that tool is accurate or trustworthy, so that's why testing really, really helps, so that you can at least prove that even if you don't understand what you've written, or more importantly, if you don't understand what someone else has written in your team, you can kind of just put a test and go, well, they pass, or they fail, so it works, or it doesn't. Yeah, so I think we got there, and that regular expression is enormous, and it lived for quite a while. I don't think it's still in existence anymore, because eventually we did implement the ability to properly match IPs, and also to do CIDR logic, so we were able to stop using request headers, and also stop using regular expressions for CIDR matching, so that functionality is now there, so it's... The nice thing is, I pulled this up in our code system, and you wrote a really nice comment describing what you're doing, how you wrote the test, everything, it's as clear as day, you can get in there, and you can see, okay, yeah, this is exactly what's going on, and myself and Lee Holloway, technical founder, we both reviewed it, it looks like you probably wrote it in the time it took me to fly from London to Warsaw, because I remember calling you from Heathrow, like, I was almost boarding the plane, I was like, can you do this crazy thing, and you were like, yeah, I think I probably can. Yeah, I think I wrote it at the end of a day, a working day in London, and then I got on the 141 bus back home to Harrogate, where I lived at the time, and I was still doing it, we were still sending emails, and we were still fixing tests and stuff, so I do remember sitting on the top of the 141 bus going around Old Street roundabout, tapping away in the terminal, trying to get this done, so I'll always remember that moment, and I'll always remember the 141 bus for that reason. I think I'm gonna have to get this RegEx, like, engraved into, you know, an acrylic gift for you, like a sort of souvenir of Cloudflare, because it's quite large, looking at it, I mean, I wish I could publish it, but it's got a lot of IP addresses in it, but it's, yeah, it's quite something, but luckily we have better functionality these days. Yes, and I think you were saying before, we were chatting before, this functionality, the ability to kind of match a bunch of, a whole bunch of IPs, is something customers ask for a lot, and we do have ways of doing it, but managing sort of large lists of IPs, independent from your rules, is not something we've ever had, but it's actually very, very close to being launched. Right. So finally, this would be very, very easy for us to do today, and I wouldn't need to be on the 141 bus with my laptop open. Well, now you can get on a 28 tram here in Lisbon, and go up and down the hills, and write a RegEx. Oh god, yeah, I've been on the 28 tram once, and I didn't even want to be on there for tourism, so I don't think I would enjoy being on there writing. Do it now, there's nobody on it, I mean, it's the perfect time, I mean, to go on it. Well, listen, Simon, thank you for talking to me about the horrible history of that, and you know, if anyone else is watching this, who is starting a company, don't feel bad about some of the crazy stuff you're going to have to do in the beginning, and some of it's not going to feel like you're doing a great job, it's easy to judge companies from the outside, and think, wow, they must be doing everything perfectly. No, we did lots of things that were expedient, and needed to get done, and then we went over time, we replaced them, and made things better, and so, you know, that's really the story of building any company. So, Simon, thank you for being here seven years, hopefully we'll be here many years more, and I'll see you again on this show sometime. Thanks, John. Yeah.