Story Time
Join Cloudflare CTO John Graham-Cumming as he interviews Cloudflare engineers and they discuss a "war story" of a problem that needed to be solved — and how they did it.
Transcript (Beta)
All right, welcome to Story Time on Cloudflare TV. I am John Graham-Cumming, Cloudflare's CTO, and I have a guest, a new guest this week, Celso Martinho, who is the Director of Engineering in Lisbon and recently joined Cloudflare.
And I thought we would talk about who Celso is, why he joined, what we're going to do in Lisbon, all that kind of jazz.
Celso, why don't you briefly introduce yourself? Okay.
Thank you, John. Good morning, everyone. It's a real honor to be at Cloudflare TV.
So my name is Celso, as you said. I've been with Internet in Portugal for my whole life, more than 20 years now.
I was one of the lucky founders of a project called SAPO, which is kind of like the Portuguese Google or Yahoo, which eventually got acquired by Portugal Telecom.
So I did a lot of Internet technology, telco stuff along the years.
And for the last years, after I left SAPO, I've also been in the world of entrepreneurs and investments in early stage startups.
It was a fun ride at Bright Pixel and now I joined Cloudflare. All right.
So tell me about that bit. What made you decide, hey, I want to be at Cloudflare in Lisbon?
You'd been with things that were all Portuguese up until now, and suddenly you decided to work for, I mean, still in Portugal, but for an American firm.
What was the driver of that? Well, there are a lot of reasons, but first of all, and I've said this in a recent blog post that I did, I really enjoyed being close to startups and helping entrepreneurs.
It's a really fun world and I learned a lot, but I think you know this, once you do technology, once you build products, once you see the result of your work in the hands of people and customers, it's a unique feeling.
And I needed to go back to that. The more I got away from technology, the more I was thrown into thinking about investments and the investment world.
The more I started missing getting my hands dirty and doing stuff with technology and being close to build and release products.
That's really my true passion and I was missing that a lot.
So, I was thinking about creating a new startup, and to be honest, I promised myself I would never work for a big corporation again in my life, but it seems that Cloudflare is the exception to that.
When we started talking and I started the interviewing process, I found that Cloudflare has a unique culture, which is basically, in my opinion, the result of the people that work at Cloudflare.
It's got a bold mission, which is very interesting to me and resonates to me, which is changing the Internet for the better.
It's also a global company, which is interesting for me. I've never had an international career.
Although I've worked outside Portugal, I've never really been in a company that reaches the world.
It's completely technology-driven, which is also important to me.
Although Cloudflare is now big by any standard, I think it's still in a very interesting phase where I think you can still make a difference.
I mean, there's a lot of work to do.
I think Cloudflare is only in the beginning of the things that it can do for the Internet.
It reminds me of companies like Google 10 years ago.
So, I accepted the invitation and I'm really proud to join the company. I'm still absorbing a lot of information, getting to know people.
It's a big company.
I'm loving it. It's been how long? Two weeks? It's been two weeks. I'm still overwhelmed with the amount of information that I'm still processing, but it's been really good.
Everyone has been really warm at welcoming me, so I'm really proud.
And you joined during this pandemic, which now a huge percentage of Cloudflare staff have joined.
Had you joined prior to March, you would have gone to San Francisco and done an in-person orientation with the company and met people and all that sort of stuff.
And then you've had to go through it in the Zoom version of that.
What was that like? Tell us a bit about the first week. What did you do during that first week?
Yeah. So, I was wondering myself before I joined how would the onboarding process go?
Because I thought to myself, nothing can substitute being close to the teams, being in the office, especially when you enter a new job.
But it actually went pretty well. I think Cloudflare did an amazing job in creating so many content, video content.
And actually, most of the orientation sessions were live in person with someone from Cloudflare teaching you the ins and outs of their department or their division or their teams.
Everyone was really helpful in asking if we had questions, follow up materials after the video sessions.
It's funny because you, I hope I can say this, so you have these classes of onboarding employees, which I think you do every month.
So, we were class 159. 159, right? Yeah. And there was people from all over the world.
There was me from Portugal, there was people from London, the US, Singapore.
Maybe Japan, Australia? Maybe, yes. So, Europe, Asia, US, a lot of people, which is very interesting.
Yeah. Those class things are kind of fun because you actually get a custom email address, which you can email your class.
And we've actually started to have class reunions as part of the fact that the pandemic has kept us all separate.
As you say, those classes cut across departments and locations around the world.
We started to organize people getting back together again.
So, having Zoom calls and having a virtual coffee together and stuff like that.
That's a good idea. It's also interesting that people like you and Michelle and Matthew, which are really busy persons in the organization, take the time to speak to the classes.
You actually take a lot of time in presenting your vision for the company and answering questions, which is unique.
I haven't seen something like this in other good corporations, big companies that I've known or worked for.
So, it's interesting. Also, I got my Cold Flare socks already. All right.
So, now you're onto the really important stuff, which is the socks. All right, good.
I'm glad you got socks. I didn't know those socks existed until I flew to Australia.
It was the last business trip I did before the COVID hit. I went to Australia.
I went to a meetup with one of our customers. Our sales team had put out Cloudflare socks.
I was like, what are these? I had to fly all the way to Australia to get them.
So, you're really privileged. You got them in Lisbon. Took a week.
So, there you go. There you go. Took a week. We got you a laptop with an escape key as well, right?
Yes. I actually have two laptops. The first one didn't have the escape key, but the team fixed that immediately.
And I got a new one with an escape key.
I really need the escape key. Yeah, me too. I have a laptop which doesn't have an escape key.
This is actually a replacement laptop for a laptop that I broke.
I dropped it on the ground and smashed it. I also think you need some new keys.
Yes. I know that my keys are worn out due to my keyboard, but I did learn to touch type a really long time ago.
So, it doesn't actually matter that those keys have got no key caps left on them.
And I'm definitely not the worst person in the company.
The worst person in the company is Dane, who's in our Austin office.
His keyboard actually has some where the key cap is missing and he's just hitting the membrane underneath.
Okay. Yes. And I also have a British keyboard, but I'm using the Portuguese layout.
So, I just close my eyes and type away. Are you using the H Caesar layout?
You're not using that crazy thing, are you? No, not that one.
It's the QWERTY layout, but the Portuguese layout. Yeah. Okay. Not the old typewriter Portuguese layout.
I saw that. If no one knows what that is, if you just Google H Caesar, C-E-S-A-R, and find that layout, it's actually, if you've grown up on QWERTY, or I think even AZERTY, if you're in France or something, and you look at the Portuguese layout, it's the stuff of nightmares.
No key is where you want it to be if you've done QWERTY.
And I'll tell you a secret. I had classes in high school where I would practice in a classic typewriter using the H Caesar layout.
So, yeah. But all that is in the past. Then I had the ZX Spectrum and I forgot about it.
Ah, yeah. So, your keyboard has a pound sign on it? Yeah, it has, yes.
Thank you. Because I have a US keyboard, because when I started doing stuff, we were using, well, actually, VT100 terminals, which were a US layout, right?
And then some microsystems with a US layout, and I'm so used to it. If dollar isn't where I think it is, I'm done.
So, like dollar and escape are the two things I really need to know where they are on a keyboard.
And if I see a pound sign, I'm like, oh, my God, I know.
Yeah, I just don't look at the keyboard anymore.
Yeah, yeah. I can't either. I don't look at the keyboard either. And it needs to be the US layout.
So, this stuff is really, it's one of those things that's funny, right?
When you see, you know, if you look at high-end keyboards, people build their own keyboards with mechanical buttons and lights and all this kind of stuff, and they're really expensive.
I think sometimes people think that's really frivolous, but you don't realize how much of your time you spend typing.
And the last thing you want is the typing to get in the way of what you're doing.
And so, this stuff matters.
So, when you told me you didn't have an escape key, believe me, I was like, IT department?
You took it personal. He needs another laptop right now.
So, right. So, we've got the laptop sorted out. You get orientation done for your first week.
I noticed that you decided that you were going to move your personal blog onto Cloudflare Workers, which is our service platform.
And that was a good thing to do for two reasons.
One, of course, it gives you experience with the product.
And also, what we really think is that people need to use our product to really understand it.
We love when people, you know, use the product before they join often, like in the interview process.
So, tell me about, I haven't done that with my blog, which is still on Blogger.
Yes, I know. I have to get it out of Google, and I assume they're going to send me an XML document or something.
Anyway, at some point. You moved yours over to Workers using Jekyll, I think, right?
Yeah. So, tell us about that experience now of actually, you know, you're on board and you decide, okay, I'm actually going to use the product.
Yeah. So, it's really easy, especially if you're into the technology world.
So, Cloudflare is this workers serverless platform, which is really interesting.
You're basically using a customized version of the V8.
You're doing clever stuff with the isolates.
So, there's a full blog post at Cloudflare blog explaining that. And I think there's a lot of potential with the worker.
Actually, Cloudflare today uses the workers internally for a lot of the stuff we do, which is also interesting, you know, eat your own dog food.
Mm-hmm. And so, I decided to educate myself and get used to especially the tooling around the workers.
And one of the things I thought it would be easy for me to start with was to migrate my blog to the workers.
I've seen that Sam already done that. I think he was using Yugo as his static blog file generator.
I'm using Jekyll and I have everything hosted at GitHub pages.
So, I started doing it during the weekend and it actually took me about one hour to do everything and then I wrote the blog post that took a bit longer.
And so, there's this tool called Brangler, which is a clear tool that Cloudflare provides, which basically builds everything locally, uploads the resulting JavaScript and assets to the workers at Cloudflare.
You can do troubleshooting and debugging locally if you want to.
And I glued everything together with GitHub Actions.
So, GitHub Actions is kind of a CI, CD solution that Cloudflare gives for free, mostly for free to anyone at GitHub.
And you can use that to automate the workflow of doing something when you, for instance, push a new blog post to your repo.
And it was really easy. I mean, there's nothing much to say other than this.
You can do it in one hour. You can automate everything. And as soon as you push a new file to your Jekyll repo, it will be online in a couple of seconds at Cloudflare.
And once it's in Cloudflare, it also benefits from all the features that Cloudflare provides in our Edge network.
So, that includes caching, HTTPS, HTTP3 support, IPv6, etc., etc., which is really good.
That's fantastic. Well, good.
Thank you for trying out the product. I hope you fed back ideas to the workers team about what could be made smoother about it, because we really want developer experience to be seamless.
You can develop on your machine and then push to the entire world.
Yeah. I have a feeling that I will be working with the workers a lot more in the future.
Great. That's what we like. So, all right. So, now you're here in Lisbon.
We've got about, I think about 45 people, maybe 50 at this point in Lisbon, and an engineering director.
So, your role is going to be a bit split, right?
So, what I think people may not know about Cloudflare is that Cloudflare has sort of three different technical teams, and there's three people who appear.
So, there's me, there's a guy called Usman Muzaffar, and there's a guy called Dane Connect.
We're all in different places. So, I'm here. Dane is in Austin, and Usman is in the San Francisco area.
And Usman runs what we usually call the engineering team, which is doing product engineering.
So, all the new product features that are coming out, that's coming out of that team.
He works really closely with Jen Taylor, who's the head of product.
And so, the two of them, they go building that.
And then Dane in Austin runs a team called Emerging Technology and Innovation, which works on the next big bets in some ways.
So, Cloudflare for Teams came out of there in January.
We bought a company called S2 Systems in Kirkland, Washington.
He's working on getting that integrated. And then there's me.
And I've got – well, you work for me. And also, I've got a little team of principal engineers in different locations around the world and the academic research group, the research group, which does stuff mostly with universities and other bodies.
So, we're all peers, the three of us. But you're a bit split between me and Usman, right?
How's that going to work out? Well, you tell me.
It's been working great. I mean, I've been – I'm still in the early days, to be honest.
But I've been talking to Usman. I'm actually now doing a project that involves teams from London, Austin, and Lisbon.
It was my first project. And I finally know everyone in the team.
It took me a while. I now have access to all the systems I need.
But it's one of the things I find interesting at Cloudflare. The fact that in many occasions, when you need to deliver something or build a product, you actually need to get teams, you actually need to get resources and help from various offices and teams across the globe.
I enjoy that. Of course, there are challenges to that.
For instance, the time zones and the tools you use to communicate.
And that's especially challenging now during the pandemic, where you don't have a physical office.
But I think it's been working. You know, it's been above my expectations.
It's been working really good. If you remember when we opened the office in Lisbon, one of the reasons we cited for choosing Portugal, apart from all the other great reasons, was the time zone.
And you now probably feel it well, why we took that so seriously, which is because you end up probably working with maybe some folks in San Francisco, maybe some folks in Austin, definitely people in London, ultimately maybe in Singapore.
Shifting an hour, Barcelona looks lovely, but it's an hour later, and suddenly things get really messed up.
So time zone, don't change Portugal, stay at the same time zone. It's really great.
Yeah. I mean, you know more than me, but I think it's a great time zone, the Lisbon time zone.
Yeah. Because you've got London and Europe on the same page.
And you can start working in the morning, but you can also focus a lot more in the morning.
And then at around midnoon or in the afternoon, the US wakes up and there's all these coordination and meetings that you need to do with the teams in there.
But I kind of like the division of time. I think it works good for us.
I think it works better for us than for the US teams. I might be wrong, but...
Maybe. I mean, US team obviously has some overlap with Asia, and they might say some of the things about the Singapore office.
I know from my perspective that today, first meeting I had was with our Singapore office.
And that's at the end of their day, do some stuff there.
Then I had that exactly as you described, a period where you can do email, do thoughtful work and all that sort of stuff.
And then the US is going to wake up later on.
So that's been really important. Just to go back to, you said you're working on something with teams around the world.
So we can't announce what that product is, but it's worth saying that Cloudflare really values shipping products.
We don't sort of pre -announce things and say, oh, maybe we'll do this.
Like here's the thing. And it's shipped and it's real. And you got thrown in the deep end because we have a project which is being worked on in Lisbon for birthday week.
And anyone who's followed Cloudflare will know that September 27th is what we consider to be the birthday of Cloudflare.
That's the day that Cloudflare was revealed to public at TechCrunch Disrupt 10 years ago now, 2010.
And during birthday week, we typically release something every day that usually is some sort of a give back to the Internet or really useful free offering.
And one of those things is something you're working on.
So what's it been like to be like, hey, welcome to Cloudflare.
You have six weeks to ship a new product during birthday week, no pressure.
And also review it with the CEO. Yeah. Well, it's been both stressful, to be honest, because I was on my third onboarding day and I just got an email saying, Celso is now in charge of engineering for this project, which we have to launch in three weeks.
So panic. But it's also kind of easy. I shouldn't say this, but it's kind of easy because everyone has been so helpful and thoughtful and everyone made sure that I integrated well with the team.
So in that sense, I feel that I had an easy job joining the team.
Of course, I know that there's a lot of stress surrounding launches at birthday week.
It's that week of the year for Cloudflare.
And I know that the project will be stressful to deliver.
There's a lot of challenges ahead. But I feel comfortable knowing that I have an amazing team working with us, supporting people like me that joined Cloudflare two weeks ago.
So it's a mix of both. It's fun because you've actually got one of my principal engineers, Sri Rao, who has actually been at Cloudflare longer than me.
He was the first ever SRE network person at Cloudflare. He literally, I don't think he slept for a few years and he literally wired everything up right in the beginning.
So he knows how everything's built around here. And he knows everyone, which is especially important for me.
Yeah, absolutely. He knows everybody everywhere.
And he's based over in Austin. So you're obviously dealing with the Austin office.
Yeah. So in this project, there's people from Lisbon.
There's people from London, Austin, and San Francisco. Yeah. And that's not actually that unusual for Cloudflare, especially when it cuts across UI and in your case, database stuff and then backend stuff.
There are different teams which will contribute to that kind of thing.
Yeah. All right. So that's great.
So birthday week is coming, but you foolishly said to me, this isn't enough. You need to give me more work to do, which I think you said that anyway, or maybe I dreamt that.
Did I say that? I think you said something along those lines. We're also hiring in Lisbon, right?
So that's the other thing. We already have a lot of openings here.
And so there's going to be opportunities for engineers and security people.
Yeah. So we're going to start building, expanding the team actually in Lisbon, but especially building the team that I will be managing more directly.
And hopefully we'll have our plan finished this week or early next week.
And I think everyone can expect a few open positions, new positions to open at Cloudflare in Lisbon.
Yeah. It's already quite a lot. If you look on Cloudflare.com slash careers and then filter down to Lisbon, you'll find that there definitely are quite a few openings already here in engineering, et cetera.
But yes, I expect that next week or maybe the week after, we'll have a lot of additional information out there about these new openings.
So yeah, look out for that. Yeah. Yes. There's a lot of opportunities at Cloudflare right now.
Now, just before we wrap this up, I received a text message while, because you said you were going to be on the show from somebody, I won't name the somebody, but somebody said that you once deleted all of sapo.pt.
Is that right? I'm going to kill that person. Anyway. I'm going to kill that.
What I really wanted to say is, can you not do that at Cloudflare? I think we hope you have access, but possibly if you're like me, you don't have access to actually do damage.
No, no. I won't do it at Cloudflare. Don't worry. Anyway, there's a word for that.
It's, I think I can say it's not a problem, but I've been known to do, we have, how many minutes do we have?
We have 30 minutes. 30 minutes. Damn it. So there's this, someone coined, actually it was a friend of mine called Pedro Melo, and he coined this term back in the early 2000s called Celsadas.
Celsadas is basically small mistakes that Celso does.
And I've done quite a few during my career, but, and some of them were quite serious, but in the end, everything went okay-ish.
And I actually deleted the sapo source code. I don't know what year this was, but I actually deleted everything.
And I remember myself and the team trying to recover part of the source code from the RAM memory in the production servers as they were serving sapo to actual end users.
It's not come up in the background check on you. I don't, I think I forgot to mention this.
I might've forgot. Yeah. You'll be glad to know that we have replication and we have backups.
Yeah. We didn't have those at the time to be completely honest.
Backups were basically zip files in our laptops and stuff like this.
You have to remember that this was the late nineties, early 2000s. So, CVS and stuff like that.
Anyway, if you want to, we can do a full Cloudflare TV session around Celsadas.
I have so many stories to tell you. You could probably do a Cloudflare TV show, which is called Celsadas, where you talk about yours, and then you get other people to admit their terrible mistakes.
I think we already have a CFTV show, which is about how you can make terrible mistakes.
Maybe you can just go on it.
That sounds interesting. I'm sure there are a couple of, how could we call them?
John Graham and mistakes. I think the worst thing I've ever seen that I did, and it wasn't really my doing, which was I, we're going to run out of time for you to tell the whole story, but I was working for a networking company and I installed a patch to a network driver live on a network at a major airline, at a major airport, because there was a problem and it needed to be done, and all of the check-in desks failed, and we were in the control room where we could watch all the people trying to do check-ins realize that their machines were not working, and that was terrifying.
If it doesn't kill you, it makes you better. I think that's true, although sometimes I'm not sure it is.
We're out of time.