Join John Graham-Cumming, Cloudflare CTO, as he interviews a Cloudflare Engineer or a Solutions Engineer and they discuss a "war story" of a problem that needed to be solved and how they did it.
This week's guest: Simon Moore, Principal Supportability Engineer at Cloudflare.
All right, well hello, welcome to Storytime on Cloudflare TV. I'm John Graham-Cumming and my guest today is Jen Langdon, who is Engineering Director in our London office, well was in our London office.
Jen, welcome. Thank you very much, thanks for having me John.
And I saw the tweet went out, it said you were Engineering Director-Security, but that's not true.
So why don't you just give us, you know, if you, unless it's a surprise to you to discover that you have more than just security under you, but what, tell me the scope of what you do in London?
Absolutely, so I'm responsible, I am responsible for a group of our security products at Cloudflare, including the firewall, the WAF, our frontline platform, that is kind of the HTTP entry point for, you know, requests coming into Cloudflare about 18 million a second.
But I do also manage our London UI team that's responsible for, you know, the interface of how our customers actually deal with our products, as well as kind of the EMEA delivery team.
That's right, quite a lot. And in fact, what's interesting is that I think one of the things that surprises people sometimes about Cloudflare is that you're obviously we're a San Francisco headquartered company, that's where the initial stuff was done, but a lot of our core products are actually in London.
So you mentioned this WAF, this firewall, there's DDoS, there's the user interface that everybody uses, and there's FL, frontline, which is actually the core serving infrastructure of Cloudflare, like the thing that actually handles those 25 million websites.
So that's all done out of London, pretty much.
Yeah, exactly. Exactly. So I manage a bunch of those teams.
And yeah, as well as like I said, also a delivery team that kind of facilitates or rather enhances how quickly we do product development.
Right, exactly. The delivery function is very important, because we're always shipping at breakneck speed.
And this week is no exception, right, being birthday week, like things coming out all the time.
What's the London team's involvement been with birthday week?
It's been pretty extensive, actually, as you'd expect, we sort of build up to birthday week, the London UI team, as I already said, is responsible for that interface.
So all of these great things that we're building in in birthday week, that team has been involved with, you know, for several weeks in the planning for months, essentially.
And of course, as you said, anything that frankly hits the Cloudflare edge, the frontline team, the FL team were also involved in so a number of the London teams have been involved in birthday week.
And it's been it's been pretty exciting.
It's also been great to be able to deliver that and do that.
You know, in the current circumstance, and yeah, hit those deadlines.
Yeah, I thought it might be worth us talking about the current circumstances.
We are now six months into that lovely office, which is behind you. People don't know Cloudflare's office is in County Hall, the old County Hall building in London, on the sixth floor, view out over the Thames and everything, a really wonderful split space that we built out.
And then we closed it in March. And so six months, I think it's worth, you know, now we have a bit of six months of perspective.
You know, let's talk about that. What was it like at the beginning, when the office got shut down?
Let's sort of walk through how we dealt with stuff. Yeah, absolutely.
I mean, I think, gosh, it seems like a long time ago. And I think, initially, for people, it was such a, that space was was so integral to kind of the way that we all work together.
Okay. And I think when that shut down, I think initially, there's shock, I think there's shock for everybody.
And, you know, I can talk for me sort of personally, what I didn't realize is actually how much I relied on as a manager observability.
So I would walk around the Cloudflare office 2030 times a day, I would be able to visually, at least in some capacity, understand how my team were doing, how my colleagues were doing, are you having a bad day, etc.
And then and then very, very abruptly, that's that gets taken away.
And for me, as an as a manager, honestly, I needed to sort of adapt myself and learn a new skill of how do you actually manage that remotely?
And how do you how do you sort of keep that contact when you're kind of you're kind of doing that, you know, within your day, it's sort of something that's that's constantly surrounding you.
So I think initially, you know, we closed it, we did as well as I think any anybody could, but I think there was definitely, you know, the initial part of shock of, oh, that incredible space has gone, that incredible interaction has gone, how do we adapt?
I, you know, if you remember, you know, I used to work in the London office, too.
And I used to wander around as well. And I missed that, because it was the random interactions that were so important, you sort of see a conversation that was going on, or you would bump into someone, they'd ask you something.
And I think that's been, for me, that's one of the things I've really missed.
And I, it's difficult to schedule that you can't schedule randomness so easily.
And I think we've tried a few things. And the other thing I was interested in was, I actually looked back to see when we closed the London office, I knew it was six months ago.
And if you remember, initially, we didn't actually close it on the very beginning of March, we told people they could optionally work from home, if they felt like it, and then, boom, you know, Boris shut everything.
So there was that weird period where I think we all were living in the fantasy that maybe this was going to go away.
I think, I think that's, I think that's right.
I think that's definitely where some of the shock sort of came from, is that we sort of thought this, this was a temporary thing.
And then, you know, it turns out very quickly that it wasn't.
And we had to deploy very rapidly to support people at home.
And I'm not talking just about, you know, connecting with them, I'm talking about setting them up physically at home, making sure they have everything that they need.
It happened so, so quickly. And we had to, we had to really scramble for that.
So we need to talk about that, because we had the technical means for people to be at home, because we have this thing called Cloudflare for Teams, we access, we can access everything from anywhere, we had the ability to do that.
So we didn't have the problem of like, you had to be in the office to do something.
But that doesn't mean it was easy for everybody. So what were the sorts of problems that were encountered at the beginning?
What did we have to support people with?
Yeah, so I think that's, I think that's actually a great point. Because from a technology perspective, we were kind of built to do this.
But, you know, we were co located.
And actually, that was, you know, it was a core part of sort of our culture.
And it was kind of how we ran. So the things that we need to support people with were, you know, frankly, people that were set up in homes that didn't facilitate, you know, homeworking, because we didn't do it.
How do you get people?
There's logistical things of how do we get people equipment when they need it?
You know, there are people who lived in shared housing, so they don't actually have a quiet place to work.
Yeah, these are the sort of things that we initially faced.
And then I think there's the there's a practical stuff, Jonathan, I think there's the the the other side of it, which is people are living on their own, for example, is suddenly going into lockdown.
Okay, so you have through our technology, you have everything you need to work.
Through our great IT logistics team, you have the equipment to work, but you don't have the shock of the isolation.
Right. And, you know, so you're kind of we were kind of dealing with the practical and then also making sure we looked after our people.
Yeah, we definitely saw that right with people who that they you know, I mean, I saw it here in Lisbon, we have a much smaller team there, you know, the people who are suddenly, they're on their own all the time.
And especially that initial period where you were not allowed to go out except to do grocery shopping.
That was really quite something.
I think one of the things that really struck me at the beginning was the variety of situations that people live in, right.
So, you know, it's like, some people are alone, as you said, some people in shared housing, suddenly, everyone's been sent home, and maybe they had one small dining room table, and they're all trying to do Zoom calls on it.
Many people have children, those children were no longer in school.
I mean, I remember many very early Zooms, where children would turn up.
And very early on, I was just like, we're just going to roll with it, right, we're just going to recognize that people are not in an ideal situation.
And if you know, they actually do one last night where I said someone Oh, is that one of your kids leaving the room?
And she said, Actually, that was my husband.
I think you've touched on a really important point.
I mean, for me, I am I am a parent, I've got I've got three sort of young children.
And I think one of the things that this this period has actually taught taught me, in a way that I don't think we would have experienced being in the office is understanding what is going on in our colleagues lives.
So I have a three year old son, he has made numerous appearances, indeed, with YouTube, the dinosaur, I remember the dinosaur.
It was very, exactly.
And the level of empathy that you have, you know, for colleagues, I think, like, I've been really humbled by it, to be honest, you know, I've had to, on occasion, abandon a call because he's not okay.
So you go from being a full time employee, you know, in an amazing office, you know, kind of, you know, all day, Monday to Friday, and then suddenly, you are, you know, engineering director at Cloudflare.
Yeah, homeschooling your twins, you have a three year old toddler, and utterly overwhelming, but the support of colleagues, I think, is what makes it all manageable.
And I think the interesting thing you say there about utterly overwhelming, I think that is something that all of the employees felt in different ways.
So the parents would be overwhelmed because of the children, the people on their own, there's an overwhelming, you know, isolation, the people are sharing a place, they're overwhelmed by the competing things.
I think that was, to me, at least, that was one of the initial things was like, there's all this stuff we have to deal with.
And I'm, and I'm trying to do my job.
That was, you know, the beginning, that was, that was really tough. Totally. And I think to talk about one of the things that you mentioned earlier, one of the things I found really very challenging was those walkarounds the office that we used to do, I kind of didn't appreciate the fact that I would do those between meetings.
So those opportunities I would have between meetings, to get to see, as you say, either random people or checking on my team on a desk kind of walk by, trying to find that time in that initial period was so hard.
And feeling like you're kind of letting either the fact that you haven't done your kids homework, or the fact that you haven't checked in on somebody that you know, you should have, you know, made it very challenging.
One of the things I tried to do was, I tried to be as open as I could be with with with with the team and sort of say, look, you know, I'm struggling with this.
Like, you know, we're all struggling with this. And actually, I think that's something that just made the whole experience a lot easier.
Well, I think that empathy thing is enormously important and recognizing that, you know, people are trying to do a lot at the same time.
It's funny that you mentioned the between meetings thing there, because you may or may not remember this, but when Sylvia joined, so Sylvia is my executive assistant, and I hadn't had one before at Cloudflare, she went through my calendar, and she, I have a huge number of meetings, she organized them all to be in one conference room, because that would be more efficient.
And I had to say to her, Sylvia, the only time I see other people and get any exercise is when I walk between meeting rooms.
And what I actually need you to do is make them every other meeting needs to be as far away from the meeting, the next meeting room as possible.
And that actually was real, I really asked her to do that.
And I think that the other thing that happened was that we, you know, as soon as we're into lockdown, there's no change.
And that's the thing that really gets to me, it's like, I'll be on zoom after zoom after hangout after phone call, it's just deadly.
And it was interesting, we discussed this at the Cloudflare management meeting a few weeks ago.
And Matthew said, Matthew, the CEO, said one of the things he had started doing was alternating having his camera on and off as a way of just not constantly being sort of presenting himself.
Which I think is interesting that, you know, people are people are at every level are trying to deal with that, that side of it as well, which is I find enormously tiring.
She was like, there's no break. It's just like, go, go, go.
Yeah, absolutely. I think that's I think that's definitely right. And I think one of the things I've said to members of my team is that because because, because of our nature, because of our culture, we're used to interacting together.
And so when we were remote, everybody had their video on, you had it on whether your toddler was there, whether your mother was there, it didn't really matter.
And one of the things I tried to encourage my team to do is, is it's very stressful.
If you want to switch your camera off. Yes. It's, you know, it's totally fine.
So I think it's very interesting. And I think it's good advice. I think it is exhausting.
And it's important, I think, to adapt as you need to in these times.
Yeah, I mean, just imagine if you're if you're daily life, and I suppose people in the beauty business, this is real, look at them have to look at themselves all the time.
You get to look in a mirror all the time. And that's what you're doing with zoom.
And it's like, most people don't want to look at themselves.
It's, it's hard. So that's something but what about, practically speaking, are there things that you try to encourage those random conversations or, you know, stimulate those things?
Absolutely. I mean, I think that to come back to the point of where I felt like I'd lost kind of part of my armour as a manager, I didn't realise, you know, how much I relied on this observability.
And we tried a number of different things.
We have remote tea and coffee times. And we have a lot more sort of interactions.
I've also taken to reaching out to people that might not be in your reporting line, because frankly, they don't have sort of a local manager.
So I think people have adapted really, really well. And we've also come up with some great stuff that we didn't do before.
So the way we work at Cloudflare generally is we have engineering, under an engineering director, you have a series of engineering managers, and they will manage teams.
And usually in the morning, they will go off and they will do a stand up.
And once a week, you would get to get I would get together my engineering managers for a staff meeting.
I work great when you're walking around the office, but a week is a long time remotely.
It's a really, so actually, every morning at 930, my entire staff group get together.
It's not always work related is literally just how you doing?
Yeah, what's going on? What do you like? What do you need it? We don't always talk about work, but it's that initial kind of starting the day, that connection.
And I think even when we go back to the office, we'll keep that.
So I think that you know that there are practices that we sort of discovered, I think it worked very, very well.
And otherwise, it's about we have teams that have open zooms or hangouts all day.
And that works really well for them. But it's a lot more sort of touch points.
And to your point about sort of calendar and stuff, I've had to really change the way that I work.
I had to really sort of make spaces to actually make sure I'm checking in with people, making sure I'm doing my skip levels.
And that was just time that you would get going around the office that you don't get.
Right. One of the things I've noticed is that one of the problems of this remote environment is that what would have been a five-minute conversation, 10-minute conversation, is now a scheduled thing.
And it's now 30 minutes that are blocked out on the calendar.
And you suddenly, you sort of quantized time in a way that was really less efficient than it was before.
You could have three of those five-minute conversations around coffee, and someone else could join you.
Soon as you schedule it, you're in just a different world of like, oh, it's a scheduled meeting.
And especially when it's me, it's like, oh, it's a scheduled meeting with the CTO.
And you must feel this.
The engineering director has a serious position. Oh, why does Jim want to talk to us?
And I think that is a thing that's difficult. I think definitely so.
But I think that's one of the things I've been quite cognizant of is there may be people within the team, within my engineering managers teams, who wouldn't think anything if I came up and stood by the desk and said, how's that going?
They wouldn't think anything of that. But yeah, you're right. It's a scheduled meeting with them.
So I've tried to do a bit more of that, a bit more general interaction to sort of say that it's not formal when you do that.
But it is hard.
It is hard, right? Especially because in that in-person thing, your demeanor as you approach the person, all that sort of thing, tells a story, whereas Zoom link tells a different one.
Absolutely right. You're approaching sort of smiling or being sort of jovial.
You know, that's OK. That didn't go the way we wanted to.
It's very different. I completely agree. I think one of the things that I was thinking about, though, John, is I was trying to think about how we've adapted and why we've adapted quite so well in this circumstance.
And I think, honestly, I was thinking back to the way that Cloudflare deals with things when things aren't going our way.
So what I mean by that specifically is working in security, working at Cloudflare generally, we're a services business.
We have 25 million properties.
When something goes wrong, it's stressful for the people involved.
It's stressful for our customers. It's stressful for anybody. But the way that we actually managed that when we were in the office was that culture of collective responsibility, calm.
Let's deal with the issue, and then we'll learn from it and we'll move on.
I think that's really transferable to now, like understanding the stress of our colleagues, our peers, and supporting them.
I mean, has it ever been more stressful? I'm not sure. But I was thinking of how transferable that actually is.
That's a very good point. I hadn't really thought about it so clearly, but you're right.
We've always had that collective sense and also lack of blame and things like that, which means that it is a sort of resilient culture.
Because if something happens, then you're like, OK, how do we deal with the situation?
How do we solve it together? Who needs to be involved?
Hearing from different people. I think your eye has played very, very well. I think also the fact that we explicitly had empathy as a feature of how we work, because Matthew has emphasised that many times, just like figure it out, be empathic to the situations people are in.
Yeah, I mean, for me, the resilience thing is actually huge.
Because to your point, we have a plethora of different situations that our employees are in.
And I think back to, it doesn't matter whether we're in the office or not, how do we do what we do at the pace we want to do it at, which is generally fast?
Do it safe for our customers and look after our people. And I think looking at how different people have dealt with this situation and being empathetic irrespective of what that is, I think that's what's given us some leverage here, which has made it as good as we can in the current circumstances.
And now we're looking probably to months more of this, right? I mean, the reality is until there's a vaccine, until there's rapid testing, it's going to be very difficult for people to go back into the office en masse.
And so are we settled into it?
Or are there still things we should be adjusting, do you think? I think we are settled in the sense of, you know, Cloudflare, even prior to COVID, has obviously a variety of locations.
So we're kind of used to communicating, you know, online via chat, and we actually do that very well.
And that wasn't the case for kind of our co-located teams.
But I think we have adapted in that sense. I think we have to realise that circumstances change, autumn, winter, and we still don't know what the current situation is.
I don't think in terms of what we need to do, that we need to adapt.
I think people are coming to terms with the fact that it's going to be sort of several months.
And I think we're pretty well set up now. The thing that I think is interesting, and I think the thing that we need to continue to work at, and we've done a great job of is, for those that don't know, when you're on board Cloudflare prior to COVID, you would on board in San Francisco, a huge investment.
Everybody would spend, you know, a week, you would learn everything about everything you could about the company, you would meet your colleagues.
Suddenly, abruptly, we had to put that online.
So I have a series of team members, everybody does at Cloudflare, who I've never met, and they've never met any of their colleagues.
I think that's the thing that we, and we have several classes or cohorts, as we call them, that have gone through that process, and successfully so.
I think that that's something that we're going to, we don't need to adapt, we just need to, I think, you know, invest in the fact that they got that online, you know, out of admiration, frankly.
Well, that was, it seems to me you bring that one up, because you're in Lisbon, where are well over 50% of the office has never been to the office.
And actually, I think over 50% got interviewed online and never got interviewed in person.
Yeah. And so that was like a big change.
And I agree the, the, you know, we had this orientation thing in San Francisco, which was a week or two, you know, to get people, you know, really understanding how everything works from the technical perspective, and the legal and, you know, Matthew would speak and Michelle would speak and all those sorts of things.
And we flipped that to online, which has been, which has been very interesting.
The interesting thing is that this is not all bad. So it was a big commitment for people to go a week or two, especially if they had family, to San Francisco, and they were dealing with jet lag and visas and all sorts of stresses.
And now, in fact, they're doing it from a place where they are not jet lagged, and they are not sleeping in a hotel, maybe unfamiliar and things like that.
So that's actually, I think, in some ways, levels of playing field.
And I really feel this, and I don't know if you do because your boss is in the US, right?
So you're dealing with someone who's remote.
And I found that everybody being on a video call is actually good sometimes.
So there are some meetings where I used to be like the one person who was dialing in, and you've got people sitting in a room, and they can all interrupt each other.
And you're there kind of like, hey, I want to say something.
And I don't know, I think in some ways, that's been actually great. As much as I find it exhausting, it has actually smoothed some things.
I think that's a really interesting point.
Up until very recently, I do now, up until very recently, all of my peers were in San Francisco.
And I definitely experienced exactly what you're talking about.
You're kind of the only person sitting in what was one of our lovely conference rooms, with a slight delay, kind of trying to you know, and that has, it's leveled the playing field, in a sense.
And so I think that definitely has helped.
Yeah. But now your local buddy is not that local. He's in Lisbon, right?
So still, it's still pretty local. You know, it's still pretty local.
So I've got, I've got, yeah, I've got a European buddy now. I'm excited about it.
Very good. Very good. All right. Okay. So, you know, my sense is we're going to be, we're likely to be in this working from home, or I heard somebody describe it as living at work situation for many months.
Because of the way, you know, just look at the reality of the things.
And we're going to keep growing. I mean, Carfair is still growing at an incredible rate.
We have not slowed down in terms of software releases or anything, right?
I don't think we've seen, you know, I think our productivity pretty much stayed up there.
Yeah, I think that's, I think that's absolutely right.
And I think that also talks a little bit about resilience that I was sort of saying, that's not to say that individuals, of course, or many individuals didn't sort of struggle.
But what I was really impressed with was sort of how we banded around, actually, to support that and realise that, you know, we have a job to do.
And actually, our job is more in demand, actually, than it ever has been.
So no, software releases did not slow down. Birthday week is a brilliant example of, you know, us continuing to sort of do that.
And it's about how we, I think, continue to sort of invest in the current scenario.
You know, making sure we're onboarding people, as I said, you know, well, which we are.
And that will facilitate our growth, which I'm sure is going to continue.
Yeah. Yeah, absolutely. I mean, just looking back at your time at Cloud, because I think it's been about 18 months, something like that now.
Yeah. What was, you know, the pandemic obviously came along sort of a year into it.
But what was probably the big surprises for you coming from outside into Cloud?
Because I always ask people this, because, you know, I'm the frog that's been in the slowly boiling water, right?
So I've seen Cloudflare grow from nothing. But what surprised you coming in from a different organisation?
I think it's, I mean, it's probably a little cliche.
But I spent the last 15 years, either side of the Atlantic, in startup or scale ups.
And I knew when I joined Cloudflare, obviously, about about scale, and, you know, all of that sort of stuff.
I knew how big the company was, what you don't realise when you first get to Cloudflare is the sheer breadth of what we build.
And so I, you know, bit like you were talking about the frog in the boiling water, my previous company, I was CEO for six years, other companies, I came in fairly early on the ground.
So I kind of knew everything. And suddenly, for the first time in 15 years, I was coming into a company where, you know, two months in, I'm still going, oh, what's that?
Oh, what's that? And I do a series of orientation classes now for EMEA on products and engineering.
And one of the things I try to say to all of the cohorts is, I remember sitting there probably on week four and going, why don't I have this?
Like, like, how do I not know this?
Right. And, and, and one of the first things I say to the orientation classes, give yourself a break.
This is going to take you some time. This is, this is a big place.
You know, and I kind of knew it, but until you sort of, until you land in it, you don't actually realise how quickly we build things, the breadth of what we build, you know, and, and the level of knowledge that you need to have, but that's okay.
Because the person that's hardest on you is actually yourself, because everybody else is going, well, yes, of course, of course, of course, you don't know this yet.
Yeah. I totally agree with that. And then the sort of the opposite has happened to me, right?
So I started when it was really, really small. So I knew everything was going on as we grew and grew and grew.
And now I'm in the same situation, which is, I often say to people, you better assume I don't know the details of what you're doing.
Yeah. Because the engineering organisation is huge and globally distributed.
And it's like, I try to keep up, but there's like, at some point, no, you can't keep up.
And so it's natural that people don't know things.
And also, I mean, the higher you up in an organisation, you know, the sort of in some sense, the less you know, and the one thing I need from people is that they actually tell me stuff.
Yeah, please educate me, because I don't know, and especially if you could, the worst case is that I had a great email about this yesterday, somebody sent me an email with like, we need you to make a decision about X.
And I was like, I don't know anything about X. I can't reply to this email with a decision, I actually need to sit down with you and three other people.
And I'll help make the decision.
So exactly. And I think that's, you know, either way around, whether you're just entering Cloudflare, or you've been here for a long time, like yourself.
And it's, it's, I think what's important is knowing that it's, it's encouraged, it's fine to say, I don't know, I'm sorry, I just don't know.
I don't know what you're talking about.
Can you can you sit down and explain? And I think whichever end of the spectrum you're on, that's an important, it's an important value.
Yeah, yeah. Well, we're getting close to the end of this, but I can't let you go without asking you about that lovely day when we took down all of Cloudflare.
You've been with Cloudflare for about, was it three or four months, I think, maybe a bit more.
And then you broke it all. I mean, what are you doing? It was a, it was a, it was a very, very interesting day.
So for the viewers, you've only got a minute left.
So I'll be very quick. The engineering manager responsible for for the WAF, and I've been there three months.
And as I just said, I was very open about the fact this place is huge.
I'm still learning here. And then and then yes, in July, we suffered, my portfolio suffered the biggest outage, global outage that Cloudflare has seen for many years.
Thanks very much. I really appreciate it.
It was it was it was how I reaction to that incident is how I realized that I was absolutely right to join Cloudflare.
John then did spend the next several weeks every day asking me if I'd broken anything.
It's just to make sure you, you know, didn't break anything.
So I think I mean, I look back on that day as both terrifying and fantastic.
It was terrifying when we're in it. I look back and I'm like, everybody got in that room, figured it out, talked calmly, collaborated, made it happen.
And I look back and I was like, that was actually great.
I was proud. I was genuinely proud of the team that day. Yeah, let's just not do it again.
I'm not going to do it again, John. All right, four seconds left. Jen, thank you so much.
Hopefully I'll get to see you in London one of these days or in Lisbon.