Between Two Clouds - A Look Inside Cloudflare Support

Oh, hi, I didn't see you there. Welcome to Between Two Clouds Inside Cloudflare Customer Support Cloudflare TV segment.

My name is Shane Ossa and I'm the host and this segment is all about getting to know the support team at Cloudflare, who we are and what we do, our processes and everything.

And we just talk about customer support.

Today I have with me Lindsey Monyelle, who is in the Lisbon office.

Hi, Lindsey. Hey, how's it going? Great. Thanks for coming on. Yeah, this is exciting.

It's a cool opportunity to just kind of chat about what we do and what it's kind of like on the inside.

Yeah, that's right. We have about 120 people spread across the world and I put out the call to get volunteers to come on the show and you are one of the people brave enough to come on and do this.

So thanks for being willing to come on in front of the whole world, the whole tech industry and talk about, you know, your experience on the Cloudflare support team and whatnot.

So thanks.

Yeah, of course. Thanks for bringing up this concept. I think it's a really cool thing to be able to sort of show the inside of support and how, you know, kind of we function from the inside, as opposed to just being like this sort of wall of techs that people usually see us as.

Yeah, exactly. Yeah, there's not a lot of visibility into what we're doing.

I mean, we're, we're a customer facing team. And so the customers see us every day.

But the rest of the company doesn't necessarily see what we're doing every day and who we are.

And so we just we just want to kind of surface that because it's pretty interesting stuff and I think we have some really talented people on the team, such as yourself.

So, yeah, you've been on the team.

How long now about eight or nine months. Yeah, yeah. So it's going to be eight months in the beginning of the beginning in the middle of this month.

So, yes.

It feels like I've been here forever and it just started yesterday at like the exact same time.

Yeah, yeah, I was thinking you already feel like a veteran that we can turn to for great feedback and advice on how we should be doing things and it's like, Oh, you've been here eight months.

That's crazy. Well, and not only that, but just like how much I feel like I've grown as a person and how much I like the knowledge that I've gained since I started here.

Yeah, it doesn't feel like it would fit in eight months, you know, like it feels like it should have been over a period of years.

And it's actually interesting because I had a video chat with this woman that I was studying for the AWS solutions architect associate exam, like a year before I started and she, she thought I sounded like a completely different person, basically with my level of knowledge.

Yeah. And it's all from working here.

Yeah, yeah, it's incredible. I mean the amount of different types of products were required to learn about, and the products are very different in nature.

I mean they're all interconnected you know you've got your DNS and that underlays everything but, and this between all of them but yeah I mean learning HTTP and learning, you know, everything JavaScript you have to, you're exposed to all sorts of different protocols and programming languages and tools that we're using and the customers can can be doing anything right and then we get tickets sometimes.

And for the, those of you that are watching who don't know, the customer support team handles customer contacts to us in the form of tickets, it's like an email but we use a clever ticketing system so that we can keep it organized.

Sometimes we get tickets that just say things like, Hey, my website's down.

Right. And you're just like, Okay, where do I start. That's broken.

And there's all sorts of fun cliches within the tech support and IT industry around, you know, did you turn it off, did you turn it on type of thing right.

And we do actually tell people to try those types of things I mean not like reboot your server but but nearly.

Well, and especially like, I mean that's essentially what training like purging the caches or if you put on developer mode and off that's exactly kind of does that same kind of function.

Yeah, cancel and reorder your SSL certificate.

Yeah. Yeah, but we also have to do a lot of kind of what you might consider the more fun stuff right like investigating, you know, and debugging why there is some weird behavior issue, you know, between the customers origin server and Cloudflare's edge servers right.

Absolutely. I mean, there's, there are some things that are pretty easy to figure out in a few minutes.

And those are kind of exciting in their own way because usually it's some sort of like, Oh, I've already like I figured this out.

This is reminding me of something I did in the past.

But then there are other things you have to dig in for a good amount of time and follow a thread and sometimes the issue isn't even what you thought it was when you started.

Yeah. Yeah, that's such a challenge for us in so many ways so for transparency for those of you watching, I'm the training manager for the team so it's my job to help everyone get the training they need to do their jobs and Lindsay is one of our tech support engineers so frontline working directly with the customers debugging their issues.

And so, you know, from a training standpoint I think also from the tech support engineer and learner standpoint.

It's easier when there's one of those ones you described where it's like okay here's this process and we've, we've done this before we practice this and there's some great you know documentation on how you might be able to fix this problem, but that's only there are some that are novel, and we're relying on your, you know, sort of talent and troubleshooting skill sets and, you know, almost just like process of elimination and all sorts of all sorts of things like that to like figure out how we're going to solve a problem that we haven't seen before right and it's a little scary but it's also a little fun.

It is it is and, you know, especially coming from, from my, my background which is mostly customer support and very little technical.

Yeah, I can't believe that you've done so well. It's, thank you.

I really appreciate that. But it's, you know, it's. There we go. Sorry, I'm had to get the screen back.

But basically, when I started, I thought that everything could have a series of steps from A to Z like, you know, solving five to two errors, that process could be easily replicated across all of these.

And it's not.

No. It's sort of almost like a choose your own adventure novel because you come to a certain point and you have to look at it and see which path should I take to investigate further, right.

Yeah. Yeah, sometimes we're just giving advice to customers, you know, I mean, we try to limit things to what we would consider break fix right something is broken, and then we'll help you fix it.

But the reality of is reality of it is, we're also doing some consulting, we're also you know giving people advice as to how they might configure something and that requires that we learn about what their, you know, back end infrastructure is and you know what server types of servers are using and, you know, what other networks are running on top of right and stacking CDNs and.

Exactly. Yeah. Yeah. And I tried to be clear, you know, sometimes when I, when I did more into that direction I tried to be clear that like hey, I can give you some information, like some advice from what I see from this small piece of your puzzle but I don't know like other things that this may be impacting.

I can't see your entire system your entire use case if it's security I don't know your full threat model.

But, but we absolutely do because especially seeing more of the caching and performance tickets like I tend to see.

There's a lot of things that basically come down to okay well how can I make this better.

And that's analyzing the performance and seeing where you know where the sort of the weak spots are and how we can bridge that back up.

Yeah, yeah.

And another kind of direction to go with this is, you know, not every customer plan segment is necessarily going to get the same level of support from us, full disclosure, you know, a customer is a free customer not paying anything and of which we have millions right literally millions.

We can't possibly give them the same level of service as we give our enterprise customers right so with your enterprise customers Lindsay's going and making suggestions and analyzing, and with your average free customer it's a little bit of a teach to fish analogies which we use a lot, and I think the same is true for all of our customers even our, even our highest paying customers we really it's in everyone's best interest that they can fix their own problems as much as possible and find the information they need to fix their problems as much as possible right they don't necessarily want to contact support right they want to work, and they want to be able to figure out themselves if it isn't working so like we're like third or fourth on that chain of things that they wake up and like, hey, I can't wait to talk to cloud for support today no they're saying you know I can't wait to do my job as this web developer webmaster and oh wait something's not working and let me see if I can find the information and oh okay I can't so let me contact support.

Yeah, ultimately people want things to work right like they don't they don't clock social hours with us.

So, at the end of the day it's everybody's best interest to like what would you rather do what I would you rather have to like contact support, have them make a page rule for you have them make a firewall rule for you wait for that process to go, or just like have the have the tools to know how to do it yourself.

And that's why I think, absolutely, like, regardless of what the plan is really anything that we do do should have a heavy dose of, and this is how, and this is why.

And then that way, if it's not something we have to configure on our end which for the most part we don't because we're pretty hands off.

It's it's in people's best interest to gain that knowledge.

Yeah, and it's yeah, exactly. And it's in our best interest as a team of 120 people serving, you know, millions of customers and domains to enable the customers to help themselves as much as possible right.

Of course, of course, because also for those kinds of things for things that you know can be can be shared knowledge with the community of people who are using our products.

That is time that maybe if there's something urgent going on like if somebody is having an attack or somebody has a bit more of a crisis.

And that's also again you know regardless of the plan.

We would have more time to have the attention to these, like, more serious concerns.

Yeah. Yeah, you mentioned those under attack scenarios. Those are some of the more fun ones I know they're not fun for the customer while it's going on but from the tech support agent standpoint.

It is, it is fun to be able to help someone restore their website and figure out what the bad traffic is the malicious traffic and help someone mitigate that.

Yeah, because Cloudflare does all these different things and they're all fun but that that that one is kind of like feels like sometimes more fun, especially from the support engineer, do you work those types of tickets and we can talk about skills based routing, which yeah yeah I do.

You are part of that security group yeah yeah so security is my is my secondary group.

So I see more performance, but I, I got into tech with an interest in security so it's something I'm very passionate about.

And the, I think one of the reasons why we, I hate to use the word enjoy but they're one of the reasons why you know they are, they're appealing the under attack tickets is you have the combination of immediate results, being able to do something for somebody like it satisfies that desire to help.

It's also technical you get to dig in you get to analyze, you get to sort of construct a solution based on what we see, so it, it sort of satisfies a few different areas to feel like you really accomplished something.

Yeah, yeah. I mean that's where we get to look at we're looking like a fancy graphs right and there's like, you know, little lights going off that are showing us packets per second and like, Okay, and this is the threshold and this is their peacetime threshold this is what they expect and there's all sorts of fingerprinting and signatures going on and the requests and, and you know without giving out too much of our secrets you know we're looking at how we can block block the bad traffic and keep the good traffic in because that's from, you know, keep the good traffic coming in because that's the challenge right I mean that's the sort of constant battle here is that they're making the malicious, you know, traffic, they're making it look really real right.

And so how do you keep the good traffic in and block the bad traffic it isn't just always a volumetric attack and it's like oh this is obviously just a rate limiting deal.

Well, yeah.

Sorry, and it comes back to what you were saying before about the lots of different customers of different sizes, different server sizes, different experiences like there is no one size fits all.

What if you are an eCommerce site who who's pretty small who just got got like retweeted by a celebrity and suddenly had a ton of legitimate business that that spike is legitimate us blocking it is not going to do you any favors like that's going to be a bad choice.

Right. Likewise, for the amount of volume, like what I could host on a Raspberry Pi is going to be way different than what somebody could host like massive server rooms.

We can't sit set a one size fits all for these things, because there isn't one.

Yeah, yeah.

Yeah, I mean so these are just all these things that you've picked up so quickly and, and, you know, you come from a customer support background.

But, and, you know, we hire people with customer support expertise we hire people with technical expertise.

We have people with both but I always feel like we can sort of teach the technical stuff, although it's hard for everyone.

And it's fun and challenging.

Sometimes the customer support skills are some people consider them soft skills but sometimes I find them a lot harder to train on.

And so I really think it is, you know, you said, people shouldn't sell themselves short as is having that skill set and not technical necessarily or, you know, less technical, because I think it's really, it's a really underrated skill to be able to, you know, communicate well with customers, you know, you know, yeah, for sure.

And I haven't seen a push lately to instead of calling them soft skills but to call them skills.

And I think, I think that kind of highlights it because it is it's easier to explain how DNS works than it is to explain how to have a delicate touch, or how to have some like have some in tuneness to what might be going on with the customer.

Here just have empathy, do empathy. Yeah, just do it. Yeah, that's not easy.

Yeah, yeah. Fine. Yeah, exactly. Could you turn the empathy up to seven. Yeah, so those are great core I like that core skills and they have to take that.

But speaking of skills.

We touched on it a second ago but the team has migrated over to a completely new workflow model which we call skills based routing, and what we were doing before wasn't formally called this but it was more of a what I've heard referred to as a swarming model.

So, cut the notes and sound good. Customers right in and if you're an enterprise customer.

We, you know, put all the enterprise customer tickets in a queue, and it's first come first serve.

Right, so we did we work the oldest ones first.

But we also are triaging those and we're working the most urgent ones first right so if your website's under attack we're going to help you before someone else who just has a general question, right.

And we sorted that we had an enterprise queue.

We have it all nicely prioritized and sorted. And then we have all these enterprise trained people that were, you know, trained on all of our products right.

And we said, you know, go go crush that queue right and answer those customers and so solve those customers and we to work that in the sort order.

But then it got to be a little bit crazy I think both for the learner and the trainers to say, we need to train all 120 people on all 256 Cloudflare features, right, yeah, that's, that's not reasonable so let's, let's start actually category we were already categorizing our tickets by feature.

But now let's actually route those tickets to people based on those skill sets, and we set out to give people a set of core skills right DNS SSL caching under attack performance.

And, you know, HTTP response code errors right we have core skills and then we specialize past that in some of the more complex Cloudflare product lines magic transit is a big one that we have to specialize in because it's just more complex right.

And there's lots of others workers.

And so, you know, Lindsay was mentioning that she was, you know, getting into the security path and, you know, some of the security specializations how I know I'm rambling here but how have you found this kind of cut over, and the new workflow model and kind of concept of specialization and skilling in this way.

So I, I do like being able to focus on areas that I feel like I'm stronger in.

I also think that it's, it's really good, like we've, we've talked about how being in a skill does not mean that you can only work that skill for the rest of your tenure at Cloudflare like if you, you decide you're bored with a certain topic or it's just not motivating you anymore you can branch out and move on to other areas.

It's definitely, it definitely helps especially for people like with my background and similar having less technical background, it's less, you know, learn all of the things at once.

Although I was kind of on the end of, of the wave that had to work.

Yeah. Yeah. And I will say that it. I've, I've never been so tired and so fulfilled in like my entire job career history, so it's been amazing.

I do think it'll be really good for people to focus though because the more time you do in one area, the more you're going to be able to build up on knowledge as opposed to kind of starting over to a certain extent, because maybe you haven't seen that in three weeks.

How you forgotten like how do I do this again you know it's gonna, it's going to help it build to your long term memory, I think a bit faster when you're new to the technology.

Yeah. Yeah, that was the other thing we had to figure out was with the swarming model, everyone would be exposed to kind of everything.

And that was good in some ways because then we could expose people to things that were harder to categorize that are gray areas right and we were feeling good about our team being really versatile with them.

Right. And now we're, we're going to see the effects of, you know, never routing some uncategorized ticket some novel problem to somebody, and what kind of effects will that have on our team long term in terms of versatility and right.

Yeah, just being, oh my god I've never seen this type of ticket before.

And that's interesting. Am I going to solve it right.

That is interesting. I wonder how this is going to impact, maybe people who started completely after skills based routing.

And if, if, like, you know if this is.

I don't know I just had off the top of my head, maybe having like 1% of the tickets could be random.

So maybe you see one thing that's an Argo tunnel when you don't do our goal.

Yeah, for the day. Yeah, yeah. Yeah, exactly. Yeah, we're figuring out we're still actually tuning it, if you will.

Yeah, I'm saying the tickets are we're still because we built this, you know, we have a support operations team.

And these, this is like a sub team on the support team of programmers that build these types of things that build tooling, they build, you know, this type of system for us and maintain it.

And so we're figuring out how to balance the tickets to people properly right where some people are getting all really hard tickets you know more than they could handle in one day and so we're getting, you know, too few tickets and, and, you know, so we're still figuring out exactly how to route this into sort of tune it so that it's efficient because at the end of the day, you know, we're trying to make a better customer experience right we're trying to make it so that when I write in with a caching problem I get people who know caching really really well right yeah as opposed to like I don't know maybe your TLS is expired.

Yeah, yeah. Yeah, yeah, for sure. And, and I think that some of this, you know, we're doing the skill based routing and we're also trying to maintain a bit of continuity now that so that people can continue like working on the same issues.

And, and in the meantime, while you know this is all sort of getting sorted out and and sort of finalized.

One thing that I will say is the the primary group that I'm in.

We've been, we've become pretty close knit, as far as sure that nobody has too much was like oh so and so has all of the enterprise tickets today let's let's work together like let's yeah let's help each other and watch each other's back.

So that's been really cool. Yeah, yeah, I mean there's some manual intervention right going on right now where we're, we can reassign tickets to other people.

Yeah. So we can like while it's going, you know, while we're still sort of putting on the finishing touches for the system like we do have ways to like manage it and it's, it's just a matter of having sort of a group awareness of what people's loads are and who needs more help who, you know, who's got a pretty easy day who can jump in.

Yeah, I have all the tickets that are going to breach in 30 minutes.

Right. Yeah, we're figuring out because we prioritize tickets p1 p2 p3 p4 if you're a customer now watching this.

Tell us how urgent your issue is it really helps us if we know the impact to you and your, your site infrastructure, and you can even use the P level you can say the P levels if you want but you just tell us when you're when it's really urgent to you.

But we categorize those and that starts a timer.

It starts a timer for Lindsay and the team to help them prioritize which tickets to work next and that timer is based on the urgency of the issue we call it our SLA response timers.

But, yeah, you mentioned, you know, the group that you're kind of working closely with, and the offices and that's one thing I really love about the support team is the people that are on the team, and the kind of team camaraderie of it we're like a small.

We're actually are like a, like a team in, you might imagine a sports team is like right where we have like positions and times and things that you do and plays almost like that.

You know, whereas we're not just kind of all I see is kind of programming on the side when you can and doing sprints were like, we're pretty operationalized regiment and that makes it kind of fun but you're one of the people who has started since everything has gone remote right we've been 100%.

Yeah. I've never met any of you.

That's weird. What's that, what's that been like to come into, you know, an established company and established team, and kind of not need anyone in person and navigate that.

It's so it's interesting, because I think, I think for me it actually hasn't impacted me too much.

One, because I've worked remotely before. And two, I've, I've been part.

I've been a part of like online communities that have grown where we've become friends and I even married one of them and moved across the world, live with them.

Yeah, yeah. So, it's almost been sort of an extension of that.

So, I have plenty of friends like around the world that I've talked on the phone with that I video chatted with, I've never met them face to face.

So, I guess. Yeah, it's very 2021.

Yeah. Yeah, I love that. Yeah, I think I agree I think we've done well to transition and it's suited a lot of people.

Well, the one thing that I've missed from training standpoint is some of the more casual training in, you know, inverted commas, that would occur just by being in the same room as people like particularly I'm thinking around our phone support is one.

So we do phone support for our enterprise customers and it's mostly emergency based stuff right so they don't necessarily aren't necessarily supposed to call us unless it's an emergency.

But anyway, if you're in the office. You're hearing your colleagues take calls we don't we're not a big call center we don't get tons of calls but you'll hear one or two per day.

And so you'll hear you know Benedict or, you know, or Demeter you'll hear them take a call, and how they handle it and what they say, and we don't get that now like everything now has to be scheduled right like if we're going to do something, we have to have a meeting invite on the calendar and join a video room and like, so all that just kind of casual ad hoc.

Hey, could you just look at this ticket, let me show you what I'm doing, you know, I'm curious about this, or, you know, the other way around where they go hey come check this out I got a really interesting one.

That's much harder to do, and it's sort of intangible but so we've actually kind of replicated that with my skill group, and I kind of hope that it catches on that, you know, other people think like start doing this, but we have a we have a standing video hangout room that we meet every day.

And we're pretty much in that room all day long, and this is actually. So, this, this ties into a couple things skills based routing in general, when you're on phones or chats, you kind of have a roster of people that you can call on if you need and, hey, can I just quick to hear about this topic, you're in this group I'm not, I've utilized that for sure.

And as far as the casual training, we absolutely are like, oh hey, can you can you help me with this ticket really quick I have a quick question, or we'll bounce the ideas off each other.

And it's, it's way more comfortable to do it in in a skill group that's working on the same stuff so it's relevant for all of your learning.

And as opposed to having to do it in like a larger conference room.

I've described this as imagine in a physical space, every time you had a question, having to go up in the middle of a room of the whole office, stand up and say, I have a question, right, yeah, it's, it's a little bit more intimidating.

Yeah, that's true. Yeah. That's a great but so good to get that feedback, you know, and to, and to flag that because, you know, we need to hear that more often I think as I'm not part of these conversations but as management teams think about, you know, the future of remote work or hybrid work, right.

They're wondering what we're losing right by not being in the same office. Yeah, yeah.

And like it's great because we have this sort of this communication flow that makes it way more comfortable to ask questions, and then other people see you ask questions so then they're more comfortable and it's just, I don't know I feel, I feel like it's really helped my confidence, and it's helped.

I've had, you know, co workers talk about how they feel less lonely, since we've started doing it.

So, I think it's been a really positive experience.

Cool. Well, speaking of positive experiences we're almost done here I can't believe it went by really fast.

Wow. How did that happen. I have no idea. I just talk a lot. That's well, you were great.

Thanks for coming on. Come on again. Thanks and see it was really great chat.

Yeah, for sure. Thank you for having me on. Cool and I'll see you tomorrow at our handover I'm sure.

Yeah. All right, well thanks for watching between two clouds inside cloud first support see you next time.