How We Train Technical Support Engineers

Hello, everybody. Welcome to Cloudflare TV. Thanks, Chris. My name is Shane Ossa. I'm with the customer support team at Cloudflare. I'm the technical training program manager, so it's my job to make sure everyone gets the training they need to do their jobs as support engineers. I'm joined here by my friend and teammate, Jamie, who will introduce himself. So I'm the technical trainer for the United States, so I help train everyone for the program that Shane's created, and we work together to create the training program for all of our CSOP engineers. Yeah, that's right, and Jamie does an amazing job training all of our US new hires and training other technical support engineers all around the world. He was, fun fact, he was a tech support engineer for several years before he joined the training team, and before that, I don't want to out him, he was a Cloudflare customer as well, so he knows a lot about Cloudflare and how it works. So really excited to be here to talk to you today about the training program that we have for tech support engineers at Cloudflare, and if you have any questions, you can email, I think, down here on the website. If you're looking somewhere, there's the email address to ask questions. We'd love to answer them live if you want. Go ahead and ask them. So I'm just going to launch in. I've been at Cloudflare about three years now. When I joined, there was about 400 people, and the customer support team was about 25, and since then, the Cloudflare has grown to about 1,500 people. Don't quote me on that, and our team has grown to about 90 plus people, almost 100. So it's been an amazing journey so far, and it's just been really fun, and I've learned a lot in being here. So when I started, I was tasked with building training for the customer support team, and we had some training at the time, which was delivered by some of our team leads. We had a few notes and a couple of calendar invites that had agendas on them, but for the most part, our training involved, hey, why don't you sit next to this person that knows everything about Railgun, which is one of our products, and then biosmosis, you'll learn, and we were able to get by successfully for a long time with that, but we have since added dozens of products, thousands of customers, millions of domains, and so it's made a lot more sense to structure it even more. Just want to talk about how we've done that. Jamie, do you want to add anything? No, it must have been when you first started this role. Where did you start with that? Because you came in, there's this basic program of getting people to span up, but it's mainly them learning themselves, right? So where did you kind of begin with your process of creating a comprehensive program like we have now, two and a half, three years later? Yeah, that's a great question. I mean, I had something to go on. We had like the skeleton of a boot camp for support engineers, so we had a couple of trainings that we were definitely giving everyone at the beginning, a few like one or two hour sessions on how the Cloudflare dashboard works, all of the features, and sort of some of the behind the scenes back -end stuff. So I had something to go on, but I decided to do what in the training industry we call a needs analysis, so we're trying to determine what the big training needs of a team or an org is. I interviewed everyone, interviewed all of the tech support engineers. I was basically trying to get a picture of what the job involves. I had done technical support in my previous role, but it wasn't at a software company. It was at a solar PV manufacturer. It was a bit different. So I interviewed everyone. I started to create sort of a, we call it job task analysis of what the job entails, what are all the tools we use, everything like that. The other thing I realized what we're going to need is a lot of documentation to support any training that we do, because we knew that we had the experts on the team that could train other people how to do it, and we would train them and then hope that they remembered what they were told. And so we realized we needed a lot of support documentation, which we call run books, which are explained and have it written down how to solve various customer issues. So one of the things we just really went about early on was to document everything that everyone knows and just write how to solve this type of issue, how to solve that type of issue. When a customer writes in with this question, what should you do first? So there's a lot of interviewing and documenting and then started building courses and trainings and creating training outlines and the curriculum eventually based on what the job entailed day to day. Cool. Yeah. Yeah. It must have been like at the beginning of doing that, finding like you came in this role and have all these people who are really technical trying to find like who to create these documentation, like people have this knowledge, like even now that these people that have this specific knowledge, but the documentation to show everyone has to be pushed right to exist. So back then that must have been quite a feat just to get that all together at the beginning. But now we've got like a vast array of run books, wiki pages, etc. There's a vast array of everything so we can share knowledge in a centralized position, which is exactly perfect now, I'd say. And it improves every day. Every single person on our team adds new nuances or niches in different products and how to troubleshoot certain types of issues or how a new product works to help the rest of the team spin up on at least the basics of how a product works prior to launch, for example, right? So yeah, we've got that in place. So yeah. Yeah, so I like to think of all the training that we do as falling into these four categories or these four buckets, if you will. So one of the big buckets is Cloudflare products and features. How do they work and what to do when they break or how to troubleshoot them or debug them when a customer has an issue. So that's a big one, right? What are all the Cloudflare products and features? The other one is our team's processes and our policies. You know, we're a security company, so when a customer writes in, we have to validate them and make sure the person we're talking to is authorized. There's stuff like that. There's a lot of policies and procedures we just have to train on our standard operating procedures. We also train on all of our tools, of course. So command line, bash, git, other internal tools we've created ourselves like crossbow. There's loads of others that we have to train on that our customer support team uses every day to analyze data against customer issues. And then the last big bucket, and this is a really big one that we train on, is just general subjects, concepts, technology. So this is like, how does DNS work? How does SSL work? How does TCP work? How does HTTP work? And the people that we get tend to already have experience in one or more of these fields where they have relevant experience. So we're typically never starting from scratch. And another facet of the training program is in advance of them actually starting, we send them a bunch of homework to do. And we send them a list of public support KB articles to read. We ask them to create a test account. We ask them to install a web server if they don't already have their own website. We give them some instructions on how to create one because they're going to need to know how to troubleshoot them. So they go through that experience. So yeah, all the training we give sort of fall into those four main categories. That's kind of how I like to frame it. But after that, that's throughout their career at Cloudflare. We never stop training. But the big kind of initial blast of training is bootcamp. So after all Cloudflare employees go through a Cloudflare orientation, which is amazing. It's an amazing experience. And you get to kind of bond with other Cloudflare employees from all around the company. And you stick with these people for years. I still chat with the people that were in my class. And then after that, we start our customer support bootcamp. We call it CSAP. Customer support for short. We say CSAP. I might say that throughout this talk. So then we have CSAP bootcamp, which is pretty intense. And you want to talk a little bit about that? Because you're actually the person that delivers a lot of the bootcamp trainings, if not all of them. Yeah. I want to step back slightly on that in terms of the bootcamp itself. So I came into Cloudflare after using Cloudflare for, I can't remember, three or four years prior. And the reason I joined, used Cloudflare in my personal space, web development, etc., was because of the free SSL Cloudflare started to deliver. And I just wanted free SSL. So I joined Cloudflare in that capacity as a customer on the free account only. And the amount I learned just from using Cloudflare in that way really helped me during bootcamp. And when I got to the bootcamp, which I believe I was one of the first to go through it, maybe second or third class, I can't remember. And it really opened my eyes to how Cloudflare works and etc. It's just really interesting. I enjoyed it so much. I was overworking during that bootcamp, you know, like extra learning outside of hours just to really get, you know, get as much knowledge as possible before I hit the queues and had to help real customers. Yeah, the amount of products and features Cloudflare has, like, as a customer on the free plan, you have a plethora of things. But when you join support, there is so many new products being released all the time. So we have to be really on the ball. Like Shane, Shane has to try and coordinate with different people to try and ensure the training's in place before new products come into place. And being in the, going back to me going through the bootcamp, I found it really good. So we first of all, learned the real basics, and then you learn from other people as well. So we still do like the shadow sessions, we call them, where you sit with someone experienced for things like that. So I found those really beneficial. But me myself, I learn more by doing. So I used my previous knowledge to spin up these things on like VPSs, or I think I had a dedicated server back then. And it was just really such a great learning experience to take everything with all these theory, and then putting into practice. Yeah, and I remember that. And you were one of the people, the earliest people that went through one of the early iterations of our bootcamp. And that gave us just really great feedback. Yeah, it's changed so much from when I did it. It's now a whole new, it's like this. Comparisons are very small now, right? It's changed so much and evolved based on feedback. Every person that goes through it, we asked them multiple times for their feedback, which adjusts the program based on that feedback. Exactly. And you gave us some great feedback. People over the years now have just given us amazing feedback. We survey them. But we also ask them, we ask other people to ask them so that we can get extra perspectives to make sure we're just not missing anything. And we just want to make sure we're constantly iterating and it's evolving and we're improving the program. So to get back to the program a little bit, we have the bootcamp, which consists of hands-on trainings conducted by Jamie, myself, other experts on the team. We have online recorded trainings, which we call e -learnings. So this is sort of a common strategy for training programs, is to employ what's called a blended learning approach where you have some training that is recorded. And I mean, most people have taken these at their company when they start a company. There's what's called a learning management system usually, and it tracks what trainings you've completed. And there's curriculum or learning paths in there. And you get maybe even little badges when you complete them. So we have that too. We didn't at the start, but we kept giving the same presentation over and over again. And we realized, hey, why don't we record that presentation? And then that person can watch that. We can go do something else while they're watching that. And then we can do a hands-on training and we can focus on getting them to actually get on the command line, share their screen, practicing and doing real exercises as soon as possible. And when they come to those, they've already watched the e-learning about the subject. So I should say that during the bootcamp, we sort of section everything. Each day is about a different subject, which kind of is logical. So one day is about DNS, and one day is about SSL, and one day is about caching. And it's in a specific order. And we tweak the order around, of course. So the typical bootcamp day would be you watch introduction to DNS, and then you come to a hands-on how to troubleshoot DNS issues with Jamie. And then in the afternoon, and you've sort of alluded to this already, we have what we call a shadow session with an experienced support engineer. So someone who's already been on the team for a while. And we change this around, and this gives everyone a chance to be exposed to everyone on the team's different styles and different processes. And some people are on macOS, and some people are on Linux, and some people are on Windows. And so you get exposed to everyone's different styles, which is great too. And during these shadow sessions, the shadow host, we call them, will share their screen and work customer issues, which we call tickets. And they will share their screen, sort of narrate what they're doing. Okay, now I'm running this test, and now I'm going to tell the customer this based on the results I got here. And then so you're watching someone for a few weeks you do this. For a few weeks through bootcamp, before we even have you doing any customer tickets, we have you in these trainings, watching trainings, taking hands-on trainings, and doing these shadow sessions with experienced support engineers. So it's a pretty big production, and it's quite intense, I think, for new people a little bit in a good way. We like to expose everyone to everything at first. Just give them a little bit of information about everything, enough to be dangerous, enough so that they can, you know, get exposed to everything, and then later on we're going to double click on specific topics. And that's sort of been our approach. We adapt it all the time, but that's the bootcamp approach. Do you have anything you want to add in there that I might have missed on bootcamp? Free tickets. Maybe do you want to expand on that part, just because the learning by doing? Yeah, so another big component is learning by doing. Learning by doing, we have people do free tickets. Cloudflare has a free tier, which is an amazing product for the world to come and have free SSL, like you're saying, free DDoS protection, free CDN. And so there's a lot of free customers. We onboard thousands and thousands a day, and we see a lot of tickets from these customers. And this is a good training ground for our news support engineers, because you tend to see some of the more basic issues. You might be dealing with someone who just has a blog, and they just have a simpler question about DNS. And it's not high pressure. You're not working with our most, you know, highest paying customers yet, right? We definitely don't start people off working with our enterprise level customers. We don't work with enterprise customers for several months. And after a lot of training, and after a lot of checks, boxes get ticked. So the learning by doing involves doing free tickets, and we QA them. The training team QAs them. Your direct manager, which we call a team lead, will QA your tickets and just give you feedback and say, that's correct. That's exactly how I would have answered that ticket. Or maybe we'll say something like, oh, you know, I would have worded that a little bit differently. Or you'll say something like, oh, did you know that there's this tool where you could have checked the logs and gotten to the answer faster? So we encourage new hires that are going through our program to actually go and at least browse the free tickets and then come to the shadow sessions with specific questions. A big component of training is customizing it and catering it to someone's individual needs. We realize that everyone is different. Everyone comes to Cloudflare support team with different experiences, different past experience, different work experience, different skill sets. Different way of learning. Yeah, total different way of learning. Some people love the e-learnings. I want to watch all the e-learnings. Some people are like, I don't really like the e-learnings as much as I like the hands-on. And some people are like, I love the shadow sessions. So we realize everyone has a different style. We realize everyone is different. And so we try to cater it as much. We have this standard program, and then we customize it to meet everyone's needs. And one of the ways we do that is by asking people to look at the tickets out there and bring their questions. And they'll say, well, I have a question about DNS. And someone else might come and say, I don't have that question about DNS. I have this other question about SSL. And so this lets us calibrate our training and our instruction to meet the individual needs of each person, which is a big challenge, to be honest. But it's something that we do, and we do our best to try to stay with that. I see that we got a question in on the question line. Ask your questions down here. The email is down here. Are most training materials publicly available outside the obvious internal items? Some. I mean, our public KB, the knowledge base, support.Cloudflare.com, has amazing amount of content. I would also like to plug our community. If you watched the previous segment with Chris Scharf, there's amazing content in the community with other Cloudflare customers. But going back to the KB, the public KB, we have a team called the content team. Shout out to the content team that's just making the content better, creating videos for the content that you learn a lot from. Now, there are stuff that isn't in there. We have some tools that we use internally that we can't expose to our customers or the backend stuff. You won't have access to the same logs that we do, obviously, to troubleshoot certain things. But the KB is a wealth of information. Knowledgebase .com, the KB, is a wealth of information on Cloudflare products, how to troubleshoot them. And the other thing I should plug is that we're trying to expose, if we have a troubleshooting tool that we're using, we want to expose that to the customers if possible. So there are big initiatives going on with our team to take Crossbow, which is one of our troubleshooting tools that allows us to test servers. We want to expose that, if we can, what we can, to the customers and let them troubleshoot their own things when they need to. And the other thing I'd like to plug here is the developer portal, developers .Cloudflare.com. That's another place that you can go that has amazing information about Cloudflare products. It's a little bit more for the slightly more technical audience and the developers and the people building websites and whatnot, web apps. But it's got amazing content in there on how all of our products and features work and how there's usually troubleshooting sections in those as well. So could I just add a few bits in regards to the question that came in? So the question of, it's like none of our training material itself is public available, as Shane said, but the KB exists in the way to teach you. So a KB article exists because we have had multiple questions from customers coming in regarding that, right? And we want you as customers to help yourself because it's the most efficient way, right? If you Google something and it comes up with the Cloudflare KB article, it means the answer in there will probably help you solve the issue yourself. Because let's say the vast majority of customer issues are not something we can technically fix. We can point you to where the issue is. And usually in the replies we send you, we link to one of these knowledge-based articles because they are common questions or they are slightly different nuances in the way the products work. So each KB exists to train you. So they are your training material as customers, I'd say. So yes, we do have training materials for you and that is them. And we use our KB articles, our public-facing KB articles as internal training material as well. Sometimes it's best content we have to describe how certain things work. But yes, to kind of echo what you said, we have a policy where we don't make configuration changes. We don't change settings for customers, right? So we are going to teach you how to fish. We are going to teach customers if something is misconfigured, this is how you will configure it properly. So we are never actually behind the scenes changing any settings. We are troubleshooting and making recommendations. And then as much as possible, we know that people want to fix their own issues. They don't necessarily want to go through the experience of troubleshooting via email. So that's why whenever we are seeing a lot of issues and there isn't a KB article about it, then we take note of that and we get it on our list of KB articles to create and we work with internal subject matter experts, SMEs, to create more and more public -facing KB articles. So yeah, we actually use our own KB articles as training materials, but the main difference between something that would be internal and external would be that there's some mention of some internal tools that aren't exposed to customers. So after CISA Bootcamp, we then sort of progress our support engineers through what we call the queues. So customers write in and we segment these customers by the segment. Our highest priority customers are enterprise customers. They pay us the most and they have the access to the most features. They also are entitled to 24 -hour phone support and also 24-hour chat support. And so you, as a new Cloudflare tech support engineer, you will not start out by working with these customers, enterprise customers. We're going to train you up to that. So that's sort of one of our big goals and milestones for new tech support engineers is to work with enterprise customers. And since they have access to the most products and features, we will layer on product knowledge. We can't teach you about every product in the first few weeks. For one, there are too many, but for two, it would just overwhelm someone. It's overwhelming enough as it is just going over our core products that are available to free customers. So we layer on the product knowledge to match our plan level. So Cloudflare has a professional or pro level, business or biz level, and then enterprise. And so we start new hires with the free customers, as you mentioned, and then eventually they start working with the pro customers once they've gotten a little more training and experience. And eventually they start working with the biz customers once they've gotten a little more training experience. And then eventually once they've gotten a lot of training, they work with the enterprise customers and they tend to be our highest trained people. And that allows us to layer on the product knowledge and really sort of create a curriculum backwards from there. And we say, okay, well, if you're going to work with the biz customers in week eight of your Cloudflare tenure, then we need to make sure in week seven that we, or week six and seven, that we train you on all the products and features that the biz customers. That's the initial ramp, right? The training doesn't stop. The training never stops. So we continue, we continue on. We have refresher trainings that we do all the time. We, like you mentioned at the outset, there are new products and features we need to keep, we need to train on before releases and while releases are happening. And so it's just, it's constant. There's any given day I could pull up our training calendar and there's a dozen trainings going on globally today. So we are a global team. If you tuned into Otto's discussion yesterday, we talked a lot about the Cloudflare support model, where 24 seven, we have people all over the world. The training team has people all over the world. So Jamie and I are here in the U.S. in the San Francisco Bay Area, but we have a trainer based in Europe. We have a trainer based in Asia. And we also have another teammate who manages our learning management system and who also develops all of our e-learnings. It's a huge job. It's an amazing piece of work that he's done. And so that's our training team to give you a look at that. It's myself, who's the manager of the team and then technical trainers who were all tech support engineers before they moved to the training team. And then our LMS administrator and e-learning developer. So that's our little training unit. We're hoping to grow it as the company grows as well. So people that are really interested in technical training and want to come make a better Internet and you'd like to play around with servers, give us a call. So yeah, to keep going on the boot camp and the training program, we do shadow sessions, we do e-learnings. We also have a bunch of test servers that we've created that are, let's say, broken in certain ways for our new hires to debug, right? So it used to be when we started that we were training someone and we said, okay, you know, a common error that a common misconfiguration, which throws a common error. And we want to train you on how to debug this. And we sometimes hoped for the best. We hoped for that kind of a customer issue to just be happening so we could show them how to do it. And we said, well, why don't we just, why don't we create a bunch of broken servers for people to fix? Do you want to go into that a little bit, Jamie? Yeah. So with that, I started with like the common 5xx errors. So I created some actual servers that were in broken states for like 522s, 503s, et cetera. So they weren't just throwing the error code, like there was an actual issue at the web server level or the server level. So it could actually be troubleshooted as a real issue that a customer would have. And that was the first time that was available like 24-7. So someone going through the ramp didn't have to wait for a ticket to come up or know how to break a server. Like breaking servers is hard. Like it took me quite a while to work out exactly a good way to replicate real issues on some of these servers, just because you want it to be realistic. And when you Google how to use a search engine to how to break something, there's not many results, right? So just having to break it in different ways and just testing things. So that took a fair bit of time just to get the core errors we have at the moment. But I really enjoyed that. We need to expand on that. We want to try and have as many as the common issues as possible in this like dashboard for all of our technical support engineers to go over, even if they've been here a while. Maybe they want to refresh themselves on actual issues. They can log into this dashboard, look in the other link into the SSL dashboard and see the common issues and troubleshoot something on their new laptop, for example. They just want to make sure all their stuff works, you know. So we really want to make that something to help them going forward and ensure everyone has the best experience. And there's sort of two things going on there, right? So we have servers that we have created that are broken in specific ways. These are common issues that you will see when you're actually helping customers. And we task them with fixing those, troubleshooting and fixing those. We also task our new hires with creating their own servers that have Cloudflare services running on them. So a big part of the program is getting someone, this is pretty logical, but getting someone, giving someone a Cloudflare test account with test domains and that they're going to try, they're going to install a custom cert on their origin server. They're going to put it behind access. They're going to install Argo tunnel. And we have them do all of this, not only to go through the experience that the customer goes through, the customers that they're going to be helping so that they have the same experience as the product, they're learning the product. But then if they want, they can try to reproduce customer issues. That's a big part of tech support engineering is someone writes in and say, I'm seeing this error. And they're like, all right, let me see if I can reproduce this error. And so that's one of the first steps of troubleshooting something is can I make that happen myself? Am I seeing that? Is this a bigger issue? So that's why we do both of those things. We have test servers that people can fix that we task them to fix. And we use these in our trainings ourselves. And then we help them create their own servers with Cloudflare services running on them that they're going to use for training and for issue reproduction. Now, we have lofty goals. We've only been at this for a few years. We want to build a system that we have these broken servers. And when someone debugs one and then logs in and fixes the misconfiguration, that it tracks them and then marks them as complete or accredited or even certified on that particular issue. So we're just getting started, as we like to say at Cloudflare sometimes. So we have this big dream and roadmap on our training plan. Because Jamie works on training people, but he also works on projects just related to our training program. So we have it in our roadmap to have this amazing system where we can see, oh, Jamie has fixed all these servers. He's certified in caching. He's certified in troubleshooting caching because we can actually see. Right now, we observe them. We can say we can check it off kind of manually. But eventually, we're going to kind of build this really cool automated system. You go in and you fix the server we have that's broken. We can see that you did that. It's going to tell our other service here that Shane's fixed that thing. It's going to give him the checkbox. So that's on our roadmap. That's one of the many kind of cool projects we're working on as a training team while alongside training all the people that we're hiring. And Cloudflare is still hiring. It's great. That brings us on to a question we have about hiring. So if you want to check the question, there's a question about hiring, I think. Yeah, I'm interested in applying to become a TSE. How best should I prepare myself? Oh, that's an amazing question. I would say reading our knowledge base articles. There's a few that I think there's ones that are like getting started. The getting started ones will take you through creating your own Cloudflare account. I would recommend you create a Cloudflare account if you don't already have one. I would recommend that you create a website. And on top of that, actually install a web server. So get yourself a server on one of the many kind of cloud platforms that are out there. And then actually install Apache or Nginx or one of the other web servers out there on it and get a website in HTML and get your website running. And then start putting it behind Cloudflare. Get a Cloudflare account and start using our products. And if you did that, if you've done those things, you'll have a leg up. Definitely. You have anything to add to that? Is that fair? Yeah, I think you've covered them perfectly. In terms of how I myself because I applied to be a TSE, right, like two and a half, nearly three years ago. And I prepared myself in a way in which because I've been with Cloudflare for such a long time as a customer, all I kind of did was just from the questions, you know, the homework questions, I did those and exposed me to things I didn't really understand. So I learned those things. So if you show willingness to learn, if you really show yourself as a learner and someone that would be, you know, be able to, okay, for example, if you don't really have a background in web, but you are a technical person, if you can portray that and show that you've learned Cloudflare between your applying and getting a date for your in-person interview and show that you really do understand or, you know, you've got the willingness to learn, you're the most likely to succeed. And we'll welcome you with open arms and teach you using our curriculum. That's great. Yeah, I totally agree. So get yourself a Cloudflare account, install a web server, learn some Bash, the Linux command line tool. Just search for, you know, Bash tutorials and Bash introduction to Bash shell terminal. And you will get a lot of videos, just start watching those because we're, you know, we live on the command line where that's where we do all of our testing and troubleshooting. So if you came and applied for the TSC tech support engineer role, already having a up and running website behind Cloudflare that's on a web server that you installed and you have some Bash skills, that's what we're looking for. And like Jamie said, we're looking for people who want to learn and who are curious. And that's what I would recommend. Great question. So keep it coming. If you have questions, ask them down there. So we'll keep moving. Another component of our training program besides the test servers to debug and everyone installing and trying out every single product is micro learning opportunities. So the Cloudflare support team has a wealth of talented, knowledgeable, experienced people, and we have to tap into everyone's knowledge. So we do that by having knowledge shares. We try to, we shoot for a monthly hour long knowledge share in every region, but we also strive to have micro knowledge shares more often. Lunch and learns, several of the teams do their own coding clubs. Usually this is done outside of hours, maybe an hour before or an hour after work. But we also do during work, we do lunch and learns. We just try to share knowledge as much as possible because what happens is as you're doing the job, you write a script, a script, something that you keep having to do. And then you want to share that with the rest of the team. And we try to tease that out. That's another thing that the training team tries to do is to try to just facilitate knowledge share. Some of the training is us giving training or coordinating other people, other experts to give training. But it's also just to sort of tease out the knowledge and skills that the team has itself. So we run these monthly knowledge shares. We try to facilitate micro knowledge shares all the time. We sort of have team meetings quite frequently that we bring presenters from the other teams in the company to come and present to us. Product teams present on the new feature that they're going to release next month. We bring in engineering team and engineering team members to talk about something they're building. Success team members to talk about various things that would be useful for us to know. So that's a big component. Cloudflare has a vast array of products. You will never get bored with learning and new things. So yeah, as Shane said, you will be constantly learning. Like as a TSE, you always learn. I'm learning now as being the trainer. So I have to try and stay ahead of the curve in terms of knowing enough to train people on the new products that are coming out. So then I have the knowledge for that. So we haven't just got one product where you learn it and then you follow these specific run books or anything like that. But we do have like things to help you, but you are very open to your own, you know, your own personality, your own troubleshooting skill set. Like you don't have to know bash. You can know fish, which is another type of shell or any of the other type of shells that you're comfortable with. And you can integrate it into your workflow. We don't set you on one specific path. Like all of the tools are available and you can build them yourself on your machine. So those type of things show that we are very broad spectrum of people's knowledge and people's technical backgrounds, people that need the whole concepts being explained to them, people that fly through the concepts, but then they just need extra assistance on maybe some of the enterprise-y things like, you know, enterprise log share. So we have the, it's just a huge vast array of things there. Yeah. Yeah, definitely. Um, HTTP, you know, layer seven stuff, layer three, layer four, uh, we're constantly, you know, adding new features at Cloudflare. We added magic transit, which was a layer three, four product. Um, we have to dive into some protocols that we haven't been training on yet. So, you know, a big priority of ours is training new people, um, onboarding, uh, ramping up new people as quickly as possible, but we're also trying to upskill the existing people that we have, the people that have been in our team for two years, two, three years, four years. That's a big challenge for us, to be honest, as a program, as a, as a team. Um, but we, that's where we try to tap into each other. Um, our priorities, you know, onboarding and new hires, but we, we still, uh, like to deliver additional, you know, professional development training to, um, existing team members. You know, we're not just training on hard skills, like how Cloudflare works and how to debug, um, issues. We also train on soft skills like communication and writing and empathy and how to give feedback and how to receive feedback. Um, we do trainings on hiring, how to, you know, how to get, how to conduct an interview, because eventually we have people help us interviewing new people once you've been here a while, and we have you help us interview for new people and does another thing we train on. So, um, yeah, it's really everything. Um, and so it's, it's really fun. Soft skills, hard skills, Cloudflare products and features, um, you name it, and we're building courses on it. And so, uh, we have, you know, materials, slide decks, recorded trainings, hands-on trainings that we've built that we give at certain intervals. Um, and we're constantly improving this, you know, we, we ask for feedback. Um, I guess I can segue to sort of one of the topics I have here on my talking points, which is measuring the success of a training, um, is a notoriously, uh, fascinating and challenging thing to do. Um, within the training industry, there's sort of a famous, uh, methodology, which involves four levels of measuring the success of the training. And most people have experienced level one, um, after taking a training, have you ever been surveyed by the trainers, um, or the, whoever the teacher was, and you usually get a survey and it says, um, how did you like the training? Did you find it useful? Did you find it relevant? Um, did the instructor present well? Um, how would you improve the course and stuff like that? So that's just level one, um, reaction, did the learner, the trainee, uh, enjoy the training and did they find it useful? So that's good. Uh, we always look at that. We definitely are interested in that, but interestingly, that doesn't always tell you if the person is going to be able to do what you taught them to do. Um, they may have loved your training, um, but are they then going to go and apply that knowledge to the job that you're training them to do? Um, so the second level would be the evaluation. Uh, so on all of our e -learnings, we have an assessment, um, at the end to make sure you, it's sort of more, it's something to make sure you know it. It's actually more like reinforcing it, um, as sort of like a spaced repetition methodology in order to help you memorize some of the things you need to memorize, um, when it is that kind of prescriptive knowledge. Uh, so that's evaluation. So level one, again, did you like the training? Level two is evaluation. Are you able to, uh, repeat some of the stuff that we just taught you in the form of an evaluation? Um, level three is where it gets really interesting is where we see, did the thing, uh, we taught you, uh, are you now able to take that information and apply it to your job? So this is where we actually need to do some observation on the work that, uh, our people are doing to see if we gave them a training on a product, are they actually able to solve issues related to that product? So we're doing this not to assess the learner. We're doing this to assess our job as trainers, um, to see what do we need to change in our training? Maybe we are missing and we're going over, uh, maybe we're spending too long talking about one subject and not long enough talking about another. How are you going to know that? So this is why it's really important when we do a training that we make sure that we have very specific learning objectives and goals about this training to say, look, after this training, the learner, the trainee, the attendee should be able to solve customer issues related to this thing. They should be able to use this tool that we showed them how to do. And then we can actually observe that in the field, um, and say, yep, that worked. Um, and now they know how to do that. Uh, the level four is a really interesting one as well is it has to do with, did it actually move the business metric that you were hoping to move? Um, you know, one of the goals of the training program is to, you know, increase the efficiency of the Cloudflare support engineer, uh, or improve the productivity of the Cloudflare system engineer, uh, reduce costs of the Cloudflare organization by increasing this, the time to productivity or speed to proficiency. These are metrics that we look at. Um, so how, how long does it take us to get someone, uh, working with enterprise customers? Uh, we measure these things. These would be business goals, right? So the program is, is here to support the organization, um, and to provide the structure for support engineers to show them how to do the job, but this has to, um, you know, translate to business metrics. So that's level four, right? Are we able to reduce the time to proficiency? Are we able to increase efficiency? Are we able to increase productivity? Um, are we able to reduce the amount of escalations that we're sending to other teams that maybe we did not need to send up, right? Um, to engineering teams, things like that. So that's how we measure, um, you know, effectiveness of training. That's how we're constantly trying to improve the training by looking at these, this data basically and saying, Hey, we need to, we need to fix that training. Uh, people are, people are struggling with this thing. Um, we gave them a training on that. Um, maybe the training needs to be better. Um, so that's the measuring success of training. Uh, we look at metrics, we look at the QA, uh, so that's quality assurance take week, week QA, uh, tickets. And we see, you know, we have a criteria, a rubric. Did the customer support engineer answer these questions correctly? Did they use the correct grammar? Did they reply quickly enough and all these things. And we look at these things and they get a quantitative grade, um, on that ticket and that feedback gets delivered to them. And then, um, we look at that data as well, because it tells us how good, uh, we're, you know, how well we're doing our jobs as trainers. Um, another component of the training program is peer feedback from each other. Um, so we created a program for, uh, our tech support engineers to flag, uh, customer tickets that other tech support engineers worked on where they would say, uh, and, and kind of leave an internal note. And, uh, it pings a special channel in our chat room and it says, hey, um, Jamie, I saw you worked on this ticket. Did you know that, um, you could have used this other tool? Uh, did you know that you could pull these logs and parse these logs in a different way? Um, and so you actually see a lot of this going on all the time where tech support engineers are training each other, uh, on how to do things. And we call these training opportunities or learning opportunities, or we call it peer feedback where you're just saying, hey, um, I would have worked this a little bit differently, or, um, you have to follow this process, right? Yeah. That also helps us by finding trends in things that may be missing from the team as a whole. So we get these peer feedbacks or training ops, and if we're getting a number of them over weeks regarding the same type of product, then we're missing something like maybe people need a new refresher because it's been a year since we taught them on X, but then it could be just a thing that's been totally missed. And it's a new thing that the, uh, that just no one has knowledge about properly because it's not written down. It's just the grandfathered knowledge in a couple of people's minds. So these things come up and then we create new trainings based on that feedback as well. So it all funnels back and we make sure we improve in every aspect based on people training themselves and training each other via feedback in that way and with our own surveys, et cetera. So all funnels back, tries to improve the whole program as a whole as well. Yeah, it's really critical as we scale, right? We're 90 people now, you know, we're going to be a hundred something, you know, keep going, we'll keep going on our trajectory. We keep hitting our plans. We're going to be hundreds of people and we're going to be, that's a lot of people for our little team to train. And so we have to rely on each other and our own expertise within the team. Uh, and so, yeah, sometimes we see these training opportunities and it's just a small thing, a little reminder, Hey, friendly reminder, Jamie, did you know that you could use Crossbow to do this thing? Sometimes it's a, like you're saying, it's like, Oh wait, this is a gap in our documentation. Um, let's go look at that runbook we wrote on how to solve this customer. Oh, that's missing from the runbook. We better add this. Um, or sometimes it's like, we have to create a whole new training on this. No one knows how to do this. Or sometimes it's like, maybe we need to send a memo out to the team on a reminder because this is coming up. So, um, yeah, that's a big component. I'm relying on the rest of the team to come sort of flag each other's in addition to the very structured, you know, hands-on trainings we give and the online trainings, the e-learnings that, uh, our e -learning developer, Scott Jones is building. Um, we have, you know, we rely on each other to do sort of smaller, uh, you know, little remind friendly reminders and to surface knowledge gaps on our team. Right. So this is another form of needs analysis. Um, so, you know, one of the challenges for our org and everyone else in our team is how we've been handling the transition to full remote work. Um, you know, it was great and we really liked all being in the same room together. Um, and it really facilitates, uh, knowledge transfer. Um, and I can just lean over and say, Hey, Jamie, um, I've got this ticket. Have you ever seen this issue? And you just say, yeah, I've seen this issue. Here's, here's the doc you need to follow. Um, and that's a lot harder. So we've been coming up with some creative ways to get around that and to try to provide that type of experience and solutions. So one of the things that we have is just a team hangout, just a video chat like this, um, open all day with the whole team on it. And we're trying to really foster a culture where it's okay to ask questions. It's okay to ask beginner questions. There's no such thing as a dumb question. We were all beginners at one point. Um, but we break out into smaller rooms. Sometimes we actually just have all of our new hire group, um, people who are new, um, in a separate room where they may, might feel a little bit more free to ask, uh, you know, beginner questions. And we staff those rooms with our, um, senior support engineers. People have been here a while and have more experience. Um, our team leads who are our most experienced technical people. Um, and so we have sort of a more casual program set around that. Um, and we have special chat rooms, um, because we're very, of course we use chat applications and we have certain chat rooms that we use for certain, uh, cases. Like we have a questions room where it's just like, ask any question, ask me anything. And we have people watching that and we have video conferences like this. We have actually onboarded several classes, um, of people now having never met them in person. Yeah. And they went through the interview process. They have gone through the onboarding process. They are now, you know, really productive members of the team. And it's been a challenge for us. So I'll, you know, be honest to figure out how we're going to, uh, you know, kind of foster the culture that we have and that we had around, you know, being a really cohesive team with good chemistry and that, you know, can answer any kind of question. Hey, how do I do this? Oh, you just do it like this. Um, and so a lot of that, we have check-ins with our new hires daily. We have like a new hire hello. Um, we do, uh, after work we have little, like, you know, happy hours, hangout rooms where they do, you know, I know that the UK team does sort of like a pub quiz happy hour after, uh, after work, um, once a week, um, when they have kind of a mock pub, um, online where they ask fun questions and it's just getting to know each other. You know, we're just missing that these days, uh, that human component. So, um, we're building that, uh, as much as we can and, uh, through mostly through video conferences like everyone else's and extra chat rooms. Um, has it been, has it been challenging to deliver? How have you found delivering some of the more hands-on technical training remotely as opposed to sitting next to someone with their laptop? Yeah, it's changed. Uh, yeah, quite, it started off quite difficult, but after a while of, uh, it's been going on a while now, but the beginning was quite difficult and I asked the feedback during these sessions to ask how should I, should I share my screen? Should we do dual screen share? Shall we use this program or this program? Cause there's some programs where you can, they can share their screen and I can annotate on it, for example. So in the end I used a mix of these different methods and asked for feedback through the, you know, the first COVID class that's called them just to make sure at the beginning we added all these new things based on the feedback really early on. Cause I kept thinking in my mind if it was me starting this new job, right. And I've never met the people in real life full stop. Like I had the interview fully remote. I then started the job fully remote. That's in my mind. That's crazy. I don't think I'd be, I don't know how well I would have handled that. You know, like you need to learn with people. So we tried to facilitate that with early feedback and different GOs, like Europe did different things, APAC did different things, and we like molded them into a thing, which we then exploded worldwide quite quickly to make sure we covered every Avenue to help each one of them in their different ways of learning and needing to ask questions like Shane went over. So some people don't like asking a massive room. Like when you're new, sometimes you'd like to be in the small room just to ask someone you're familiar with, like me, like I put myself out there to say, add me in DM. We have our own separate rooms for a couple of people where they ask me direct questions. And sometimes they use that, or they use the video. So it's been the different, the difficult part is me feeling like I've done enough to facilitate their learning in the best way. Right. But I'm constantly asking them, does that make sense? Is this okay? Like just to reinforce to them and myself that they're asking questions. So that's the thing I've been pushing more is feedback, but we keep going back to feedback, but feedback is the pinnacle of training improvement, right? And making sure things go the best way. Yeah. And that, that's a great point. And I should say, you know, we're, we're doing our best to train people and build, you know, material and content and deliver, but I have to say that it's the people that are learning. It's our tech support engineers that have to learn. We're, we're trying to train them, but I have to give my hat, you know, hats off to our new hires and everyone else that has gone and learned how to do this. That's something that they did. So it's, it's Jamie's hard work, you know, delivering technical training, and it's the learner's hard work of learning how these Cloudflare products and features and the tools and the processes work. So, you know, I I'm just perpetually impressed by the people that come through that we work with, and they're just, you know, dedicated and hardworking and curious. They want to know how things work. They want to know how things don't work sometimes and how to fix them when they don't work. And so, you know, it's just, it's, it's amazing group of people. And it's just an extremely talented team that we've put together over the years now. It's just very diverse skill sets, diverse experiences has allowed us to, to really build an amazing program that trains people to do this job. And it's not just the tech support engineers that we train, you know, we've, we've actually developed some specialists on our team. We've had to specialize. And at one point, it worked out to, we can all know everything. And then at a certain point, it was like, well, we're all going to know a little bit about everything. But then we're going to have some people specialize in some areas, and some people specialize in other areas. And we've actually naturally broken into other smaller teams, we've created an escalation engineering team to handle our escalations to engineering, before we enter, you know, we're not going to escalate directly to an engineer, we're going to escalate to a team on our team of our highly technical experienced people that know how to troubleshoot and debug and fix servers that we escalate to. We've created a product specialist team to help us build, create content, public facing content, internal content, to help lots of things. We've created a training team, right. And we have to train the trainers to be good trainers. We've created a support operations team, who's built an amazing suite of services that are automations and tools. You know, so we have managers, team leads that we need to train on how to be managers. And so, you know, we've developed all these specializations and career paths, so that you come into tech support engineer, after a few years, if you're in the manager, if you're the manager type, then there's the team lead path. If you're in like us, and you want to help other people grow, and you like teaching, then you can join the training team. If you're really into product, you can join product specialists. We've got one last question. I know we've got three minutes. Yeah, another question. If you want to grab that quickly. Let's do that. Let's take questions. Callers. How do you tell if your training efforts have been successful? What gives you satisfaction as a trainer? One minute, say it in like a minute. So we can read. So the big one for trainers, for me, at least, speaking for myself is the aha moments. You're training someone, and you can see it sometimes. Yeah, they get it, it clicks in their brain. And I get it now. And they can do something that they couldn't do before. They know something that's, that's what they live for. Seeing people go off and become successful. Other people go off and excel at this job. That's their hard work. But, you know, we, we, we helped them. There was some, you know, we gave them some help. And we just love that. So we like seeing, at least six feet from myself, I like seeing when other people are successful. I like aha moments. And I'll just like to add that my satisfaction as a trainer comes from when people understand what I'm explaining to them, right? Like Shane saying, that is the best part. I'm, I'm constantly analyzing them with my eyes, like in the video chats now, because it's all remote, because you have to try and get the feedback that they are understanding when they get it. And then I asked them a follow-up question and they understand it fully. Then I feel great, right? Because I've done what I tasked to do as the opportunity, the learning objective is complete. So I'll leave that there because we've got hardly any time left, but thank you for the question. Thanks, Jamie. You're amazing. It's amazing to work with you. Shout out to you, Jamie, technical trainer, our trainers around the world, Scott Jones, our LMS admin, e-learning developer, doing an amazing job, our whole team. They're going to cut over to the next segment pretty soon. It's been awesome to talk to you about the tech support engineering training program at Cloudflare. We've got some more segments, great segments. If you want to see me later again, talk, I'm going to have my daughter and another parent at the company's son. We're going to talk about Cloudflare to our nine-year-olds. Check that out. Welcome to Cloudflare TV. It's really fun. We hope you enjoyed it. Thank you for all the questions. It was great having questions come through and asking us to answer them for you. Awesome. Good conversation. Tune in to Cloudflare TV on the next segment. It's probably good. I'm going to watch it myself for a little while after I get a coffee. All right. Take care, everyone.