How We Train Technical Support Engineers

♫ Upbeat Music ♫ Hello, Cloudflare TV. Welcome. My name is Shane Ossa. I am the Technical Support Training Manager for the Support Team at Cloudflare. And welcome to Episode 2 of How We Train Tech Support Engineers at Cloudflare. Last time, my guest was one of our technical trainers, Jamie. And today, I have the pleasure of talking with another colleague and friend of ours and mine, Scott Jones, who is our eLearning Developer and LMS Administrator. Hey, Shane. How are you doing? I'm good. How are you? Good. Good. How long have you been at Cloudflare? I just passed my two years about a couple weeks ago. It's been great. It's been amazing to see the training department grow and certainly the C-Sup grow come a long way. Cool. Well, we love having you and you've been amazing. And so basically, for everyone watching out there, we're going to talk about how we train tech support engineers at Cloudflare. The tech support engineer team is over 90 people globally. We have people in all of our main offices, San Francisco, Austin, Lisbon, London, Munich, and Singapore. And we provide 24-hour tech support to all of our customers, phone support, email support, chat support. And we support our customers on how to use Cloudflare products and everything. So as you can imagine, it's a very technical job. The people who do this job are amazing and really hardworking and really dedicated and intelligent. And I admire them all. And they come to us with different skill sets and experiences. We've built a training program that helps enable them to become tech support engineers at Cloudflare. And we're going to talk about that. If you watched the last episode, we talked a lot about the overall program, and we're going to talk about that again. But we're going to double click a little bit on the e -learning and remote element of the program, which I think is particularly interesting and relevant these days with pretty much everyone working from home. Yeah. Yeah. That's been one of the most fun things about working with this team, because everyone is so skilled to begin with and intelligent. And so it's really fun to build training for people who are already that sharp. It's also a challenge, because a lot of time it's trying to figure out how do we train something that they don't know or that reach across the different skill levels. So it's a great challenge. Yes. I mean, there's some early decisions we have to make with regards to what content to cover. There's Cloudflare specific stuff. We know we need to cover that. But then there's general stuff that's relevant and valuable to the job role, such as using Git or how to use Curl, the command line tool, and a bunch of other things for which there's a lot of training that already exists online. And we don't want to recreate the wheel, and we want to make sure we're using our time and resources wisely. So there are times when we point out to other training that's really good that we review and vet and recommend, and there are times when we're going to say, hey, let's build this internally. I started at Cloudflare about three and a half years ago, and I was brought on to help create training for the support team. We knew that the company was growing fast. We knew that the support team was going to grow fast correspondingly across all regions of the globe. And so we set about building a program to train tech support engineers on how to do the job. So when we started, we did have a good skeleton start of a program, and I want to give a shout out to the early employees, some of which are still on the team, some of which have moved to other teams internally, some of which are no longer at Cloudflare, but they'll always be part of the Cloudflare family. But they built some trainings that we sort of modeled the early program on. But we had some outlines. We had some calendar events that had text we could copy, and this is what you need to cover. But a lot of the training, what we started with at the beginning was, hey, sit next to this person. They know a lot about this thing. Watch them do what they do. And so we started from there and with a couple of course outlines, and we went about building courses, modules. At one point, we decided that we should do online training as well. So in the industry, in the training industry, we call this blended learning, right? So you give some training hands-on in person. It was in person. Now we're doing hands-on via Zoom. Virtualized. Yeah. So the blending is some hands-on and some stuff that you watch. And you had and have ample experience developing and administering that element, and that's one of the main reasons we brought you on. Do you want to talk a little more about that now? Sure. Shane's been a little modest also. When I came on, he already had a great, huge curriculum, and it's really nice to come into a job and know, wow, I have steady work coming up for a long time. So it was mapped out well. The idea behind blended learning is it frees up your live trainers. Basically, you don't make the trainers go through doing the same routine stuff over and over. You take that routine stuff and put that into a static, more static e-learning type environment so the trainer can then build off of that. So they always know these are the foundational things that always get covered. They know that they can trust it's been covered and the message is consistent. And then they are free to improvise, not really improvise, but be more creative with what they want to do and more flexible so that they can really tailor to an individual learner's needs and not have to worry about making sure they cover this same ground over and over again. Exactly. So to that point, we found ourselves giving the same presentation over and over again, and we had good slides, and we had a good script, and we delivered it, and we anticipated the questions, and we said, well, let's take this presentation and turn it into a recorded presentation as sort of your minimum viable product. Sometimes we might call it an MVP. It's just a recording of us kind of voicing over this presentation, which we gave over and over again. I might say something like introduction to DNS, but the Cloudflare specific version of DNS and behind the scenes in the back end, right? Not the intro to DNS that you could find on the Internet, how DNS works. We're talking the type of training we tend to give is very Cloudflare specific. So we give this presentation over and again, let's record it. Scott is really great at taking it a step further and then developing it into an interactive course. So you mentioned a great use case of the e -learnings is to kind of take this repetitious training presentation that really we could have recorded, and then that allows our trainers, instructors to give hands-on, and you mentioned that's catered specifically to the learner's needs, and that's a huge component of our program is trying to really, and that's also a big challenge, is trying to make sure that we give each person the training they need because everyone comes in with slightly different experience, job experience, slightly different skill set, and so we have this kind of standard training, and then we try to find out what does this person need to know, and so they watch e-learning, but the other use case of an e-learning, online training, is to be able to track the completion. So we have a curriculum or learning path that contains multiple trainings and courses of e-learnings, and then we can see when people have completed them, and what Scott is really great at is taking it a step further and developing them into interactive training so that it's not just something, you know, we've all been there, and you're watching an online training, a corporate training, and you know, it's a little, you know, mechanical, and you kind of are just waiting to get through to the next screen, but we've noticed, and it's a common, studies have been shown that if you make this interactive, and people are using their brain, and they're engaged, and you have them click over here and match this and interact in the training, then you're engaging their brain, and they're able to remember what you're trying to teach them better. Yeah, you're engaging different parts of their brain, and addressing the different ways people learn. You know, there's auditory learners, there's visual learners, and there's kinesthetic learners, and so if you can address, touch all those ideally in a course, that's the ideal. One of the other advantages of e-learning is it's always accessible, so somebody can go through it, and maybe they want to refresh themselves on some of the course information they needed to know before. They can always retake the course, you know, use that, and supplement with the run books, and things like that, so it's, it establishes the foundation that they can always go back to. One of the nice development processes we've been using is set, Shane set up a more of a staggered kind of development process, where we, like you said, we started out with an MVP, then moved to more complex. It's, a lot of it is necessity, as you know, we wanted to get a full curriculum as quickly as possible, and now we're to the point where we have 100 plus courses in there. Yeah, and we can start doing more with the interactivity. That's where the, a lot of the instructional design kind of challenges and fun comes in, is, you know, you don't want to just build interactivity for the sake of interactivity, right? You know, everything in the course should have a purpose, and should, should support one of the learning objectives that you set out at the beginning of the course. Right. You can just, you know, I know we've all been in those kind of corporate courses, where somebody just sort of, you know, oh, well, okay, do this, but, you know, you wonder, why am I doing that? So, it's trying to find what's relevant, what makes the, supports what they're going to do, reflects what they're going to do. Right, exactly. And so, that's, that's where it's, it gets more fun and more challenging. Yeah, you're, yeah, that's right. And we, we created a wish list sort of early on, you know, once we did our needs analysis, and we looked at, you know, the job, and what the job entails, the tech support engineer job entails, you know, what tools you need to use, what knowledge you need to have about products, you know, what processes you're going to need to follow. And we created this list, okay, we're going to need a training on this, and this, and this, and this, and this, and this, and this, and all these, like you said, hundreds of things we know we want training on. And so, we kind of took an approach of, like you said, let's get MVPs out for all of the really required things, and then we'll go back and make more interactive, more robust versions of the ones that it's worth investing that. Exactly. And so, that, that is found to be a pretty successful approach. We, our trainings are, are on, I like to think of all of our trainings as sort of following, falling in to these four categories, basically. We're either giving a training on a Cloudflare product or feature, how it works, what's the back end, how to troubleshoot it, our team's processes, our policies that you have to follow, the tools you're going to use, command line, git, other internal tools that we have at Cloudflare. And then the big, the fourth category, the big category is just general subjects and concepts or technologies, things such as DNS, SSL, TCP, HTTP. These are all things we train on. So, tools, processes, Cloudflare products, and just general subjects or technologies. And so, all the trainings we develop fall into these four broad categories. The way we like to do it is we like to prime people by having them watch an e-learning that's developed by Scott first. So, they might watch this one on intro to DNS, that kind of covers the basics, the FAQs, so that then when they come to the hands-on training, the trainer can maybe review those if they have questions, but then get them hands-on, practicing, doing things what they need to do, actually on the command line, actually using the tools we're going to, they're going to be using when they're going to help our customers. Immediately doing relevant, you know, activities relevant to their job. So, that's the kind of one, two punch that we try to, you know, replicate over and over is watch this beforehand. It's going to prime you. It's going to, you know, take a, you know, give you the baseline basic information, and then we're going to do a hands-on one. And since we've covered the basic stuff, the trainer doesn't have to use their time covering the hands-on, like the basics. And one thing that's kind of exciting right now, we're kind of moving into a newer kind of middle ground. We're going to start exploring, because Jamie, who was on last week, has been building test environments that practice environments. And so, we're going to start working on integrating those into some of the learnings as well, where it's appropriate, where people can actually go in and simulate pretty much live experiences, troubleshooting, things like that, that normally or would have had to be mocked up in e-learning. That just takes a lot of extra effort, and it's static, and it's not really, you know, realistic. So, that's kind of an exciting sort of new area to start moving into, and that bridges the movement from the online training to the hands-on training even more completely. So, that's definitely another aspect we're trying to build in. Yeah. I mean, really important components of our training program. One is that, you know, Cloudflare tech support engineers actually spin up test web servers that have Cloudflare services and features running, a test Cloudflare account. And so, we use this for two things. We use this, one, just for issue reproduction. You know, it's very important during troubleshooting to try to reproduce the issue that the customer is reporting, so that we have these reproduction environments, so they get that out of it. But they also just get the experience and the training, really, going through what the customer went through in setting up that Cloudflare feature on their test server. And they also get that training and that experience, the customer experience. You know, we have them follow the documentation that our customers follow, and we also drive improvements to that through via that process as well. So, in addition to that, is what Scott just mentioned, as you mentioned, is the test servers. So, it used to be that when we trained somebody, we wanted to train them on how to troubleshoot a 5.2x server error, right? And sometimes, we would just hope there was a customer ticket with that error when we went to train them. And then we got to the point where we're like, well, I have a test server. Let me just misconfigure it to throw this error, and then I'll have you debug it. And then what we said is, well, let's just have a suite of test servers throwing specific errors that we can have our tech support engineers debug. And we'll make this part of our curriculum. And each training by subject or by product will send you to this test server, which you then debug. Once you've debugged it, you can fix it. We can let you SSH into the server, fix it, show that you fixed it. And that also gives us a layer of confidence around your qualifications, right? We want to qualify people to do this job. And how do we do that? We do that in several ways. We observe. We do ticket QA. We say, hey, can they actually do a ticket related to this customer issue? We might look at the server and say, have they completed that activity where they were able to debug and fix that server on their own? It's a huge project. It's a massive undertaking. As you know, we all are super busy training people every day, working with our team every day to improve the team overall. But we also have these side projects that are building towards having a more robust program that enables us really to ramp up our support engineers more quickly, to try to bring in the time to productivity as much as possible, to try to broaden the skill set, to upskill our team so that we are capable of solving more issues, to try to escalate fewer issues to the engineering team. As much as we can keep in support without engaging other teams, if it's something we can fix, we want to enable our team to do that. That's just another one of our tools there. One of the things that I really do like about our art department and our program is the underlying philosophy that we're trying to develop people and make them better and bring them along. I've been in other situations where it's almost a gotcha situation where, okay, we get them in and we continually watch them until they do something wrong, and then you catch that and point that out. Whereas here, we're building people up and we're bringing them along, guiding them. It's a good, it's sort of a surprising thing. I've talked to a lot of the new people and the more experienced people on the team and to a person, they'll say, you know, it takes about six months to really get your feet solid underground. But all the way along those six months, you're supported by the training program, you're supported by your coworkers, and everybody is pointed in the same direction. I think that makes a huge difference. People aren't afraid to make mistakes. People aren't afraid to ask questions. They're encouraged to, actually. A lot, they're encouraged to. That makes a big difference. Absolutely. I mean, it's a culture of learning and a culture of curiosity that we've been able to cultivate. That's definitely something we look for when we're hiring people, is people that are really interested in learning. We have a training program, but hats off to the tech support engineers and all the hard work they've done to learn the skills necessary to do this job well. The training team aren't the only people that are involved in training. To recap, the training team is myself. I'm the manager of the training team. Scott, who's the LMS admin and e-learning developer. We have three technical trainers. We have one stationed in every geo. One for the Americas, one for EMEA region, Europe, and one for APAC, Asia Pacific. All of them were tech support engineers who were interested in teaching and helping other people grow. They came on to the training team. The training team isn't the only, we're not the only people that do training. The team leads do training, but like you just referenced, they're training each other all the time. They're sharing knowledge. Some of the components of the training program as well involve just building some of the structure to try to get more knowledge sharing. This involves creating job aids and documentation. This involves creating little programs like one we have, we call training opportunity, or peer review, where another teammate can flag a ticket and say, hey, did you know that you could have used this tool to solve that more quickly? Or did you know that you could have pulled the data from this database and you would have found the answer? Everyone is really helping each other. It's just amazing to see the team really engaged on that front. It's not just us. We're doing what we can, but it's really the whole team's effort to help each other grow. It is a culture of growth. You just have to. Cloudflare is not a one product company. We pride ourselves in having lots of different product lines, lots of different solutions. That's a big challenge for a tech support department. How do we support dozens of products and hundreds of features beneath those products and different customer segments that are entitled to different combinations of these products? We do have to layer on that knowledge. To speak to the program, we do start with a boot camp off the bat. All Cloudflare employees start with a Cloudflare orientation week. After that week, if you're on the customer support team, which we call CSUP, you do a CSUP boot camp for two weeks. We have e -learnings and we have hands-on trainings. Another thing we do a lot is something we call shadow sessions, where we have a support engineer actually sit next to a new support engineer, sit next to an experienced support engineer. The experienced support engineer shares their screen and narrates their workflow and talks about what they're doing. So, it's not just e-learnings and hands-on trainings. We want to give the people the on-the-job training to actually see how other people are doing it. We rotate who the shadow hosts are so that the new hire, the trainee, gets exposed to different methods, different styles. They also get to know each other. It's a really important facet now, especially that we're all remote. I think one other part that gets built into that as we go along is we're also training them to train themselves and to look for the answers themselves. We don't expect everybody to know everything about every Cloudflare product there is. It's kind of impossible. But we do encourage them, drive them, teach them to look for that information where they can find it in the runbooks, in the dev docs, things like that, and each other. But it's sort of a philosophy of if you have a question, we kind of encourage whoever they're asking that question to say, what have you looked for? Have you looked for the answer yourself? Did you find anything? It just gets a pattern going where, okay, they know their information is there. If they don't find it, A, okay, they can get the better information, and B, okay, we discovered a gap in the information maybe, and maybe we need to do a new runbook or a new course or whatever. So, we're constantly developing. It's a really nice sort of symbiotic cycle that everybody has. The information grows, gets better, gets more broad and in depth. That's such an important point. There are so many products and features that it's impossible to have an expert-level knowledge of every one. Everyone has to have a baseline-level knowledge of everything, kind of a minimum knowledge of everything, and then we start to specialize a little bit over time. I could see us specializing a lot more and more over time as we grow. But yeah, it's not just these hard skills that we're training on, how to use this specific tool and how the backend and the code works behind this Cloudflare feature. But like you're saying, there's some other processes like where to go searching for information. First look here, then look there, then run this test. There's other soft skills as well that we try to train on. Communication, time management, writing is a big skill. I don't know if I would consider writing a soft skill, but it might fall into that bucket. Empathy, customer service. There's other things that we need to train on as well. We tend to get people that are really good at this stuff already, but we also have to kind of review. And we do a trainer training for everyone to kind of, like you mentioned, hey, if I ask you a question, don't just give me the answer. Help me learn how to find the answer myself. Help me learn how I can test that theory out and maybe reproduce that customer issue. And one thing that we try to build into any kind of training, and just in general, this is sort of a general concept, is if you can find a way to get the learner to explain it in their own words, explain what they just learned in their own words. They hear their own words a lot more than they hear a trainer's, the voice on the e-learning, anything like that. And that builds into whoever leads a shadow session or whoever is answering a question. If you regurgitate, sorry, but if you regurgitate the concept, it means it's embedded better inside of you and it's going to be retained. So as much as that can be built into the program in a whole bunch of different aspects, we try to do that. It's the show your work model as well, right? Because you can come in and say, oh, that's how you got to that answer. You arrived at the answer, but let's talk about another way you could have arrived at the answer maybe more efficiently or the official process, right? One thing we've been doing, which I really like, is our knowledge shares. Right. Sort of informal, get a group together, maybe over lunch, whatever, and talk about either specific cases that were interesting, things they did or learned, discuss different techniques. Troubleshooting is an art. It really is. And everybody does it a little differently. And they can learn from each other. And sometimes, especially now that we're all distributed and virtual, that's a little harder to do. It's easier to do when you're sitting right across from somebody and say, oh, how do you do this? But it's great when you hear somebody say, oh, I didn't know you could do it that way. That's really cool. It goes on like that. So it's a constant atmosphere of learning, really, and maybe not obvious where it's beat into, we're going to learn today, but it's more, it's just a natural part of the whole situation and process. Yeah. Yeah. I mean, we do knowledge shares. We do micro learning opportunities. We're trying to facilitate more and more because our team, I mean, our team is so intelligent and motivated and they're all really great problem solvers and they all are always developing their own little tools and processes. People are creating scripts, bash scripts to automate things. People are creating their own little web applications behind Cloudflare for just making everything more efficient. And so, or maybe they found a cool third party website for troubleshooting, or maybe they found a new browser extension that helps with the headers. And so we need to share that knowledge across the team. And since we have 90 plus people spread out over the world, this creates a big challenge for our department. It's not just this program that we have that has a bootcamp and then things after that and milestones and QA, but there's also just this kind of tribal knowledge or, hey, I wrote this cool, I'm using this alias now and everyone else should know about this. And it's difficult because our tech support engineers are working on helping customers all the time. And so we have to build in time for them both to work on their, developing their own skills and professional development and their career paths, but also we have to build in time for them to just share this with each other. Because if you think about all the different skills and experience we have as a team, we need, we're trying to facilitate the whole team benefiting from everyone's little tricks and tips and shortcuts for things. So we do monthly knowledge shares. We do lunch and learns. There are a few coding clubs that happen and we're trying to facilitate more of these and structure them as much as possible. But in many cases, there's some times we don't want to structure things. We want to keep them fairly casual and just do them ad hoc as well. So there's sort of a real balance there to saying we got to do knowledge shares. We got to do micro knowledge, micro learning opportunities, but we can't make everything so structured that it creates a lot of friction for people to share. If you have something to share, share it. And we try to create little avenues and venues for doing that. And we're really the type of team that you see people innovating and creating really cool tools all the time. And so we're just, we're, you know, at times we're just trying to help bring those out and share them with the, with the team. One thing on a personal note, like Shane, I come from an instructional design background and not necessarily a technical background, but I've done training in a lot of different fields. One of the things I love about being instructional designers is it gives you the opportunity to always be learning these things. And working here is like being a kid in a candy store. It's there. It's always something new I'm learning and finding out about that. Wow. I had no idea. Recently we had a little presentation on deterministic failover, which I thought was just fascinating. You know, it's something that just happens. And one of our escalation engineers gave a presentation on that. And so just things like that, that, you know, maybe you don't use it right away. You don't, you know, it's not something you need to do every day, but just these bits of information just come in and you just, it's been wonderful for me and for what I do. And then just, I think a lot of people on the team find that too, because they, we do hire for those basic qualities and we're really good at that, but everybody comes in with a little bit different angle of knowledge, angle of perspective, and they're all learning as well. So. Yeah. I mean, if you're, if you're a curious person and you like learning, this is a great place for you. We have an endless supply of things to learn about. You know, you can really go deep on any of Cloudflare's core features or any of our, you know, not core features or other features and really build up expertise on them. So there's, there's just no shortage. And in fact, sometimes I feel like that's a real challenge for us in some ways. And for the tech, the average tech support engineer to go, okay, well, I have to learn enough about every Cloudflare product and feature to be dangerous, to help a customer that writes in. But then where do I go next? You know, what, what should I, what should I focus on after that? And that's one thing we were trying to get better at. We help people on this. We're trying to get better and better at pathing people. Maybe it matches up with their career path as well. Their career goals at pathing people and, and, and, you know, getting the deep technical training that they need on certain subjects to, you know, to help with the really tricky customer issues. But, and, and to help us be able to provide a better customer experience. So there's so much to learn that there's more stuff to learn than what we have time to learn about. So a lot of it is about prioritization and time management, building in time to go, well, we have to learn about this now. I can see that there's a question coming in from the audience. And if you have questions you can ask, there's an email address listed on the website here. I can see one that says, how do you deal with new products being released and preparing customer support to support X new feature? That's a really good question. And that's a big challenge, but we work really closely with the product management team, the PMs as we call them for short. We have another team of specialists on the customer support team, which we call the product specialists that was sort of born out of our subject matter expert program. Technical trainers, especially the ones we have are experts on a lot of things. And it's just, there's Cloudflare has so many features that it's hard to be a very advanced expert on everything. And so we do rely on SMEs and we know that one person over here is an SME on DNS. Our technical trainers know a lot about DNS, but if they want to know a little more and they want to work with someone over here to develop deeper training on DNS, they need to go work with one of our DNS SMEs, subject matter experts. Now these subject matter experts are also tech support engineers doing the tech support engineer job. And we realized it was a lot of work that we actually had a huge body of work to be done in the SME role. Part of that is helping the training team develop and deliver really valuable, deep technical training on a subject. Part of that was working closely with the product management team to ensure that the support team is enabled and prepared ahead of a release or a launch. So we work with the PMs and now we have product specialists on our team that were tech support engineers but are now dedicated to helping with a variety of tasks like preparing us for launches and releases. So we do have sort of a SPAC or a policy that says, here are the milestones as we get closer to a GA, general availability of a new feature, we require a training a week beforehand and we require something else two weeks beforehand and a heads up email three weeks beforehand. So we have a defined process that says if you're going to release something. Now that's for big releases, not every release is a big release and the people that are watching this might know and work at other companies where most releases are small releases and they're not necessarily customer facing, you know, impact. At least we're not anticipating there to be impact. So a lot of it is just like facilitating really good communication between the product engineering team and the support team as to, hey, this is what's coming down the pipe, this is the next thing we're going to release, it's going to be a new feature. In some cases we'll do a dedicated session where we'll have the trainers and the SMEs and the product specialists get trained by the PM and the engineers so that we're like training the trainers and the experts. Then we can be the kind of SMEs for the rest of the team, determine what the requirements are for the rest of the team to know around a release or a new feature and build something there. But we really love to do like enablement sessions where ahead of the release, well ahead of the release, like around the time of a beta maybe or an alpha, we get access to the feature ourselves, we enable it on our test accounts and we play with it. And not only do we get training and learn how this feature works, but we're able to give the PM team immediate feedback and our tech support engineers are the most in touch with the customers and the most in touch with the customer experience. So they're usually the best people to give really good input to the PMs as we are testing out this feature, we're getting trained on it ahead of a release and we're also saying, you know, if you made this, if this menu had a description here, you know, or then we'll see less tickets, our customers will have better experience and we'll see less contact about that. So there's definitely a lot of facets to trying to be prepared ahead of a release of a new feature and it depends on the size of the feature and the impact, but we work really closely with the PMs and our own internal experts. I think one of the aspects and things that the training team specifically brings to new product release and things like that is applying the expertise and experience that we have as far as how do you present that information? How do you present the new information? What do you present first? What is the foundational? What do you build on? Because otherwise, you know, we've all been in situations where it's just a straight knowledge dump and somebody comes out and just talks straight for 30 minutes and you remember about a minute of it. So there's concepts that come in as far as, okay, let's layer that information in. So you need to know this, then you can know this, then you can know this, and it's determining that. That's where a lot of the fun and a lot of the real challenge and experience comes in that Shane and I tend to geek out on. We talk about that kind of stuff a lot, and I get that kind of information a lot from the trainers and the other people. It's like, okay, what do you need to know in order to do this? What's the key things? And then you can get into the more detailed and more under the covers kind of stuff, but until you know the basics, you can't really do that. So I think that's one of the more important things we can bring as far as the training team to do all this is make it so it is absorbable. What does the tech support engineer need to know? We have to contextualize this and make the information they're getting valuable, but to be blunt, there's just so much information. It can be information overload, and there's a lot of channels that we're getting information from. Our documentation tools and our email, of course, and our chat rooms, and hey, don't forget, you need to know A, B, C, D, E, F, G, and really, maybe the tech support engineer doesn't need to know A, B, C, D, E, F, G about a subject. They only need to know C, D, and F, and so we do try to refine it so that it's not like drinking from a fire hose and, oh my God, I remember all this stuff, but it's not valuable. I remember some things and not others, but it's like, here's what you need to know. Oh, and if you want to learn more, this is where you go to learn more, and that kind of ties into our thing from earlier around helping people learn how to find out more for themselves when they do need to find out more, and so we have to be really careful about how much and what information we give people early on because if it's not going to be valuable and relevant and help them do their job, we're sort of not using our time as wisely as we could be, so yeah, that's a great point. I mean, a big part of what we're doing is a big part of technical training in general is taking this kind of source material that you get usually from the engineering department or the product department and contextualizing it for the team that you're building the training for, refining it into something that's valuable just for them in their use cases, and it's something that we can help with. It's something that we do help with. It's something we could certainly get better at, but it's a big component of technical training in general. Another part that I think we address pretty well, especially, and Shane talked earlier as far as setting up the test environments and setting up their own servers and things like that, that puts our tech support people in touch with the customer experience. A lot of times, tech support people, they know stuff so deeply and you can lose touch with what the customer is actually experiencing. Yes, we have some very, very technical customers. Our enterprise customers are usually those people who know what they're doing very deeply. But we also have a large contingent of customers who maybe are not that savvy. They just have a job that they want to protect. They want it to work. It's not working. It helps the tech support people to keep in touch with that aspect of it as well so they can explain things in a way that anybody can understand and can help people get to where they need to. It's a great balance that our tech support people have to walk every day. They're dealing with an entire range of technical skills that people have and technical abilities. To be able to address all of that equally well, that's a huge achievement. I think the vast majority of our people are able to do that. You don't know what the customer knows. You don't know always what they're talking about. They might be using different terminology. A big part of it is trying to really get to the core of what they're asking. Sometimes it's like, what are you trying to achieve? We're asking the customers because maybe they might be asking a question about one thing, but if their goal is to achieve this other thing, they might not have actually asked the best question to get to their goal. That's a whole other process that we try to train on and that people pick up eventually in order to get more efficient. We want to reduce the amount of back and forth with the customer. It's a better experience. We can solve their issue quicker and then we can help the next customer. We want to try to figure out what their real problem is or what they're trying to achieve when they're contacting us. That's just another thing we need to train on and that we do train on, which is a communication efficiency element. It falls into the category of a process that we try to help train people on. One of the more interesting and definitely more fascinating and challenging aspects is teaching someone how to be a good investigator and troubleshooter. Like I said earlier, troubleshooting is an art. Certainly with Internet and computer things, there are so many ways you could come at a problem. It's the thought process. Capturing that and getting some of our better... There are some people on this team that are just incredible troubleshooters. Getting that knowledge out of them and sharing it, that's definitely more of a one-on-one approach, but there's definitely principles to troubleshooting and how to look for things. I think that makes the job a lot more fun for a lot of people too. It's really digging in and saying, okay, how do we figure that out? Let me find that out. It's developing those skills and the ability to do that so that it can be repeatable. That's a fun kind of... Definitely. There are some very specific methodologies for troubleshooting. Some people follow five whys. We do train on some of those occasionally, but the way we do it is we have very specific processes for troubleshooting different types of issues. Those bigger, higher level troubleshooting approaches are really important to learn about and apply, but what we do is we have what we call run books, but these are process documents that help people troubleshoot specific problems. We train on those and we consider those our training source material. To tie into another topic earlier around SMEs and product specialists and creating technical training that's refining existing information, we end up with an interesting challenge of sometimes having duplicate sources of truth. One big challenge for a training department is making sure that the training we deliver is accurate and up to date and verified by an expert. The technical trainers aren't necessarily... They are the experts, but we don't want to be the keepers of the truth. We want to be the ones that are spreading it efficiently and helping people learn it efficiently and contextualize it and make it relevant and refine it and all those things we said, but we don't own the source material. The engineering department says, this is how this product and feature works, and these are the ways you can pull logs in order to troubleshoot it. Then we might work with our experts to say this type of issue needs to be troubleshot in this specific way, and then we get that information and build a training on top of it. One of the big challenges for us is to keep our material, especially as we build 100 e-learnings and we have 100 hands-on trainings that we've developed, is to make sure that when something changes on one of those features, we have a new API back end, and that's a totally different way to troubleshoot this thing, that we have to maintain that. Do you want to speak to the challenge of maintenance in e-learnings or just in training in general, or the paradox of source of truth between training and the keeper of the source material? Absolutely. That's an ongoing challenge, is if I create an e-learning, it becomes a static point in the ground, but a lot of our products are dynamic, and they're evolving, and they change. Being able to balance what the product is, where it is, and determine how static is the information we're using going to be, and that's where it becomes interesting. We have courses that are now almost two years old that definitely things have changed, and so that's becoming almost as big an effort and ongoing effort as it is in creating new courses. We have a really good documentation group, plus people are always updating Runbooks themselves, our team is. There's tons and tons and tons of documentation out there, and so you start looking during a search on the wiki or something like that for information. You can come up with multiple articles, like Shane was saying, and some are really old, some are new. We have efforts going on to try to consolidate a lot of that, but it is an ongoing and perpetual issue. That's why we're careful about what we build static training on, and we try to make it as dynamic as possible when we can deep link to the source of truth and train, like we said earlier, on processes. Instead of prescriptive knowledge that you memorize, we're training on the process of going to find where it is and other things like that. Of course, we have to deliver technical training that is based on facts, and build knowledge and skills for people, and show them how to use tools and show them how our products work. It's definitely a challenge for any training team in any training department to work with source material that's constantly changing, especially at a dynamic company like Cloudflare, which is not just building paperclips and millions of paperclips over and over again. We have dozens of different products that are completely different, and they're constantly evolving and changing. We go to market quick with our products, and that's something we're really good at. To speak to the first question, that's something that's really challenging to make sure that we are delivering training to the tech support engineers that are supposed to support our customers when this product is built in such a rapid way that sometimes even the engineers and the product people are still getting their ducks in a row as we're getting ready to go to market, or at least the beta with a product. I see that you have a great question. This is an awesome conversation, and we could go back to it, but we have 10 minutes left. I see that you have an awesome question, Scott, which is coming from someone that says, what are some of the techniques you've used to make learning really engaging for learners, especially an audience that doesn't feel they have time for training? Yeah, I know. That's a challenge, and I think it goes back to trying to make the training as relevant and everything in it have a purpose. You don't want to – because time is precious, and people will think, oh, God, do I have to sit through this? But you want them to be able to go out of it and say, I sat through that, and I learned something, and now I can use it. It's trying to find ways to build things in where they are using the skills we're trying to train them. It gets back to what Shane was talking about, the show and do exercises as much as you can do that. It's one thing just to say, do a little video. Here's a process of doing it. You can hear somebody drone on about how to do it, but then if you need to find a way to make them, the learner, have to repeat it and have to do it themselves because people learn with their hands, learn with their fingers, and if they're just sitting there observing, it only hits a certain level. It's building that in and finding ways to build that in, and that's where working with the trainers, working with the technical support engineers themselves on what works for them, what they needed to know, what they didn't get out of a training before, and that's a huge part. One thing we've built in also, we've gotten more and more of and some pretty good review cycles we have on the courses themselves, and one thing I wish I could take credit for, yeah, this is a learning project we put together, but what's happening is the new are going through courses, and they'll identify maybe errors or things that are out of date or data that is no longer good. That's a huge learning opportunity for them because what it proves, one, is they're recognizing that the data isn't right, and so it means they've learned it really well, and it helps, obviously, it helps me update the courses. Yeah, and it's a great sort of cycle there that it's sort of an unexpected benefit that has come out of it, so. Yeah, yeah, you make some great points. I mean, I think the learners are engaged when the content is valuable to their job, right, so one of the classic ones is, you know, explaining at the outset why that they need to be paying attention, right? Like, you need to know how to use this tool in order to be able to solve this type of problem, which you're going to see every day, you know, which the subtext behind is if you don't know how to do this, you're not going to be able to do that, so if the content is engaging, if it's valuable, then they're engaged, and we try to explain the value front as well, but Scott uses content authoring software, Articulate, and I know that you've used various others, the Adobe, and there's a lot of content offering tools out there that we've all used, but right now we're working with Articulate, and it's pretty feature-rich. I mean, we're able to make content in there that requires that learners interact with the course you've built, and so, you know, we've all been there watching a corporate training, maybe on anti-bribery, and you're kind of multitasking, right? You're doing your email, and you're watching it in the other window, and it comes with a question, and you get to that question, you're not very engaged, you've taken this training 20 times, it's required, it's important, we're going to take it, but we're trying to create training that is super highly valuable to our tech support engineers' jobs every day, that they're going to be able to apply this process we trained on, or this tool we trained on, or this Cloudflare product, you know, knowledge that we trained on, and use that information to do their job better, to help the customer better, then they're going to be engaged, but Scott's really good at building, you know, interactive courses with Articulate. Like I said before, we have these MVP layers, because we know that we have a wish list of 100 trainings we want to build, and it takes time to build a really interactive one, right? So, sometimes what we do for e-learning is, an engineer comes and gives us a presentation on the back end of this feature, and how to debug it, and the logs that we'll have available, we record that, and we slap that in there, and that's our MVP, and we don't love that, and we know that our learners aren't necessarily 100% engaged when you're watching someone give a presentation, you know, the audio is funky, you know, someone asked a question, and they spent too long kind of going down a tangent, but in order to get some information available to our customers, immediate, you know, our internal customers being the tech support engineering team, to give them the, you know, information that they need, we have to start there, and then, hey, let's go back and build an interactive one on that, and like I said, we're careful that maybe that's, let's invest and build an interactive training on this one, because that's not going to change as much. I like to talk about things being stable, right? If it's stable, is this going to be stable? I don't want to build a really interactive training on something that's going to change tomorrow, because we'll be constantly maintaining it, so sometimes, you know, it's a little bit chicken and egg, or, you know, cart before the horse, and, you know, there's a lot of analogies you could plug in here, where there's challenges for our department to build, provide content to our team that's going to stay fresh, that's not going to create a big maintenance cost for us, maintenance cost for our team, that's going to deliver value and engage the learners, that's going to meet our business needs of providing this minimum information to the tech support engineering team that they need to be able to answer customer questions. So, these are all really fun challenges and problems to try to solve. One question that always comes up, and it's how do you tell if a training's been successful? Oh, yeah, yeah. And, you know, there's levels of assessment you can do. There's a, you know, the immediate level, and I know last time you had this, you talked about this, but the immediate level is a smile sheet. Did you, you know, you get that after the training, and did you like the trainer? And everybody always says yes, and let's say just had some tax to grind. But then there's the next level is, well, how much is retained? How much did the person retain? And then what's the business impact? And then you can go even further and ask a manager or something, is this person applying what they do? And then beyond that is a business impact, is did what we trained improve the overall business? And all those levels are, yeah, all those levels are are very important. And it's one of the things I really enjoy about working steadily in a place rather than as a contractor, which I've done also, is you, as a contractor, you don't get to go back and assess those things. You do your job, and you're gone. Whereas here, you can, we can really look into the impact and say, okay, this worked, this didn't, and that makes everything much stronger if you can do that. Yeah, that's a great point. And I love this topic. And we could spend the last few minutes talking about it. We have a couple minutes left. But yeah, I mean, we do a survey, of course, most people who've taken trainings after the training, a trainer administers a survey. And this, like you said, smile sheet is, you know, we have, it's a form, a web form, and they fill it out. And we ask them for their feedback and their input. They're candid, you know, tell it to us straight, we are, we want to improve. And, you know, we're not going to hurt our feelings. So, you know, what could we have improved about this course? Did you like it? Is it relevant to your job? We have a form, right? So that's your level one. For level two, we do evaluations, particularly on the eLearnings, so that you're not just clicking through. It's standard process to have an assessment at the end. And it also helps with that spaced repetition, right? To just kind of hammer in, that's how the brain memorizes stuff. And so that's our level two, we can look at evaluations. So level one, did you like the training? And what, how can we improve it? Level two, let's give them an evaluation, see how many of the questions they can get right. Level three, like you're saying, is observing the behavior. Are they able to do the thing that we set out to train them? So most of the time, this is done with QA. Can we go QA their tickets and see, now they can do these types of tickets? Or now they can't, we need to fix our training, right? So we look at a lot of any kind of deficiency on the team as a deficiency of our department, we're trying to, to get better and better. So I realized we have one minute left, we could go on forever, we're going to have to do this again. Yeah. So it's been great to talk to you, Scott, as usual, we talk a lot, but you know, we like to geek out on how training works at Cloudflare. It's been fun. And if you do get a chance to join the team, I think you'll enjoy it. It's, there's a lot of effort going into trying to make people as strong as they can be. So that's right. Thanks to everyone for watching how we train tech support engineers at Cloudflare episode two, and I hope you enjoy the next segment.