How We Train Technical Support Engineers

Hello and welcome to how we train tech support engineers at Cloudflare. My name is Shane Ossa. I'm the technical training program manager for the customer support team at Cloudflare which means that it's my job to make sure everyone has the training they need to do their jobs on the team which is a big job but I don't do it alone and I have a whole team of trainers that help me with this and today joining me is one of those amazing talented trainers Mark Chan who is based in Singapore and also joining me today is Gagan who is one of our newest tech support engineers based in Singapore. We're currently in Bangalore. Hi everyone, I'm Mark. I'm located in the Singapore office. I have been a tech support for a couple of years and now I'm a technical trainer. Hello everyone, I'm Gagan. So as Shane told I'm from Bangalore but I'll be joining Singapore office as a technical support engineer and looking forward to the opportunity. It's been two months and I think these other people who are training me and it's going quite good. Cool, thanks. This is the first time that we've had one of our trainees, one of our new people on the show. This is episode four. If you have been watching the other ones out there we're going to cover some of the same stuff but we're just going to have a conversation about what training is for the support team at Cloudflare and how we do it and so we have Mark who's the trainer and Gagan who's the trainee and myself who's the program manager and we're just going to have a conversation about this whole thing. So if you have questions out there we'd love to answer them down below. If you look down below there's an email down here that you can write into with your questions and someone will screen them and send them in to us and we'll get to them as soon as we can. So please ask questions if you're out there but otherwise we're just going to talk about our training program. So yeah I've got talking points and to talk about but I guess I'll just start with you Gagan. How's it been so far? I think it's my third organization. I think already I know the training program is quite different. So here it's in a structured way. So it's more into like first you're doing your hands on or learning yourself then your trainer will take you through how to do things and then you have a session with your senior engineers. So I think it's quite different. So I'm not sure if anyone else follows it but it's really good and also it's about topic wise so you get to learn each topic individually and you become good at that. Yeah that's great and from what I've heard seen you've been doing an amazing job already and are just flying along and that's just a testament to your work as well as Mark's. Mark do you want to talk about Gagan's progress so far or anything else to highlight about the program before we jump into it? Yes so before Gagan I had trained other new hires and from the experience of other new hires I've come across some feedback and so based on those feedback I have tried to come up with ways to improve the training and one of them was to pre-record some of the training whereby I usually do a lecture on a topic before I do the hands-on and based on feedback that some new hires may not be ready with the hands-on training and they need more time to to do more research on their own so I've recorded part of my lecture and then asked them to watch the lecture before the training then a day before the training and Gagan was the first person I've tried this on and what do you think Gagan? Do you think it's effective? I think that's good because you get to learn what you are going to do next day and you are ready for it so once I get on a session with you I almost know what we are going to do and it kind of helps you to learn more because you're already prepared of the topic and maybe what are the questions you have you can try that hands-on so that improves your learning so that's quite good. Cool yeah so we do that a lot you know where we record our trainings or we create e-learnings and we try to have trainees watch that before the hands-on session to sort of give the learner sort of the baseline understanding that they need and take out some of the repetition for the trainer who's just going to present on the same things over and over again so we record things as much as we can also because we have a global team but I'm so happy to hear that and I'm so happy and excited and honored that Mark is a trainer on the team because Mark was a team lead of tech support engineers and a tech support engineer himself for years before that so he just has a ton of experience super technical he's already trained loads of people so when you were excited in the technical training role when you were interested in the technical training role in Singapore I got really excited and so how long have you been doing training now it feels like it's been forever but it's probably been how long has it been I think it's less than a year less than a year yeah yeah almost a year maybe I have to go back and look yeah well that's cool yeah I mean and you did a lot of training before that but then we sort of brought you over to full-time dedicated training and I guess it's a good time to sort of start talking about our program and how we've designed it and how it's structured so one of the first things I want to talk about is the subjects that we train on what do we train on in general I like to say that all the training we do at on the Cloudflare support team pretty much fits into four main categories so we train people on Cloudflare products and features how they work and how to troubleshoot them if they're not behaving as expected we train on our team's processes and our policies that you need to follow pretty straightforward we train on our tools the basic troubleshooting tools you know that everyone has command line tools like curl and dig and whatnot but we also have a bunch of homemade tools that we've built ourselves internally we need to train people on so that's products processes and tools and then the fourth category is really big one really broad and probably the most challenging and interesting which is just sort of concepts or technologies and this would be things like how does DNS work not just how does it work at large in the Internet but how does it work at Cloudflare what what do we have running on the back end what you know web servers and services do we have and so we talk about DNS we talk about SSL we talk about caching we talk about how to increase website performance so pretty much all the trainings that we have fall into those four main buckets and when we think about training we're heavily focused on onboarding new people but we also focus on continuing people's professional development after the onboarding stage so it's not over when you're when you're done with with onboarding it's only really begun and that's one of the fun parts about Cloudflare is that there's so much to learn we're not just a one type of product company we have dozens of different products and so if you like to learn this is this is the right place for you because there's just an infinite amount of content and Mark you spoke to the feedback that you've gotten along the way and how you've worked that into improving the program and we can get into sort of feedback cycles and improvement processes but before we get back to that we've gotten feedback before that it is a lot of information to consume especially at the beginning Gagan have you found it how have you found it just at the start just kind of a little bit getting bombarded with with tons of information I think yeah because as you mentioned there are like a lot of technologies or different concepts to learn so you are overwhelmed at the start but then we have trainers like Mark I can watch on it he's highly technical and he's been very helpful so every topic that I take or if I'm stuck somewhere he's always there to help me out so I don't think it's that much overwhelming once you get to know it's like you're swimming through it it's kind of good yeah and some of the approach that we take at the beginning is exposing people to everything at the start a little bit of everything so we want everyone to have a baseline foundational amount of knowledge of all of Cloudflare's core products and features that you'll be supporting and then we start to see people sort of specialize in one or the other over time so you know over the next 3, 6, 9, 12, 18 months you can start specializing in areas that are you know that you're and already have some expertise in or that you're interested in or that align with your career path maybe and so that's one of the cool parts about the the team and the company in general yeah I think that's quite different because generally what happens you're trained for one month two months or a three month period and then you're left off but I see there is a way to track it so it's not that you're left off after three months you're still continuing the growth so that's quite good where even if I'm interested in some topic I can pick it up and learn so I know trainers are there to help me out yeah so we definitely front load the training of course like I mentioned before we focused on onboarding you know we really want to make sure we have a good a new hire has a good experience and that we equip them with the the tools and the knowledge that they need to be successful as a tech support engineer at Cloudflare and we can get in to what the role involves at some point too but we heavily front load this and so basically we have a boot camp we call customer support boot camp for short we say customer support is c-sup I might say that throughout this talk so we have c-sup boot camp and c-sup boot camp is after Cloudflare orientation so the company does a week of orientation and then our team does sort of an intensive boot camp where every day is a different subject and there's a lot of other trainings on like I said before processes and tools and technology and products so we have an intensive boot camp for two weeks and Mark gives tons of the trainings and there's a lot of recorded trainings that you watch and then another thing we do is shadow sessions which is our word for on the job training I've heard it called side by side where a new tech support engineer like a gun will shadow a tech support engineer who's experienced he's been here a while and and we take turns we rotate who the new hire is shadowing with so that they get exposed to everyone's styles everyone has some people use mac os some people use windows some people are using linux various distros of linux and everyone has their own little shortcuts and we want to expose new hires to everyone's little tricks and tips and it's also a good way for them to kind of get to know the team right especially now that we're remote this is an opportunity to kind of meet one-on-one with other teammates how has it been so far Gagan and you can speak to this too if you want Mark but an entire doing this entirely remotely we used to do it in person in the office right and you've been doing all these from zoom calls has it been going okay something before I take this I need to point it towards you so how how is that transformation like you have been I mean training people offline and currently everything has changed due to this pandemic so how is the change for you how different are you doing things so that's the question for you and to answer your point I think it's good to learn from the experienced people because they know their job and the kind of tools they use or for example they're not doing any closed ticket or previously on previous tickets so it's a live troubleshooting so we know how they handle and we can learn from them how to go about our job or what we need to check so that's quite helpful yes for me as a trainer we used to do it in person so I used to draw on the whiteboard and so with this current situation we have to do it remotely and you're mostly like handicapped and I couldn't draw what I want to express but however with zoom we can use the we can still illustrate on zoom but I tend to rely more on preparation before the training like what I want to what I want the trainee to do and I'll ask them to perform tasks that I want them to accomplish so I guess the biggest change is I have to adapt my style of delivery instead of drawing the whiteboard I have to prepare more more on my end yeah definitely a lot more preparation if we have good instructions written beforehand as to what we're going to cover and what what the trainee should expect and what the trainer is going to go over it's been it's been hard yeah I used to do a lot of classroom trainings and you know I think it's working out fairly well over remote I think you miss a little bit of the sort of interpersonal connection I think some of the cultural connections that you get team culture wise I mean and also there's a lot of kind of how would I call it you know it's unstructured or casual training that that occurs when we're in the same room with each other um you might lean over and just ask the person next to you have you ever seen this issue and they go yeah this is what you need to do and so it's a lot harder for us to do those things and so one of the biggest changes is just having to schedule everything everything has got to be okay we're going to do this for two hours and we're going to talk right as opposed to just casual unstructured hey has anyone seen this problem and then you know one of the senior people on the team goes I've seen that let me tell you how to do that so in terms of our training we that I think that's been one of the biggest changes and Singapore team sort of led the way in introducing a sort of structured version of that which was the ticket Q&A sessions that you're doing Mark do you want to talk about how those work yes so we have a daily Q&A sessions with the new hires and in those Q&A sessions I guess from a new hires perspective when you're working on tickets that you're not familiar with you definitely have questions and I think that's that's a good opportunity for them to ask questions challenges they come across by working on tickets and I think training is is good but as a new hire I guess you have trick you have questions on a daily basis and that really gives them a opportunity to ask their questions so Gagan you have been through some of the daily Q&A's what do you think of the daily Q&A's as you mentioned yeah as a new hire we get stuck quite a few times so it's good we have a place where we can post our queries and we know someone would answer that and it's not that same engineer replies to our queries every day we have a session with different engineers and we get to learn how they go about their business so we are each day learning different things maybe the issue was same but different engineers approach are different so we get to learn different aspects of it so that's been quite good yeah so that's one of the things so kind of in review you know we mentioned we have this boot camp program we have hands-on trainings delivered by mark we have shadow sessions where new hires shadow with experienced people we have these Q&A sessions with different experienced people from our team where we go over specific tickets versus the shadow sessions where you're sort of watching someone do tickets or you're doing tickets with them watching so as you can imagine everything being remote there's a like this almost all day all the time and we have chat rooms lots of chat rooms as well for specific uses there's a chat room for questions general questions that we keep separate to our sort of operational chat room we have one for interesting cases oh wow this was a really interesting case and now we use that as a case study for training later on and we can talk about case studies too so another component which we kind of breezed over which we can maybe go back into a little bit was recorded trainings or what you might call e-learnings and so we have a different a few different types of e -learnings some of them are recorded meetings or recorded lectures like the ones mark did and some of them are recorded meetings delivered by product managers engineering managers various systems engineers around the company and some of them are actually these sort of bespoke interactive trainings that we've built we have an lms administrator slash e -learning developer on our team his name is scott and he's in san francisco and he is responsible for developing e-learnings for our program and also sort of following up and making sure everyone's sort of completing them and watching them and so as we mentioned before it can be quite repetitive for trainers to sort of deliver the same presentation over and over again and so as you can imagine we can also track completion if we sort of turn that into an e-learning and have a learner watch that before the hands-on training so our general strategy is let's say we'll have an e-learning called introduction to dns and and the learner will watch that and there's a short assessment at the end most people have done these types of trainings for the corporations they've been in there's always a little quiz and we're not trying to trick people it's usually just to sort of reinforce the knowledge a little bit make sure that they didn't click through the e-learning too quickly pretty reasonable and then after the e-learning we have a hands-on session and this is usually with our technical trainers in each region so the training team is currently five people myself from the manager of the team and then we've got a trainer in each region so mark is the trainer for the apac region we have a trainer in the america's region and a trainer in the emmy region so the hands-on trainings are usually delivered by the technical trainers and these are designed to really get people doing actual relevant work so getting people on the command line curl using curl using dig and applying this so the e-learning sort of primes you it sort of goes over the faqs gives you that foundational baseline knowledge and then the hands-on trainings are sort of really like labs designed to to get you actually doing the type of things pulling logs parsing logs analyzing data doing the things that you're going to be doing to help Cloudflare customers and then we also do these shadow sessions later and then we do the q a sessions so those are our typical uh boot camp a typical day of boot camp and then after after boot camp ends um you sort of start working uh with customers straight away and Cloudflare customers for people that are watching whether you're a Cloudflare customer or not the way it works is Cloudflare customers have plan levels and there's a free plan level which is an amazing service for for the Internet the what you can do with your website on a Cloudflare free plan is amazing value and you also get access to the support team um not as prioritized as paid customers but you still get access to the support team and so it's a great place for new hires um to learn and and to start helping working with customers straight away so after boot camp we sort of let new hires uh start working uh free customers and responding to free customer tickets um how how did that go for you gagan i think uh it was quite good because once you learn all the technical aspects then you need to test how how was your learning or if you know about the things policies so when you get on the free q i mean there are like a lot of customers using a free plan so one advantage is you get lots of questions and it's all different so that's a good place to start learning because i don't think there's a repetitive issues or repetitive questions but most of them are unique and that gives you a good test for your knowledge and you get to learn a lot from these yeah yeah and the other part i wanted to mention was like generally the trainers that we have they're all been like tech support engineers previously so they know exactly what it is to work and how they got started so they share their feedback and that helps even more like they know exactly where we are stuck or what we are doing so that's been the change i think yeah mark i was going to ask you um when you joined Cloudflare how many years ago was it uh i think it's been more than four years ago almost five years ago um and we didn't have as robust a training program um what did you do how did you yes yes before we had this uh training program uh so we basically had to uh so uh but we do have a senior uh when i was i was the first first local hire in the singapore office uh however there is someone from the sent uh sf yeah who came to the singapore office and then he was a senior technical support and then uh later on he moved to another role so uh so i had a senior to help me to learn so but the senior has to do tickets as well so uh it was he wasn't a he wasn't a technical trainer so he has to split his time between uh doing tickets and uh and at the same time uh answer answer my questions so training was more of a ad hoc and uh and it was more of a q a so when i come across an issue then i'll ask senior engineer how to solve this issue and most of the time i have to learn on my own so uh how i learn on my own is i think i think it's a good opportunity for me as well i think true learning on my own i know how to fish instead of someone telling me how to fish instead of someone giving the fish to me yeah so uh what happened was uh in order to learn something i have to uh try out that feature and i i even have to break the feature on purpose and then try to troubleshoot it so that that enables me to learn new features on my own so uh and then from my experience i i i thought that if if uh if everyone adopt this uh approach to uh to learn something on their own uh not only can they understand how the feature work it also aids in troubleshooting customers issue because you're actually uh testing the product you're actually breaking on your own and then you're replicating the customer issue which is a very important step in troubleshooting replicating uh an issue and i think that that really helps me uh you know in in my current situation where i need i need to uh uh learn new cloud features and then i need to extend to new hires and yeah and about five years ago we didn't have as many features as we do now and we didn't have as many customers as we do now um and so we've had to get a little bit more structured with how we train because we don't quite have the same amount of sort of free time um you know not that you had free time but you had to you know go and learn these things yourselves and i think that was a great experience and we took that and we made that a component of our program so we make sure that everyone now has a working Cloudflare test account and we give them access to vps servers that they can spin up to reproduce issues and we walk people through this but we do like it to be as self-guided as possible i mean we have some guidance and we have some recommendations and we're here to help and we're here to get people unstuck but we do really like people to go through the experience themselves of creating a Cloudflare account creating a simple website putting it behind on a server not using like a you know website creation service but like actually your own little html you know file and then making sure that it's behind Cloudflare and then you start there and then you start adding more and more features and we have a checklist that we use to make sure that everyone's trying out every feature and not only do you get the experience of going through what the customer goes through when they you know start using Cloudflare and so you get that empathy but you also get some training on actually the feature itself and if you run into any bugs then you need to debug them and and figure it out so that that's become a required component if you will of our of our program based on the experience that you had and other people had as well and i had imagined when you started and you know there were a few products and there are few customers and there wasn't an official training program and you had to go learn that you're probably have heavily leaning on documentation that that existed and so that's become another really big component of our training in general i mean and any company and any team has the challenge of knowledge management and making sure that there's documentation available on every issue and that it's clearly written and it's step-by-step procedure as to how to troubleshoot something so we created a series of documents we call them run books but some of them aren't necessarily run books per se some of them are troubleshooting troubleshooting guides some of them just sort of talk about how a product works and talk about gotchas and and provide you some example cases but these are really critical especially for new hires like a gun was saying he's starts working with free customers and he's putting his weeks of boot camp training to work and he comes across the customer issue and he's gonna go uh do i remember what mark taught me um maybe maybe he did but then he's he's got to have a written document to go back to and say how do i go do that again oh yeah now i remember step one check here step two check there and so one of the things that we focused a lot of energy on over the last you know a few years i've been here three and a half years is trying to improve our documentation and just create run books on everything and and uh and it's been it's been really i think it's helped a lot especially scale us globally um now that we're all remote as well and you can't have those over the shoulder talk to the person next to you how do you figure this out and they just tell you everything's got to be documented now um gagan you've you've been working with our documentation now um how has it been so far um compared to your other experiences i think like mark said uh he didn't know he didn't have anyone to help him but i know where i need to go to fish so yeah at least i have these wikis so the most important thing about troubleshooting is you need to check that out on yourself so as you mentioned we have our own account and we have vps server so customer mentioning about some issue we first test out ourselves so that we are confirmed and we i mean it's good to know how it works first and then at the customer rather than just answering their question directly so uh in this process i think documentation is a key part because everything is written on how a product works or how to test it yourself so it's a good place to go first check out how things work then try it yourself and then help the customers so that customers are more satisfied and they know that we are providing the right things or right steps taking forward yeah yeah we're getting better you know we still i think there's still a lot of room for improvement in the documentation uh area but we we've definitely made some progress and we're getting better and we're we're still focusing on it um so yeah everyone gets test servers that they can use to reproduce issues um that they use to you know install and try out every Cloudflare product as well so it's training it's for issue reproduction it's for training um another component of the training program outside of you know boot camp and sort of periodic one-on-ones are what you might consider micro learning opportunities um so we do what we call knowledge shares so we're almost 100 people now on our team and and that's a lot of experience that's a lot of skills and everyone has come to our team with different backgrounds and different experiences and different skill sets and then once they come to Cloudflare uh after training during training and after training and as they start working with customers people start building up skills and you know expertise in different areas and we need to learn you know learn from each other and benefit from each other's experiences so the training team and the team at large tries to facilitate this as much as possible and so we do monthly knowledge shares i think the singapore team does them even more often that we do lunch and learns there are coding clubs as well tech support engineers some of them already have some web development and coding experience and some of them didn't necessarily have a lot of web development coding experience and that's okay and they had other relevant experiences that make them valuable and great contributors so we do these micro learning opportunities as well we'd like to open the floor to people who want to share or sometimes we notice things throughout the week or the month and we ask people hey could you could you show everyone that tool you're using so this is a chance for people to sort of share that script they wrote or the alias they started using the browser extension they found or the third-party website that's really great for checking on something and people are constantly on their own little coding projects writing little writing little tools eventually sometimes these become operationalized and they become a tool that everyone uses but yeah so we we try to do micro micro learning opportunities as much as possible and another important thing that we kind of touched on earlier briefly is case studies so we have helped hundreds and thousands of customers with issues and a lot of these issues are the same we do see the same issue more than once and so we document these as case studies as much as possible and we review these with our trainees and we review these as a team and so we're constantly adding to sort of our database if you will of case studies per product per subject and we stash these in different places we put these in our troubleshooting guides for each product we put them in our trainings to we'll go over specific ones in our trainings and we will put them in sort of a central database that people can just query if they're curious you know when we talk about troubleshooting strategies issue reproduction is really important but have we seen this issue before is another sort of pretty logical step and so we try to make that possible for people to kind of go back and search for past customer issues to see if we've encountered it before and see if someone already has the solution we don't want to recreate the wheel we need to be efficient with our time so at one point mark helped to sort of create a process for us to start start cataloging case study tickets and so now we sort of have a tagging system and then we review it and it gets categorized and and then we catalog it like i said we sort of stash it in these these various places and they tend to be pretty helpful for people yes i think you touched on something that i feel is important for for the support team which is everyone comes across a difficult ticket and they may spend a large amount of time working through that ticket so how do we ensure that all this knowledge is not lost whereby someone has to replicate the same effort to troubleshoot a similar issue in the future so i think that that's where the runbook comes in so a runbook comes in when it's accumulation of different different tech support experience and then we ensure that this the knowledge is not can be passed on to the next next agent so imagine that you come across a number of steps that you need to repeat for a certain similar issue and you record down these steps and then someone else will come across the same issue you can just repeat the same steps to test to perform some tests to ascertain whether there is an issue with certain parts of the program and then they can quickly help the customer to resolve the issue so that is the runbook and i think that runbook is a very good idea and in my training to the customers i've incorporated that into the training as well because i find very important and very useful for them as well and you mentioned about cataloging the case study i think that that is also something i find to be in in line with the accumulating the the knowledge to pass on to the new hires so yeah how about that do you find the runbooks to be useful yeah as i said earlier it's useful because you know where to go and what to do so that's quite useful and like as you mentioned case studies so other thing is we run through these case study tickets to see how we troubleshoot so that gives more knowledge for us or like what sort of tools to use if we come across similar issue and it certainly helps like to reduce the time instead of performing the same things testing at your lab you know these are the steps just run through them and it's good like we can answer the ticket earlier and help the customer resolve the issue faster so that's the key thing and other part is i've seen a lot of these things put into automations as well if something is repetitive then people start with automation and it makes the job even faster and shane mentioned about the other trainings that we get so personally i'm not good with coding so in singapore team we have something called coding club so that's quite useful so you just get into that training and you see how how people are developing or then they give you a task to do this so that's a good way to start learning something so it's quite helpful yeah i mean it's not a requirement for tech support engineers at Cloudflare to already know python or javascript or go or or anything else but it sure helps when you do and our approach has been that you know there's a lot of training on on all those things out there and we needed to focus on Cloudflare specific things in our program and so we hadn't necessarily built you know how to write javascript training because there's so much out there but it our team definitely benefits from it and so coding clubs have sprung up and it's been really great and some of the actual tools that the clubs have worked on and built are now tools that everyone are using which is really fun but as you both were talking you sort of reminded me about another component of our training which we're still working on but we have sort of a minimum viable product is debug servers so when we were training people before we get to the point where you know like i said earlier customers write in with issues and sometimes we see the same issue and we know that there's going to be an issue that you're going to encounter once you're actually working with customers what we used to do was go into the free queue you know with the free customers where our free customer inquiries are which we process as tickets and we correspond with the customer in a ticketing system and we used to go in there and we used to hope that that issue is in there that there just happened to be one and we'll show them how to debug that because they're going to run into that again or maybe it was just really the process of solving this type of issue uh sort of forced you to practice using certain tools uh you know certain databases that we have where you're pulling data and analyzing data and a lot of time we we do this training and that ticket wasn't there and we said you know let's just make servers that are broken in a certain way or misconfigured rather um and so we set about this kind of massive project of creating you know a whole suite of of servers for trainees to debug with very specific issues that we now use in our trainings and we we want to make more and more and more of these we have a handful and we have sort of decided to prioritize the ones that we thought would deliver the most value to our trainees but we have a huge wish list of servers that we want to have misconfigured in very specific ways and then that we can task trainees with okay you watched the e -learning you did the debugging with mark now fix that server that we know how we know how it's misconfigured based on the tools and the things that we've just shown you so really just getting started on that one i think there's already a few that people can go and debug but we want to have you know we want to have dozens of servers to debug and maybe i can plant that seed with your coding club if they want to look at our wish list and knock out a few of these uh debug servers that we have on our in our roadmap um that'd be awesome i could take that point and discuss it yeah that'd be great um now we have a few you know um and we want to build a lot more because they really help um and gives you that sort of real life experience so i want to sort of pivot a little bit to sort of a meta topic about training in general and measuring the effectiveness of training which is really important so how do we know and mark and i talk about this a lot you know how do we know if the training that we're giving is working um and traditionally in the training industry there was sort of a a four level model you know and we don't always strictly follow it but it is a really good sort of high level guide on gauging training this ties into feedback as well at times and it's going to sort of tie into another topic i want to get to but i can just go over a couple of the levels level one is a reaction survey and so you know most people who have taken a training a trainee a good trainer at the end of the training will make everyone in the room fill out a survey right and usually there's like you know rate on you know scale of one to ten on these sort of subjects did the trainer deliver this well did they seem to know the content do you feel confident that you can you know it does this apply to your job and there's questions and we get some data and then the big important thing here is how can we improve um and this is just an open field and so this is sort of one of the many ways we're asking for feedback how can we improve and so really that's just level one measurement and usually what that measures for us is did they enjoy it do they feel like it's relevant you know did we explain it well enough to them um do they think they understand the subject and what can we do better so that's level one now level two is uh is sort of were they able to actually retain the knowledge did they remember what you tried to teach them to remember and so this is usually done with like a post-training evaluation or assessment or a quiz if you will we're doing this on all of our e-learnings we don't currently do this with our hands-on sessions but we definitely do it with all of our e-learnings just to see and just to make sure really that usually it's just sort of trying to help people memorize certain facts like the big takeaways the key takeaways um and it also just kind of tells us you know if the question is is worded weirdly and you know if if that training um helped them memorize some things and then the third level is really the big one which is now does that knowledge that they got actually allow them to solve problems the problems we thought that they would right um and how do we measure that and the best way to measure that is by observing actual behavior or work in in our case we have tickets customer tickets and so we do ticket qa and so we look at tickets we do ticket qa for each other this is peer feedback happens a lot we're really open to improving as a team in general and the training team does qa and the team leads who are the managers of the tech support engineers do qa they look at the tickets and we have a rubric and it's a quantitative score based on several criteria did you answer in time did you answer appropriately with an accurate answer and all these things and then we could sort of line that up with our training now if they are they able to answer uh real customer problems um about that subject that we taught them on um and if they can um after having taken the training then then we can then we can go hey look they they are actually applying the thing that we taught them to do to their job um and so that's level three that's really important and then the fourth level doesn't get talked about too much but is did it actually move the business metric that you designed right so that would be something like increasing um efficiency increasing productivity increasing uh speed to to proficiency um you know reducing costs by increasing efficiency right so we tend to we want to you know uh either increase revenue or reduce cost um and so training is a way to do that by increasing someone's time to productivity we can if we can get someone to full productivity faster than then that's what we want to do um and so we're also looking at those things um so i just wanted to kind of set sort of touch on qa and how important it is really and and it's also just a way of getting training not only are we just sort of measuring ourselves can can the tech support engineers do what we need them to do can they solve actual customer problems but uh we feel we do the qa and then we see oh hey they didn't use this this tool um they didn't know about this tool now we know that we need to add that tool into that training it's not there we look at our training design outline we didn't cover that thing but we've surfaced it in qa that the person didn't use the the best tool to solve the problem so and then we go to that person as well and we go hey let me show you this cool tool that um you know we didn't we forgot to train you on um or we want to remind you of um right so it's just really important that we do qa and it's helpful for us as well because we get some pointers like there's always scope for improvement so when some experienced uh person review our tickets they'll give some pointer this thing you could have done better or you could maybe use this tool so that's really helpful for us and other thing about the feedback is i think mark's take mark takes it quite seriously and whatever feedback we provide he tries to implement them and it tells us like to learn better so it's not just for the sake of filling we are filling but i do watch on it that they take it seriously like one best example is marks instead of like what used to happen earlier was he used to go through the dashboard during the live session so which is one thing we told like this is time consuming because we already know what's in the dashboard so there's something you could do offline and maybe we can watch the record session and save some time for other things to do so that was quite useful and he took it positively and he started recording the session and shared it with us yeah i mean almost everything we're doing now is based on feedback and ideas that we've been given by people who go through the training so every time not only does do we administer a survey after every individual training hands-on session but we are constantly asking for feedback we send anonymous feedback surveys we ask the managers we ask other people to ask the trainees for feedback you know so that it comes from a different direction that's really important so that people feel comfortable giving their candid feedback and every time someone goes through the program it gets better every time someone gives us a new idea oh hey you you know you need to add this so you that document you guys have for setting up that one tool that could be better and you know that needs to be improved and we we love to hear that and the whole team in general is is really focused on improvement and feedback is is really important to our team and so we have a lot of different mechanisms for for giving each other feedback it's not just from the trainer to the trainee or from the manager to the to the tse the tech support engineer we also give each other peer feedback all the time and so we've created this program called learning opportunities or training opportunities where another tech support engineer can flag a ticket as a as a learning opportunity for another tech support engineer and we ask that they leave a good and we give actually training on how to give good feedback because it's it's an art in itself but if you give very specific feedback hey did you know about this tool you could have you know would have helped solve this faster or you know stuff like that and they mark the ticket and they leave an internal note that only we see and then the training team follows up and we look at that and we sort of determine is this something that we didn't cover in our trainings that we need to work into our trainings is this just one thing one person needs to kind of be reminded of um is this something that needs to get into our documentation it's not covered anywhere in any of our documents um or is this something we need to do a bigger refresher on with the whole team hey we're noticing a trend here um and so and this kind of since tech support engineers are we're 24-7 right the the u.s team hands over to the asia team hands over to the europe team every day uh 24-7 follow the sun we do end up working on some of the same customer issues together and so we see those things and we say oh hey shane um i noticed you didn't use this query and it would have gone a lot faster if you did and so we training up that and then mark and i will look at the training op ticket and we'll say hey that's we didn't cover that in our training and that's not in any of our runbooks and a lot of people aren't using that tool and so if we didn't have that sort of mechanism and sort of culture around uh helping each other learn and grow um then we wouldn't be as good a team without it so um it's it's really it's another big important part of our of our process i think i'd like to add one point so regarding the documents it's not that trainers are writing these documents so we have given opportunity that if we see something can be added on at any point in time we can go and add the point so that's that makes it easy maybe someone checking the document later will be helpful for them so it's looks like uh on live uh editing yeah yeah we have to do that i mean we have to we have four trainers five trainers and we know a lot i mean mark is extremely knowledgeable but there are so many Cloudflare products and features he can't be an advanced expert in everyone and so we end up having specialists specialization people end up specializing and so we have what we call subject matter experts um quite common uh in most companies to have these sme for short some people even say smee but in my experience developing training programs managing training programs in general trainers need sneeze a lot of the time we end up becoming sneeze we end up needing to become experts but we need to turn to someone who's in the engineer that wrote you know wrote the code or someone who's actually dug into it and they need to help us build the training and they need to help us deliver the training sometimes or they need to inform us and train us train the trainer and then we'll go and disseminate that to everyone else in the case of our documentation it's a great port a great point kagan is that we have literally hundreds literally maybe even thousands of documents that we need to keep maintained and it can't just be one person's job and it can't just be the training team's job and so we do have a little bit of a democratic process here we let people uh update content and it's it's really critical if you see something that's out of date inaccurate um could be worded better could have a screenshot um where there's no screenshot that would have helped it be clearer then we love we don't need people to ask permission to do that or we don't need to route that through the training team or through the sme people just just make that change and um and that's just one of the ways we can kind of keep doing this all at scale so we've covered a lot i think i'm looking at my list over here and i can see that man we've been talking for for a good 50 minutes um i've got some other stuff but um i guess we could jump down to some questions we got unless uh gagan or mark is if there's anything you want to bring up that we haven't touched on yet like how do you train the trainers oh yeah cleaning them is also important apart from the new hires yeah and this is something admittedly i could do better with uh but uh i think first off i want to say that i've been very lucky as a training program manager as a training manager to get technical trainers that are already so good so mark is outstanding um extremely technical and good at explaining things and good at teaching people and our trainer in the u.s is the same and our trainer in me is the same so i've started with with great people um but we have some content on it i have a trainer training module in our in our lms that i built um and i do you know we have weekly meetings with each other um when when they all started uh we focused for a while on just sort of training best practices um i wrote some docs internally on training techniques and i asked them to read it um i point i point them to some out outside information um one of our goals you know our quarterly and sort of annual goals as a training team itself is professional development so i want mark to become a better better trainer and our other trainers to get better at being trainers um that's sort of how i got into it initially i was in the solar industry for 10 years and eventually a large manufacturing company that actually has a big presence in in southeast asia um the second biggest solar pv manufacturing company in the world that makes the most efficient solar pv module in the world um and they eventually i was doing uh epc engineering procurement and contracting for a local company and designing solar installations and and project managing solar installations and i was working with a lot of the equipment solar equipment on the market um and uh used this manufacturer quite a lot and eventually they approached me and said hey uh we know that you know a lot about our product we can see that you design our spec our products a lot and buy a lot of our product and sell a lot of our product and project managers do you want to come do training for us and i said yeah this is a great opportunity i love it um really cool industry really cool uh product amazing product and they were actually developing new product line um for commercial installations so not uh industrial size power plants but not home systems either we're talking you know several megawatt systems maybe on a commercial facility um and so they had devised this amazing um integrated solution that uh could be installed um extremely quickly and cheaply but it was very proprietary and they had no training on it and they had so they hired me to help build a training program around that and to train all of their customers so they sold through dealers um and so they trained i was training design mostly design engineers on how to design these products and project managers on how to manage projects and train their teams on how to install it and one of the first things my manager did was send me to trainer training he said you know i know that you know all about our products and i know that you can do this job but i want you to do it well and so he made sure that i went to trainer trainings and gave me time to do that and so that at that point i started getting into training as i realized that training was a whole industry in itself um that all companies have dedicated large all companies of a certain size tend to have um training specialists and it's a specialization and and so there's lots of different training specialists there's technical trainers technical training instructors which i was for which i was for several years there are training managers there are lms administrators like ours scott there are e -learning developers there are content producers you know there's instructional designers there's courseware developers there's all these training specializations so i got really into it um and yeah it's really important that that my team you know that our team is the same that we that we get better it's not just the training team that that we talk about becoming better trainers it's everyone if everyone sort of can apply some of the basic sort of training best practices with each other then our team kind of gets better at sharing our knowledge great question i love that one um what else do we have we got some questions on the line here um if if i'm interested in applying to become a Cloudflare support engineer how can i prepare myself i think that's a good question we are hiring uh in all of our offices all over the world we're hiring tech support engineers especially multilingual uh tech support engineers um but you're welcome to apply because we're hiring and so what i would suggest for you is um well first of all put a site on Cloudflare the the free plan or even a paid plan if you you know if it's a you know if it's a site um that you could use a paid plan for but otherwise i would say at minimum put your website behind Cloudflare and a free plan immediately um and start reading some of the help information there um if you use it's fine to use a wordpress site or or any of those other kind of pre-built templates but um we'd even i'd even suggest to go a step further and you know get get a vps and actually install a web server nginx or apache or one of the others install a real web server and spin up your own website and put it behind Cloudflare and follow the instructions that online the the support documentation that exists and if you do those things um that would be the best way to prepare what do you think mark yes uh i totally totally agree with that and i think um for me i i believe it is it may be similar to a lot of people uh if you're if you're interested in web technologies yeah you're always uh wanting to know uh what the latest newest uh web technologies out there and Cloudflare is always uh releasing new products uh that are beneficial to uh the website performance security and so on if your interest your passion is in web development and uh web technologies uh being in Cloudflare is really a fun thing to do i mean if you are doing something fun it's no longer uh it doesn't feel like work anymore it feels like you're happy to do you're happy to do it every day because so so that is what i'm trying to say is uh uh this is the job for you if if you're uh you have a passion for uh web technologies and and you want to learn continuous continuously learn every day uh new things and you will enjoy this job thank you so much mark we're out of time thank you for joining me mark thank you for joining gagan it's great to talk to both of you thank you that was really fun um let's do it again and uh everyone out there joey Cloudflare tv and uh thanks for the great conversation on training thank you guys see you soon