Originally aired on March 16, 2021 @ 10:30 AM - 11:00 AM EDT
Jocelyn Woolbright, Program Manager at Cloudflare, will discuss Project Fair Shot with our customer, Tyler Penney - Web Services Administrator at County of San Luis Obispo, who will describe their experience with it.
Hi, everybody. My name is Jocelyn. I'm the program manager here at Cloudflare, and I'm excited to have Tyler Penney, who's the web server administrator in the county of San Luis Obispo in California, and we're going to be talking about the county's work in distributing the COVID-19 vaccine and how they used a new Cloudflare product called Waiting Room to keep their website online during these vaccination appointments. So for those who are unfamiliar, we launched Project Fairshot in January of 2021 to provide a new Cloudflare product called Waiting Room for free to ensure that governments, municipalities, hospitals, pharmacies, and other organizations that are distributing the COVID-19 vaccine, to make sure that their websites do not crash under the load when they first begin scheduling these vaccine appointments. So with Waiting Room, you can basically create a digital waiting room that sits in front of any of your existing web registration site without needing any code changes or hardware installation. So it really allows you to create this queue of those looking to sign up for the vaccine, and once the registration officially opens, then you're let in to make the appointment. So it's only allowing so many people in at a time at the back end of your web infrastructure that it can handle. So we're going to kind of talk about a bit more of those pieces, but to begin, thank you Tyler for coming on Cloudflare TV. Do you mind describing your role at the county and give us some insight into the county's operations, for example, how many residents you serve? Yeah, hi everybody, Tyler Penney, I'm the web services administrator for the county of San Luis Obispo. Our county is halfway between San Francisco and L.A., and so we have about 277,000 residents, and that makes us the 23rd largest county in the state of California. There's 58 counties, and so yeah, we wear that with a bit of a, you know, a lot of pride. So anyways, yeah, we as a county organization, we offer in addition to providing the COVID -19 vaccine right now, we provide a lot of other services to our citizens, including, you know, food distribution, food transportation planning, road service requests, road service maintenance, foster care, the list goes on and on. We do a lot for our county in addition to what we do now for the COVID-19 vaccine. So yeah, we're right in the middle of it right now. We've been vaccinating since late December of last year, and so yeah, we are very glad to be using Project Fairshot in the waiting room to be able to help with the process. You talk about, you know, the shift online, and we've seen a lot of government agencies, you know, do the same, and you know, from a web server administrator standpoint, what did that look like for you working with your team to make sure that all of your sensitive information was secure and having, you know, different team members working from home? Yeah, so we actually, from my perspective, it actually got a lot better when we can work from home. We use Microsoft Teams, which helps us to be able to communicate quickly, and we can, you know, we work a lot better as a team, I think, through that platform. And so yeah, there's just a lot of things changed. We spun up a lot of new forms, new secure type forms to be able, so people can fill the information out online, and then from there, you know, we could talk to them and work through whatever process we needed to. So it's been a good improvement for the local community as well. Yeah, that's really great to hear. So thinking about, you know, on the website side, how did you first become familiar with Cloudflare, and what types of issues were you facing, you know, before you onboarded to Cloudflare? Yeah, so as a government agency, a couple things that we do, we run elections. And so during elections, before and after elections, we're prone to a lot of attacks, both foreign and domestic. And so we have to, you know, we had to, especially before Cloudflare, monitor every second, every bit of traffic that came through and make sure that, you know, our website could be able to handle the amount of traffic that came through during these peak times. So when we had elections, we could see somewhat, you know, we'd see quadruple the amount of traffic to our website. Our firewall would constantly having to, you know, protect us from attacks. So we also, as our county emergency operations department deals with fires. And so if there's a fire, it causes a lot of damage, or there's some something happens in our county, that's newsworthy, not only as our only our local residents looking to our website for information, now we have national attention. And now and then all of a sudden, we could be having, you know, everyone across the country looking at our website for information on an event. So at any time, we were having to be prepared for huge spikes in traffic. And so when we we heard about Project Athenian, we were very excited. And that's where we came on board. A couple years ago, because we hold elections, we're able to use those Cloudflare services to be able to protect us with web application firewall that that you offer, and then also handle that huge amount of traffic that comes through before an election as we help people to know where to vote and when to vote. And then also after the election, when we post the results, and that election night, about 9pm, when we are done counting the balance, the you know, the provisional balance, everything we have. At that moment, we post the results, the initial results, and our website gets slammed. And so, you know, for the first, before we had Project Athenian and Cloudflare, it was always kind of a scary moment, we're always having to watch those servers really carefully. But with Cloudflare and all of the services, we can kind of focus on election now. And it's great to be able to do that. So that's how we got started with Cloudflare. Yeah, it's great to hear that. I know we've worked a lot specifically on the election side when it comes to counties. And we hear a lot of that same some of the same sentiments that you talk about, specifically with election sites. So in the US, it's like some of these county websites will get a couple of 1000s of views a day. And then on election night, it spikes to the hundreds of 1000s. And most of that is legitimate traffic. So people trying to figure out where their polling place is. And what is what do you think that part in making sure that your website stays online to provide authoritative information? What do you think that role plays in terms of the trust in elections? And I want to get to the vaccine part. But I think that trust in you know, the systems is really important in government agencies. So I'd love to hear what you think about that. Well, you just said it. That's exactly right. We want people to trust the process. And this that holding elections is kind of like a sacred duty that we have. And it is our duty as a government agency to make sure that they are held correctly. And that the our local residents feel that you know that we are doing all that we can to ensure that it's safe and secure. And part of that is having a website that is running. And we're able to provide the results quickly. And that you know, it can handle the sort of traffic that that we need to have it handle. And so yeah, trust is the trust is the word right there. We want we want people to feel safe and secure in our in the process. Great, so let's let's talk about vaccines. So what does that look like in in your county? And I know we've talked a little bit about this looking kind of at other states, but what are you doing in terms of the distribution? How are you thinking about logistics? I know it must be incredibly difficult to serve all of your residents. So what at a high level? What does that look like? Yeah, so our county is already kind of divided geographically into three locations. We call it North County. And we have a section that's kind of central San Luis Obispo area. And then we have an area called South County. And so we have placed vaccination sites at each one of these locations. And we are able to serve those sort of regions that way a lot better. So at one location, we've actually taken over a high school. Since all those students are currently, you know, learning online, we had space for that. And then centrally, we're using our local community college. And so at that location, you know, we have more of a central people that live kind of central to our county are able to use that location. And then we have in the north part of our county, an event center that we've taken over. And so that event center is also providing vaccines. And so each one of those, these locations has different capabilities, as far as, you know, freezers that we have available that, as you know, the Pfizer and Moderna vaccine have different requirements for, you know, temperatures and how to store it. So we when we get an allotment, I'll talk about that in a minute. But when you get an allotment from the state, we have to distribute it to those different regions, according to their capabilities. So we have these three locations. And since December 28th of last year, we have given 20 about 22,000 plus first first doses. And then we're at almost 6000 second doses already. So yeah, in our county, we split it up into two phases. And within the first phase, we actually have split into three sections. So phase one, what we call phase one, phase two, and phase one has one A, one B and one C. So phase one A is where we're back we vaccinated healthcare workers, medical providers. Phase one B is the 75 plus crowd. And then phase one C is where we're at right now. And we're currently that's that's the 65 plus. So that's where we're currently at. So yeah, those we have, we had about 17,000 or so healthcare workers in that area, you know, that we we targeted that 60 that 75 plus is about 26 ,000 or so residents and that 6565 plus is about 38,065, excuse me, 65 to 74 is about 38,000. So that's a lot of people. And there's a lot of people who want the vaccine. And so, yeah, it's it's been a it's been a challenge. And I'll talk about that, you know, in a minute, but it's been a real challenge to for us to be able to schedule that many people for these facts that these vaccines that are kind of in limited supply. Yeah, and as you can see from the news, the limited supply part and trying to figure out, you know, what is the best way to vaccinate people in a quick manner, but also in a safe manner, it is really interesting to think about. So what is the primary way, for example, if I'm in the county, how do I go to register for the vaccine? Do you do it online? Is there do you go into the office curious as to how, you know, this is done in a practical manner? Right. So we are we kind of have two different ways that people can register for a vaccine. We right now we're we are helping those who are in the 65 plus range. And those people, you know, don't have a lot of them don't use the Internet. They don't have they might not have a device or they're just not used to that method to be able to register for anything. And so we set up a big phone center. So we have a giant phone center that's staffed by county staff that that is currently taking phone calls and can be able to take, you know, take those registrations. But then we also have an online form that people can fill out. We have broadcast that kind of a quick and easy link that people can hear and type in or through social media they can click on. And then from there, they're able to register for a appointment when they become available. That's that's interesting. So we've seen a lot in the news about many of government websites kind of going down due to the sheer volume of people looking to sign up for vaccine appointments. So what did this look like for your county and what types of issues did you have when you first started registration for for the vaccine appointments? Yeah, it was similar to a similar similar situation for us. And as we reached out to other counties and jurisdictions around California and around the United States, you know, we saw that everyone's facing similar issues. You have a lot of demand for a very little bit of supply. And so when you go to offer that supply out, the demand is so huge that, you know, it can it can cause havoc. It really can cause havoc. So imagine, you know, Black Friday sale and you have 10 TVs and there's a thousand people trying to get that TV. What happens to the store when you open the doors? I mean, it can it's going be bad. You know, so we we had to kind of figure out quickly what to do. Our first couple of weeks, we struggled with with our systems going down. We had we had heard that there are some jurisdictions using Eventbrite, which is a great ticketing ticketing type tool. But we had heard it also had issues. And the language that's used for the for that system is mostly around selling tickets. And so it wasn't really appropriate for what we were trying to do at the same time. So we went with another solution called Microsoft Bookings. And we ran into some limitations on day one. And it's actually a really great product. It's just not really not meant for the sort of week long Garth Brooks concert type event that we were we were dealing with. There was a lot of people who want to want to get these appointments quickly. And so midday, we had to, you know, the system just actually failed. We had that we were seeing server errors and all kinds of bad things on the site when people were trying to register. So we had to take it down, apologize publicly to, you know, to everyone and then quickly pivot to another solution. So we went with Google Forms, which is actually a very powerful tool. If it because we've actually added a lot of automation behind the scenes. So Google Forms, you have the ability to create scripts, and then we can dynamically open and close appointment slots. However, each one of those, those scripts take a few seconds to be able to run. So even though it can handle that amount of traffic, all the automation that we have behind the scenes, couldn't keep up with the thousands and thousands of people all at once trying to book an appointment. So we had the right tool and we had all the right scripting, you know, in all the excuse me, all the automation behind the scenes to be able to to make that work. But at the same time, you know, we needed something to be able to meter people through that we can. So so it would just it would be it would work. You know, we could we could have people register. So along claim came Project Fairshot, you know, and so we're really excited to hear about this. And so luckily, someone who's on our team is follows the Cloudflare blog and saw something and then immediately noticed it. And so we they told us and we jumped on it so fast. We were so excited to to research about it and just start start looking into it. So, yeah, that was that was a great moment. Were you able to actually see the light of the tunnel, so to speak, and get this implemented? Yeah, that's great to hear. I'm happy that you heard it from our blog. Our blog team is going to be super, super happy to hear that. But can you talk a little bit about your experience implementing Waiting Room and what that process looked like for you on the technical side? Yeah, it was actually very, very simple to implement, as you mentioned, you know, at the beginning of this, a lot of what we did, though, beforehand helped us to implement quickly with the with the Waiting Room feature, you can actually configure your own Waiting Room so you can have it look exactly like your website. You can have all your own branding. You can have your own logo. And so we took advantage of that and we spent a couple of days, you know, working with our media relations folks, working with our web team to be able to build out this nice little Waiting Room. So we had a little bit of fun with the design of it. You know, we made these little icons with people in line with masks on and we try to make it kind of just a nice place to be. So, you know, that's that's we kind of focus on. We've, you know, with with all the design that we do, we try to have empathy be the center. That's a big value for us. And so whatever we design, whatever we do, we try to make it so it's it's empathetic. It's nice. It's a nice thing to use. And so we started there and we built this nice Waiting Room. So and once we had it ready, once we had it tested, we were able to basically go through and fill out a form. Once it was turned on for us, we filled out the form. We just did a little bit of configuration with our caching rules and just a lot of stuff. The Cloudflare team was was wonderful helping us. And when we had any questions, they were able to quickly answer and tell us, you know, do this instead of that, and gave us some pointers on how to how to how to set it up. And so once we got it up and running, and this was literally like a day, a day before we were about to release 5000 doses to county residents next week. The way it works for us really quick is, you know, every Thursday at 9am is when we release the following week's appointments. And so Wednesday night, just about we are configuring this and getting, you know, getting feedback Cloudflare was helping us, you know, and we were able to get it configured. And then that next morning, it was turned on and everyone was just so excited to just see it working so well. We all got on our mobile devices and, and checked it and we could see, you know, the process is great, you go to the link, where you're going to register for your vaccine. And if there's, if there's more than X amount of people, it will actually put you into this waiting room, right? And you have this waiting room, and it tells you, you know, don't leave, you're in line, which is so reassuring to someone who's already concerned about getting their appointment. So it reassures them. It's okay, you're in line, you know, don't worry about it too much. Just stay here, we've got you, you know, and we love that we love being able to tell people that. And then it also tells them, you know, your wait time is 20 minutes, 30 minutes, depending on how many people are in the queue. And they can see that wait time gets smaller and smaller and smaller, until automatically the page refreshes. And then they have the ability to make their appointment. And so much better experience from, you know, trying to get going to the site, it's down, it's not working. Their confidence is lost. Again, we talked about trust and confidence. So we're able to help these people to be able to get that. And it's, it's elegant. It's a very elegant solution to this problem. Because at the same time, when we are when we have this waiting room up and running, we can tell people during the process, how many doses are left, how many appointments are left. So we can update that waiting room throughout that hour or two that we're making those appointments. And so when you know, at first we can say appointments are ready, come on in. And then as we're getting close, we can say appointments are almost full. Please check back tomorrow if you don't get in. And then we can update the waiting room to say something like, you know, appointments are full, please come back tomorrow to see if there's any more appointments. And so, again, it's a very nice way to be able to communicate to be able to let people know if they're if the vaccines are, you know, if we're running out of spots, and they need to come back the next day. It's, it's a really, really nice solution. So yeah, we've, we have been constantly, you know, trying to improve the process. And, and, and every week, as we roll out new vaccines, you know, we're trying to improve our communication with everyone. But so far, we've had a lot of great feedback. And we hear from a lot of people who are very grateful for the ability to, you know, to have a better, a better experience. It's not, it's not Lord of the Flies anymore. Which, you know, basically is what it was before is, you know, you'd have, if you had a fast Internet connection, or you could, you could click really quickly, you know, that would give you an advantage on getting a vaccine. And so now, this is more ethical, we're able to walk people through, they're able to get an appointment, we're still limited in our supply, we'd love to be able to vaccine as many people who that that would like a vaccine right now. But like the rest of the world, the rest of our country, we're just limited to what we have. So we try to do the best best with what we've got and, and get those people through as fast as possible. And it's interesting when you talked about kind of the phases of, you know, certain when you a 65 and older is in this phase of when they get vaccinated and thinking about, you know, the technical expertise, like to make it fair enough for people, you know, it's easy for you to go online and, you know, sign up for the vaccine. But for others, it's not as easy and making that that process more fair, I think is it's really great to hear specifically with your use of waiting room, but kind of thinking about comparing, you know, before you were joined Project Fairshot and started using waiting room, that vaccine registration day to, you know, when you had the Thursday at 9am with waiting room, what did those those numbers look like? What are the differences between those? And do you have any, you know, numbers that you can share of improvements that Absolutely. So yeah, we would, like I mentioned, our, our, our county population, those eight, those those phase groups are all around about 20,000 or so, with each one of those groups. And that's we see 2025,000 or so site, you know, visitors in those mornings, you know, prepping starting around 7am, we see the traffic start to increase. And then it's around 23 25,000 or so site visitors right at 9am. And this is not just, you know, people in that age group, but it's also their kids trying to register for them. It's, you know, there's, there's a lot going on there. So if you can imagine, you know, we have a huge we with with the with before the cloth or waiting room, we had everyone at once trying to get all those 23,000 plus visitors trying to get those, you know, either in some weeks, we only had 2000 doses, some weeks, we have 7000 doses to administer. So you do the math, demand way outweighs supply. And so those mornings, you know, those those couple mornings that we we did this before project fair shot were, were scary, to be honest, and we were trying to hold on tight as we saw, you know, our registration systems fail, and, and, and, you know, have to publicly apologize to local residents, you know, and so we going from that to having this elegant solution, you know, is totally different. So now, we have the same amount of traffic, we still have, you know, all of these users trying to get in, but now we're letting them in 200 at a time. And they have about, you know, a couple minutes between each group. And it's, it's been really interesting, because we watch our Google analytics through the process. And it's, it's, it's kind of spooky, you can, you can actually see cloth layer letting people in 200 at a time. So you'll see this, this group of 200 come in, and then they'll, they'll go through the process and register. And we watch our registration systems. And we can see, oh, 23 spots filled up, you know, at the at the high school for 9am. And then we can see, you know, we can just see everything kind of fill up over time. But it happens at a pace that's manageable, a pace that allows our team to communicate. It's a pace that allows us to be able to make any changes on the fly if we need to. And it allows it allows that back and forth communication happen with with everyone. So by metering everyone in now, like we have in the last couple of weeks, it has really created, you know, a good opportunity for us to communicate. And it's it's a lot better for everyone, for the local residents, especially, but for our small web team, you know, there's about five of us or so that are working behind the scenes on this. And so to be able to, you know, breathe a little bit, watch the numbers, watch the watch the registrations happen, you know, watch all the slots fill up, you know, it's really great. And it feels it feels really good to be able to, especially for us, this is, you know, a once in a lifetime opportunity to do something really great. And so, you know, we're, we're, we're glad to have Cloudflare on board with that to kind of help us through that and do this really important work. Yeah, it's really, it's really heartfelt hearing, you know, your experience with distributing the vaccine. And, you know, we're happy to help in one small way to make sure that people have access to it. But what would your advice be to other counties who might be going through some of these same issues, or maybe, you know, they're not in the process of distributing the vaccine, but preparing to get there? What type of advice do you have for them? Yeah, so you have to know your audience first. So like we did before, before we started designing any of this process, we had those in that phase one B and C groups, age 65 and 75 plus, we designed the process around them. And so we understood, you know, that not everyone would want to or be able to fill out a form, there'd be a large majority that will want to just make a phone call. So you have to make, you have to kind of make plans for that and make sure that those people can do that. Another really important part of all of this is communication. We have an excellent communications manager, and she has done a great job at, you know, identifying the, where we need to communicate and when. And so having a really good communication plan for the whole process, and be upfront with the public. I think that's been really important as well, is that when we hear information, or we get information, we share it right away. We are able to quickly turn around and try and do the best we can with the information that we have and with the resources we have. So our county organization has put everything it has into this, into the vaccine distribution right now. People who are, you know, normally filling in potholes, you know, around the county are now working at a call center. And we have people, you know, who were, you know, doing all kinds of different stuff that our county does are now dedicated to this cause because it's so important. And we're, you know, it's an honor for all of us to do this. But you have to start there, start with a good communication plan, and understand your audience. As I said before, it's, for us, it's important that we start with those core values, like empathy, and any, when you start with that, when you want to have an empathetic solution, and you want to focus on helping people, then the design just kind of comes, and you're able to create something that's beautiful and elegant and helps people through the process. So it's, yeah, it's, it's like, like I mentioned before, it's not easy. The tools that you need to, you know, you have to have a good tool to be able to register and to, you know, and have something like Project Fairshot, if you can, you know, to be able to meter through people through and use it as a good communication platform. Yeah, that's, that's so great to hear. And thanks so much for sharing your experience with Project Fairshot and Waiting Room. We really appreciate, you know, the time and the great work that you're doing, and about distributing the vaccine and hearing about the great work. It's, it's really just so refreshing to hear that. But for those who are watching out there, so Project Fairshot, we're providing Waiting Room for free to any government, hospital, pharmacy, municipality that are distributing the COVID-19 vaccine. So it's open and eligible to organizations around the world, and the Cloudflare team's ready to help with onboarding for those who need assistance. So if you're interested, please go to projectfairshot.org to learn more. But thanks, Tyler. And thanks, everybody, for watching. We really appreciate it, Tyler, and we'll definitely be chatting soon. Thank you. All right. Bye.