Project Fair Shot: Helping the County of San Luis Obispo Distribute the COVID-19 Vaccine

Hi, everybody. My name is Jocelyn. I'm the program manager here at Cloudflare, and I'm excited to have Tyler Penney, who's the web server administrator in the county of San Luis Obispo in California, and we're going to be talking about the county's work in distributing the COVID-19 vaccine and how they used a new Cloudflare product called Waiting Room to keep their website online during these vaccination appointments. So for those who are unfamiliar, we launched Project Fairshot in January of 2021 to provide a new Cloudflare product called Waiting Room for free to ensure that governments, municipalities, hospitals, pharmacies, and other organizations that are distributing the COVID-19 vaccine to make sure that their websites do not crash under the load when they first begin scheduling these vaccine appointments. So with Waiting Room, you can basically create a digital waiting room that sits in front of any of your existing web registration site without needing any code changes or hardware installation. So it really allows you to create this queue of those looking to sign up for the vaccine, and once the registration officially opens, then you're let in to make the appointment. So it's only allowing so many people in at a time at the back end of your web infrastructure that it can handle. So we're going to kind of talk about a bit more of those pieces, but to begin, thank you, Tyler, for coming on Cloudflare TV. Do you mind describing your role at the county and give us some insight into the county's operations, for example, how many residents you serve? Yeah. Hi, everybody. Tyler Penny. I'm the web services administrator for the county of San Luis Obispo. Our county is halfway between San Francisco and L.A., and so we have about 277,000 residents, and that makes us the 23rd largest county in the state of California. There's 58 counties, and so yeah, we wear that with a bit of a, you know, a lot of pride. So anyways, yeah, we as a county organization, we offer in addition to providing the COVID -19 vaccine right now, we provide a lot of other services to our citizens, including, you know, food distribution, food, transportation planning, road service requests, road service maintenance, foster care, the list goes on and on. We do a lot for our county in addition to what we do now for the COVID-19 vaccine. So yeah, we're right in the middle of it right now. We've been vaccinating since late December of last year, and so yeah, we are very glad to be using Project Fairshot in the waiting room to be able to help with the process. Yeah, that's so great to hear. So how has COVID, you know, when COVID first started in March, how did that impact a lot of your team's operations, and how do you see that impacting it for the future? Well, we're still offering the same services now that we did before, but it's just at a safe distance and, you know, following all the COVID-19 protocols. So we still are offering the same services that we did before, but now some of those services are actually moved online, and which I think is a good thing. We're able to reach more people more efficiently. So in that aspect, it's been good. You talk about, you know, the shift online, and we've seen a lot of government agencies, you know, do the same. And, you know, from a web server administrator standpoint, what did that look like for you working with your team to make sure that all of your sensitive information was secure and having, you know, different team members working from home? Yeah, so we actually, from my perspective, it actually got a lot better when we can work from home. We use Microsoft Teams, which helps us to be able to communicate quickly, and we can, you know, we work a lot better as a team, I think, through that platform. And so, yeah, there's just a lot of things changed. We spun up a lot of new forms, new secure type forms to be able so people can fill the information out online. And then from there, you know, we could talk to them and work through whatever process we needed to. So it's been a good improvement for the local community as well. Yeah, that's really great to hear. So thinking about, you know, on the website side, how did you first become familiar with Cloudflare, and what types of issues were you facing, you know, before you onboarded to Cloudflare? Yeah, so as a government agency, a couple things that we do, we run elections. And so during elections, before and after elections, we're prone to a lot of attacks, both foreign and domestic. And so we have to, you know, we had to, especially before Cloudflare, monitor every second, every bit of traffic that came through and made sure that, you know, our website could be able to handle the amount of traffic that came through during these peak times. So when we had elections, we could see somewhat, you know, we'd see quadruple the amount of traffic to our website. Our firewall would constantly having to, you know, protect us from attacks. So we also, as our county emergency operations department deals with fires, and so if there's a fire, it causes a lot of damage, or there's something happens in our county that's newsworthy, not only as our local residents looking to our website for information, now we have national attention. And then all of a sudden, we could be having, you know, everyone across the country looking at our website for information on an event. So at any time, we were having to be prepared for huge spikes in traffic. And so when we heard about Project Athenian, we were very excited. And that's where we came on board a couple years ago, because we hold elections, we're able to use those Cloudflare services to be able to protect us with web application firewall that you offer, and then also handle that huge amount of traffic that comes through before an election as we help people to know where to vote and when to vote. And then also after the election, when we post the results, and that election night, about 9pm, when we are done counting the ballots, the, you know, the provisional ballots, everything we have, at that moment, we post the results, the initial results, and our website gets slammed. And so, you know, for the first, before we had Project Athenian and Cloudflare, it was always kind of a scary moment, we're always having to watch those servers really carefully. But with Cloudflare, and all of those services, we can kind of focus on election now. And it's great to be able to do that. So that's how we got started with Cloudflare. Yeah, it's great to hear that. I know we've worked a lot specifically on the election side when it comes to counties. And we hear a lot of that same some of the same sentiments that you talk about, specifically with election sites. So in the US, it's like some of these county websites will get a couple of 1000s of views a day. And then on election night, it spikes to the hundreds of 1000s. And most of that is legitimate traffic. So people trying to figure out where their polling place is. And what is what do you think that part in making sure that your website stays online to provide authoritative information? What do you think that role plays in terms of the trust in elections? And I want to get to the vaccine part. But I think that trust in you know, the systems is really important in government agencies. So I'd love to hear what you think about that. Well, you just said it. That's exactly right. We want people to trust the process. And this that holding elections is kind of like a sacred duty that we have. And it is our duty as a government agency to make sure that they are held correctly. And that the our local residents feel that you know that we are doing all that we can to ensure that it's safe and secure. And part of that is having a website that is running. And we're able to provide the results quickly. And that you know, it can handle the sort of traffic that that we need to have it handle. And so yeah, trust is the trust is the word right there. We want we want people to feel safe and secure in our in the process. Great, so let's let's talk about vaccines. So what does that look like in in your county? And I know we've talked a little bit about this looking kind of at other states, but what are you doing in terms of the distribution? How are you thinking about logistics? I know it must be incredibly difficult to serve all of your residents. So what at a high level? What does that look like? Yeah, so our county is already kind of divided geographically into three locations. We call it North County. And we have a section that's kind of central San Luis Obispo area. And then we have an area called South County. And so we have placed vaccination sites at each one of these locations. And we are able to serve those sort of regions that way a lot better. So at one location, we've actually taken over a high school. Since all the students are currently, you know, learning online, we had space for that. And then centrally, we're using our local community college. And so at that location, you know, we're we have more of a central people that live kind of central to our county are able to use that location. And then we have in the north part of our county and event center that we've taken over. And so that event center is also providing vaccines. And so each one of those these locations has different capabilities, as far as you know, freezers that we have available that as you know, the Pfizer and Moderna vaccine have different requirements for you know, temperatures and how to store it. So we when we get an allotment, I'll talk about that in a minute. But when you get an allotment from the state, we have to distribute it to those different regions, according to their capabilities. So we have these three locations. And since December 28 of last year, we have given 20 about 22,000 plus first first doses. And then we're almost 6000 second doses already. So yeah, in our county, we split it up into two phases. And within the first phase, we actually have split into into three sections. So phase one, what we call it phase one, phase two, and phase one has one a one B and one C. So phase one A is where we're back we vaccinated healthcare workers, medical providers. Phase one B is the 75 plus crowd. And then phase one C is where we're at right now. And we're currently that's that's 65 plus. So that's where we're currently at. So yeah, those we have, we had about 17,000 or so healthcare workers in that area, you know, that we targeted that 60 that 75 plus is about 26,000 or so residents and that 6565 plus is about 38 ,065 excuse me 65 to 74 is about 38,000. So that's a lot of people and there's a lot of people who want the vaccine. And so, yeah, it's it's been a it's been a challenge. And I'll talk about that, you know, in a minute, but it's been a real challenge to for us to be able to schedule that many people for these facts that these vaccines that are kind of in limited supply. Yeah, and as you can see from the news that the limited supply part and trying to figure out, you know, what is the best way to vaccinate people in a quick manner, but also in a safe manner, it is really interesting to think about. So what is the primary way, for example, if I'm in the county, how do I go to register for the vaccine? Do you do it online? Is there do you go into the office curious as to how, you know, this is done in a practical manner? Right. So we are we kind of have two different ways that people can register for a vaccine. We right now we're we are helping those who are in the sixty five plus range. And those people, you know, don't have a lot of them don't use the Internet. They don't have they might not have a device or they're just not used to that method to be able to register for anything. And so we set up a big phone center. So we have a giant phone center that's staffed by county staff that that is currently taking phone calls and can be able to take, you know, take those registrations. But then we also have an online form that people can fill out. We have broadcast that kind of a quick and easy link that people can hear and and type in or through social media they can click on. And then from there, they're able to register for a appointment when they become available. That's that's interesting. So we've seen a lot in the news about many of government websites kind of going down due to the sheer volume of people looking to sign up for vaccine appointments. So what did this look like for your county and what types of issues did you have when you first started registration for for the vaccine appointments? Yeah, it was similar to a similar similar situation for us. And as we reached out to other counties and jurisdictions around California and around the United States, you know, we saw that everyone's facing similar issues. You have a lot of demand for a very little bit of supply. And so when you go to offer that supply out, the demand is so huge that, you know, it can it can cause havoc. It really can cause havoc. So imagine, you know, Black Friday sale and you have 10 TVs and there's a thousand people trying to get that TV. What happens to the store when you open the doors? I mean, it can it's going to be bad, you know. So we we had to kind of figure out quickly what to do our first couple of weeks. We struggled with with our systems going down. We had we had heard that there are some jurisdictions using Eventbrite, which is a great ticketing ticketing type tool. But we had heard it also had issues. And the language that's used for the for that system is mostly around selling tickets. And so it wasn't really appropriate for what we were trying to do at the same time. So we went with another solution called Microsoft Bookings and we ran into some limitations on day one. And it's actually a really great product. It's just not really not meant for the sort of week long Garth Brooks concert type event that we were dealing with. There was a lot of people who want to want to get these appointments quickly. And so midday we had, you know, the system just actually failed. We we had we were seeing server errors and all kinds of bad things on the site when people were trying to register. So we had to take it down, apologize publicly to, you know, to everyone and then quickly pivot to another solution. So we went with Google Forms, which is actually a very powerful tool if because we've actually added a lot of automation behind the scenes. So Google Forms, you have the ability to create scripts and then we can dynamically open and close appointment slots. However, each one of those those scripts take a few seconds to be able to run. So even though it can handle that amount of traffic, all the automation that we have behind the scenes couldn't keep up with the thousands and thousands of people all at once trying to book an appointment. So we had the right tool and we had all the right scripting, you know, in the in all the excuse me, all the automation behind the scenes to be able to to make that work. But at the same time, you know, we needed something to be able to meter people through so that we can so so it would just it it would work. You know, we could we could have people register. So along claim came Project Fairshot, you know, and so we're really excited to hear about this. And so luckily, someone who's on our team is follows the Cloudflare blog and saw something and then immediately noticed it. And so we they told us and we jumped on it so fast. We were so excited to to research about it and just start start looking into it. So, yeah, that was that was a great moment. We were able to actually see the light at the end of the tunnel, so to speak, and get this implemented. Yeah, that's great to hear. I'm happy that you heard it from our blog. Our blog team is going to be super, super happy to hear that. But can you talk a little bit about your experience implementing Waiting Room and what that process looked like for you on the technical side? Yeah, it was actually very, very simple to implement. As you mentioned, you know, at the beginning of this, a lot of what we did, though, beforehand helped us to implement quickly with the with the Waiting Room feature, you can actually configure your own Waiting Room. So you can have it look exactly like your website. You can have all your own branding. You can have your own logo. And so we took advantage of that. And we spent a couple days working with our media relations folks, working with our web team to be able to build out this nice little Waiting Room. So we had a little bit of fun with the design of it. You know, we made these little icons with people in line with masks on and we try to make it kind of just a nice place to be. So, you know, that's that's we kind of focus on. We've, you know, with with all the design that we do, we try to have empathy be the center. That's a big value for us. And so whatever we design, whatever we do, we try to make it so it's it's empathetic. It's nice. It's a nice thing to use. And so we started there and we built this nice Waiting Room. So and once we had it ready, once we had it tested, we were able to basically go through and fill out a form. Once it was turned on for us, we filled out the form. We just did a little bit of configuration with our caching rules and and just a lot of stuff. The Cloudflare team was was wonderful helping us. And when we had any questions, they were able to quickly answer and tell us, you know, do this instead of that and gave us some pointers on how to how to how to set it up. And so once we got it up and running, and this was literally like a day a day before we were about to release 5000 doses to county residents next week. The way it works for us really quick is, you know, every Thursday at 9am is when we release the following week's appointments. And so Wednesday night, just about we are configuring this and getting, you know, getting feedback Cloudflare was helping us, you know, and we were able to get it configured. And then that next morning, it was turned on, and everyone was just so excited to just see it working so well. We all got on our mobile devices and, and checked it and we could see, you know, the process is great, you go to the link where you're going to register for your vaccine. And if there's, if there's more than X amount of people, it will actually put you into this waiting room, right? And you have this waiting room, and it tells you, you know, don't leave, you're in line, which is so reassuring to someone who's already concerned about getting their appointment. So it reassures them. It's okay, you're in line, you know, don't worry about it too much. Just stay here, we've got you, you know, and we love that we love being able to tell people that. And then it also tells them, you know, your wait time is 20 minutes, 30 minutes, depending on how many people are in the queue. And they can see that wait time gets smaller and smaller and smaller, until automatically the page refreshes, and then they're there have the ability to make their appointment. And so much better experience from, you know, trying to get going to the site, it's down, you know, it's not working. Their confidence is lost. Again, we talked about trust and confidence. So we're able to help these people to be able to get that. And it's elegant, it's a very elegant solution to this problem. Because at the same time, when we have this waiting room up and running, we can tell people during the process, how many doses are left, how many appointments are left. So we can update that waiting room throughout that hour or two that we're making those appointments. And so when you know, at first, we can say appointments are ready, come on in. And then as we're getting close, we can say appointments are almost full, please check back tomorrow if you don't get in. And then we can update the waiting room to say something like, you know, appointments are full, please come back tomorrow to see if there's any more appointments. And so, again, it's a very nice way to be able to communicate, to be able to let people know if the vaccines are, you know, if we're running out of spots, and they need to come back the next day. It's a really, really nice solution. So yeah, we've been constantly, you know, trying to improve the process. And every week as we roll out new vaccines, you know, we're trying to improve our communication with everyone. But so far, we've had a lot of great feedback. And we hear from a lot of people who are very grateful for the ability to, you know, to have a better, a better experience. It's not, it's not Lord of the Flies anymore. Which, you know, basically is what it was before is, you know, you'd have, if you had a fast Internet connection, or you could, you could click really quickly, you know, that would give you an advantage on getting a vaccine. And so now, this is more ethical, we're able to walk people through, they're able to get an appointment, we're still limited in our supply, we'd love to be able to vaccine as many people who that would like a vaccine right now. But like the rest of the world, the rest of our country, we're just limited to what we have. So we try to do the best, best with what we've got and, and get those people through as fast as possible. And it's interesting when you talked about kind of the phases of, you know, certain when you a 65 and older is in this phase, and when they get vaccinated and thinking about, you know, the technical expertise, like to make it fair enough for people, you know, it's easy to go for you to go online and, you know, sign up for the vaccine. But for others, it's not as easy and making that that process more fair, I think is it's really great to hear specifically with your use of waiting room, but kind of thinking about comparing, you know, before you were joined Project Fairshot, and started using waiting room, that vaccine registration day to, you know, when you had the Thursday at 9am with waiting room, what did those those numbers look like? What are the differences between those? And do you have any, you know, numbers that you can share of improvements that you saw? Absolutely. So yeah, we would, like I mentioned, our, our, our county population, those eight, those those phase groups are all around about 20,000, or so, with each one of those groups. And that's we see 20 25,000, or so site, you know, visitors in those mornings, before, you know, prepping, starting around 7am, we see the traffic start to increase. And then it's around 23 25,000, or so, site visitors right at 9am. And this is not just, you know, people in that age group, but it's also their kids trying to register for them. It's, you know, there's, there's a lot going on there. So if you can imagine, you know, we have a huge we with with the with before the cloth or waiting room, we had everyone at once trying to get all those 23,000 plus visitors trying to get those, you know, either in some weeks, we only had 2000 doses, some weeks, we have 7000 doses to administer. So you do the math, demand way outweighs supply. And so those warnings, you know, those those couple of mornings that we we did this before project fair shot were, were scary, to be honest, and we were trying to hold on tight as we saw, you know, our registration systems fail, and, and, and, you know, have to publicly apologize to local residents, you know, and so we going from that to having this elegant solution, you know, is totally different. So now, we have the same amount of traffic, we still have, you know, all of these users trying to get in, but now we're letting them in 200 at a time. And they have about, you know, a couple minutes between each group. And it's, it's been really interesting, because we watch our Google Analytics through the process. And it's, it's, it's kind of spooky, you can, you can actually see cloth layer letting people in 200 at a time. So you'll see this this group of 200 come in, and then they'll, they'll go through the process and register. And we watch our registration systems. And we can see, oh, 23 spots filled up, you know, at the at the high school for 9am. And then we can see, you know, we can just see everything kind of fill up over time. But it happens at a pace that's manageable, a pace that allows our team to communicate. It's a pace that allows us to be able to make any changes on the fly if we need to. And it allows it allows that back and forth communication happen with with everyone. So by metering everyone in now, like we have in the last couple of weeks, it has really created, you know, a good opportunity for us to communicate. And it's it's a lot better for everyone, for the local residents, especially, but for our small web team, you know, there's about five of us or so that are working behind the scenes on this. And so to be able to, you know, breathe a little bit, watch the numbers, watch the watch the registrations happen, you know, watch all the slots fill up, you know, it's really great. And it feels it feels really good to be able to, especially for us, this is, you know, a once in a lifetime opportunity to do something really great. And so, you know, we're, we're, we're glad to have Cloudflare on board with that to kind of help us through that and do this really important work. Yeah, it's really, it's really heartfelt hearing, you know, your experience with distributing the vaccine. And, you know, we're happy to help in one small way to make sure that people have access to it. But what would your advice be to other counties who might be going through some of these same issues, or maybe, you know, they're not in the process of distributing the vaccine, but preparing to get there? What type of advice do you do you have for them? Yeah, so you have to know your audience first. So like we did before, before we started designing any of this process, we had those in the phase one B and C groups, age 65 and 75 plus, we designed the process around them. And so we understood, you know, that not everyone would want to or be able to fill out a form, there'd be a large majority that will want to just make a phone call. So you have to make you have to kind of make plans for that and make sure that those people can do that. Another really important part of all of this is communication. We have an excellent communications manager, and she has done a great job at, you know, identifying the where we need to communicate and when. And so having a really good communication plan for the whole process, and be upfront with the public. I think that's been really important as well is that when we hear information or we get information, we share it right away. And we are able to quickly turn around and try and do the best we can with the information that we have and with the resources we have. So our county organization has put everything it has into this, into the vaccine distribution right now. People who are, you know, normally filling in potholes, you know, around the county, are now working at a call center. And we have people, you know, who were, you know, doing all kinds of different stuff that our county does are now dedicated to this cause because it's it's so important. And we're, you know, it's an honor for all of us to do this. But you have to start there, start with a good communication plan, and understand your audience. As I said before, it's for us, it's important that we start with those core values, like empathy. And any, when you start with that, when you want to have an empathetic solution, and you want to focus on helping people, then the design just kind of comes and you're able to create something that's beautiful and elegant and helps people through the process. So it's, yeah, it's like I mentioned before, it's not easy. The tools that you need to, you know, you have to have a good tool to be able to register and to, you know, and have something like Project Fairshot, if you can, you know, to be able to meter through people through and use it as a good communication platform. Yeah, that's, that's so great to hear. And thanks so much for for sharing your experience with Project Fairshot and Waiting Room. We really appreciate, you know, the time and the great work that you're doing and about distributing the vaccine and hearing about the great work. It's really just so refreshing to hear that. But for those who are watching out there, so Project Fairshot, we're providing Waiting Room for free to any government, hospital, pharmacy, municipality that are distributing the COVID-19 vaccine. So it's open and eligible to organizations around the world. And the Cloudflare team's ready to help with onboarding for those who need assistance. So if you're interested, please go to projectfairshot.org to learn more. But thanks, Tyler. And thanks, everybody for watching. We really appreciate it, Tyler, and we'll definitely be chatting soon. Thank you. All right. Bye.