Originally aired on April 19, 2021 @ 2:30 PM - 3:00 PM EDT
A Customer Cloudversations segment where Brian Batraski, Product Manager, Cloudflare and Kayla Geigerman, Customer Advocacy Manager, will interview Rob Madole from Font Awesome about their usage of Cloudflare products and the industry in general.
Hi everyone, thanks for tuning into this segment of Customer Cloudversations here on Cloudflare TV. If this is your first time tuning into our series, Cloudversations is a place where we like to spotlight different customers, learn about what they do and how they do it, discuss industry-related topics and go over best practices. Today we are joined by Brian, who is my colleague here at Cloudflare and is helping me co-host today. We also have Rob from Font Awesome, who is going to talk about all the great work that they do with Cloudflare load balancing. But before we get down to the nitty-gritty, we'll do some introductions. I'm Kayla, I'm on the Customer Advocacy team here at Cloudflare. Brian, go on. Hi everybody, my name is Brian Batraski. I'm the Product Manager here at Cloudflare for our load balancing product and a few others as well. Nice to be here. Thanks Brian. And Rob, why don't you go ahead and tell us a little bit about what you do at Font Awesome and what Font Awesome does in general. Yeah, I'm Rob Madol and I work at Font Awesome. We really make icons for you to put on your website or your project. So it's kind of a simple mission. We started off as an open-source project in 2012 and our CEO Dave Gandy invented the project and it actually got really popular. A lot of people have probably heard of Font Awesome. If you haven't heard of it, you've probably seen it. You talk to go throughout the day and surf the Internet without seeing Font Awesome icons. So in 2015, Dave and our CTO Travis, they did Y Combinator and launched a business. And now we're, I think we're 13 strong today. We just had somebody come on today. It was our first day. So we're now up to 13. Oh, that's awesome. Congratulations to that lucky 13th member. It doesn't sound lucky, but we're excited. So you said that it's hard to surf the Internet without seeing Font Awesome. So who and where are your customers? What can you tell us about them? Yeah. So we're kind of developer focused. So we've got, and I should have looked at our website before I came on there, because I'm going to forget a few customers, but McDonald's, Amazon, we've got kind of icons everywhere. And so since we've been doing it for a long time, and since it's open source, a lot of people will reach for our icon set because it's free. They can start out for free and it's, it's liberally licensed and makes it easy to use it in whatever project you have. So when these large customers like reach for your icons, what do they end up doing with them? So, I mean, primarily they're, they're looking for a way to get, I mean, iconography is everywhere on the Internet, right? You can't hardly go to a site without seeing an icon. Yeah. They have sign in icon or the share for Facebook or Twitter, all those things that kind of show up and you would, okay. So you'd think, you know, how do I get this thing integrated into my site? Well, there's quite a few different ways. And depending on what tool you're using, it can actually be more complicated than it should be. So our main mission here is, is to really make it as easy as possible for you to just accomplish this simple mission of, I just want to put this icon next to this text and then get on with the rest of my day, because that shouldn't be that hard and it's not really my focus. So that's what we try to do. We try to make it as easy as possible. And to that end, we've got some services. So we are a SaaS company, Software as a Service, and we've got some kind of CDN content delivery network products that again, in that same vein, is try to, let's make it as easy as possible for you to get this job done and get on with your life. And that's kind of where Cloudflare starts to come in is we've got people that rely on us and this thing has to be up and it's always got to be serving your icons. And so that's kind of where we're at. Awesome. So let's specifically move on to Cloudflare Load Balancing, which I know is a product that you guys are heavily using in our stack. So can you talk to me a little bit about the problems that you were having before you moved to Cloudflare Load Balancing? What were you missing? Sure. We have this product we call Kits, and it's a way to get a unique token that identifies your site and gives you some extra features. And some of the things we were doing with that, it's for the most part, really simple. We need to serve icons, there's some CSS assets, there's web font files and JavaScript files. And for the most part, it's really simple. But the things that we add on top of it, it starts to get complicated enough that we need a more sophisticated setup. So with this Kit service, you can do things like upload your own icon, that was a feature we just added. And the issue we were dealing with is we were trying to get this service up and we kept thinking to ourselves, it's got to be easier than this, because we tried two different providers and one was Docker based, one was JavaScript based. And we just were having scaling issues for one thing. I mean, Fun Awesome is a popular project, we do have quite a few customers. And right off the bat, we're trying to launch this, and we just can't get it to stay up. Our monitoring is telling us this thing's down, these origins aren't responding, we've got issues with database connectivity, all this kind of stuff. So we were in a tough spot. We had this product kind of completed for over a year, and we're just waiting to get the infrastructure up. Super frustrated, it's holding up the rest of our development pipeline. And then we ran across you guys, it was actually a kind of a cold email from your sales team. And we started to get into conversations about, okay, so let's see if Cloudflare can help us solve this issue. So really, it came down to, we couldn't get the thing up, couldn't get it stable and running for us to say, okay, it's a complete product ready to go. Awesome. So when you were having those problems with stability, how was that really impacting your business and your customers? Were you losing sales or money lost or anything like that? I think we had to have been. We had enough problems and it was user facing, it was visible. People were emailing us telling us, hey, this thing is down. I've got it in my production website and it's not stable for me. So we had direct impact. I'm sure we lost some sales over it. We lost definitely some reputation and we've had to work over the past year to try to rebuild that. But it was impacting us because we couldn't continue on. We have just a list of great features we want to put on top of this. And our reputation was getting tanked because of it. Yeah. And I can only imagine, considering today you just added your 13th member that having only less than 12 employees having to deal with all of that work must have been really hard. So I'm glad that you guys were able to get that under control. So let's talk about your experience deploying Cloudflare Load Balancing. What was that like, whether that was an onboarding experience or getting it to actually work? It was fantastic. So we were able to, through the sales cycle, we got hooked up with a solutions engineer. His name was Chris and he set everything up for us. We were able to try everything out and see if it actually worked, which is a big deal for us. We're coming from this place where our trust is kind of low. We've been burned twice now. And so they set it up. Chris suggested, let's get you an account and we'll kind of do a trial thing. All the features are going to be there. You can even try out stuff. Eventually, we tried stuff that we didn't end up using, but we were able to try everything out from the beginning. So much so that I had Terraform as kind of an infrastructure's code tool. We had that set up. We were going to make sure that the Terraform provider was working like it should before we kind of signed the deadline and started writing checks. That was fantastic. Chris was there the whole time. We ran into a couple of weird things that we were able to get on the call with him and he helped us solve it. And we were off to the races. Awesome. Well, I'll make sure to send Chris an email after this and tell him that he's graded his job. I've seen them one every couple of days. Okay. Then maybe I won't. We don't want Chris to get too big of a head. I'm kidding. So Brian, why don't you move on to a little bit of more of the specifics around load balancing that we would all love to hear about? Yeah, absolutely. Rob, I think this is an incredible story. You went from a point of the product being ready, ready to go out to customers, wanting to hit the market strong, and then the infrastructure wasn't there to be able to support you. And so I wanted to, before we get into some details, let's talk about how your customers are really interacting with your application. Do you see the majority of traffic being kind of SDK or API-driven traffic, or are people coming to your application, going through your UI? Walk us through that a little bit. Yeah. So the basic flow is they go to our fontawesome .com site and we have a management interface there. So they create these, we call them kit tokens or kit codes. It's a URL and a little chunk of JavaScript that you integrate into your site. So that's where they start. Once they put that into their site, if it's WordPress or whatever other tool they're using, they deploy that on the Internet. So now you go, let's say it's a retail store that has an online presence. You go to that retailstore.com and that little chunk of JavaScript loads the icons on the page. The icons are visible and they go from there. So they interact, our customers interact with us on that level. And then they've got, of course, all their end users that go to their site and buy things from them and use their website. So it's really, it's a tool to develop sites and a tool to kind of accompany sites. And that's how we interact with our customers. Makes sense. Makes a lot of sense. So I can imagine, as the developer market is growing every day, you have more penetration on market, more folks knowing about Font Awesome. And like you said, there's really no time where any of us can go through our day and not see something from Font Awesome. Let's talk about what impacts, when you were getting load balancing and running, what were the features that really kind of stood out to you that were kind of game changers that you're leveraging heavily today that's made a really big impact on your reliability? The biggest thing for us has probably been the dynamic traffic steering. So that's a feature, and you can explain it better than I, but from my standpoint of view, we've got, we have Amazon web services that act as our origin. So our applications running there, we've distributed that around seven different regions around the world. And the Internet's a dicey place. So we can have an origin go down and Cloudflare reacts to that so quickly that oftentimes we don't even get a whole lot of 500 or any kind of error messages on our side. It reacts really quickly and can reroute traffic to a different origin. Now, it may not be the fastest origin for that particular region, but at least it's up and we can go deal with the one, the origin that went down, figure out what that problem is, because this is still a new product for us, right? We do have, every once in a while, we got stuff, weird stuff that happens. And that has been great for us because we don't have to deploy in every single country around the world. We can hit kind of the highlights for us in our traffic patterns. And then that dynamic steering gets to make the smart decision of, okay, I need to send this request to this origin based off of, well, it's the fastest or it's the one that's up right now. So that's been great for us. We don't have to worry about that. So huge, huge headache off of our, or a huge problem off of our plates. That's incredible, right? Like for all the listening in, this is a capability that has allowed the Font Awesome team and Rob to be able to focus on their table stakes, to focus on their customers and not have to worry about the infrastructural maintenance or overhead of, am I going to be serving errors to my customers? Is my brand reputation going to be tarnished because I'm serving errors? And not only that, performance is maintained and performance is held paramount because I think we all can agree here, every day we see more people learning how to use technology earlier and earlier. I mean, I see my niece and nephew able to use an iPad better than they can go kick a ball outside, right? And this is the bar of what is expected for performance reliability is increasing every single day. And so, have you been able to quantify or see the performance increase in the request as they go to the fastest origin based on the round trip time from the health monitors and the load balancer? We have, we have. We haven't done extensive testing and by extensive, we haven't tested every region that we get traffic for, but for the ones that we have tested, we've seen on average about 60 milliseconds of increased performance. And that, if you don't run a CDN, that may not sound like an impressive number, but for a request that is sometimes the total time is 60 milliseconds. If you get a really fast edge network for other locations, it can be higher, but that's a huge savings for us. So, what that tells us is that the edge servers that Cloudflare have for the areas that we tested, they're fast, they're geographically close, because we're dealing with speed of light when you get down to that kind of timing. So, they're geographically close to the place we're testing. And that performance has a huge impact on our clients because if they're running a retail site, we all have read the papers and the reports, it's kind of known in the industry that you shave time off of your requests, that equals sales, that equals conversions, that equals engagement, whatever the metric is you're trying to look for. So, it's fantastic. So, 60 milliseconds is a very, very small duration, but it's a pretty huge number whenever you, yeah, exactly. Agreed. And that's amazing. I'm so happy to hear that. And as we continue to make improvements on the load balancer side and the dynamic steering feature, we're going to continue to drive that number down. But something that you had also mentioned was that not only do you not have to worry about the performance aspect as that's continuing to get better and scaling with your business, but you don't have to worry about these errors being served because of the fast failover, right? When you get proactively, you see that an origin goes down from your health monitors, Cloudflare's load balancer is going to steer that traffic away from that unhealthy origin altogether. Now, can you talk a little bit about what the reliability impacts that you've seen with that fast failover, with your fallback pools? What does that look like to your company? What does it look like to your customers? What does it look like to you personally? Yeah. So, contextually, we were starting in a place where we couldn't even get this product released. We were in a tight spot. So, the first bar, the first thing we mentioned is, is this thing released or not? Not only how reliable is it or what kind of availability does it have? We were just looking for that, is this thing up? Can we actually have this product be released? So, we hit that bar and from a reliability standpoint, we've had exactly one incident that caused downtime and it had nothing to do with Cloudflare. It was something we did on our side and we were down for about 10 minutes. Other than that, we don't have, we haven't had any incidences that caused downtime. So, we, and not to say we haven't had origins that have gone down, because we certainly have. We've had that happen and I've had to go and our teams had to go and fix those, but the pressure of, oh my gosh, we've got production websites that are losing sales or our reputation is getting hurt again because we're not providing service. That worry was gone because I knew that Cloudflare had already worked around it. It was diverting that traffic to a healthy origin and I could go in there and debug without stress and without the worry that kind of accompanies downtime. I knew that Cloudflare kind of had us there. So, that's from our standpoint and then from our customer's point of view, they have something they can rely on. Our incidents of having a customer tell us your site is down before we knew it, we just don't have that anymore. We've invested in the monitoring. Cloudflare gives us all this capability to kind of pull the different, we're hooked into APIs that Cloudflare provides so we know when the stuff is happening. So, we're able to be proactive and build our systems in a way that we were able to avoid downtime and truly reach that high availability that we want. Don't have any numbers. I haven't actually done the numbers on it but I'm pretty sure we've got several nines after the first two nines. That's amazing. I can't tell you, especially as the product manager, the product and hearing directly from you to see the massive impact, the reliability of being able to achieve and be able to focus elsewhere and really direct on your customers is the best thing that I could probably want to hear and for us as Cloudflare to hear. So, thank you. That's incredible and we're looking forward so much to increasing to support you even further. So, you have all this time now where you don't have to worry about the maintainability. You don't have to worry about the site constantly going down or you have a new release that these other areas of unknown items kind of shrinks down what negative can take place. Now, what has this opened up you and your team to be able to focus on now that you have these issues of something of the past? Right. Well, we are working on version six. So, that's a new thing. We were at major version five and with the release of kits and getting our infrastructure stable, we've kind of been able to focus on that now. So, our whole team is really working on our next big version. We're also adding more features to our kit service. One of the things we did most recently is we are dynamically building web fonts. So, you're able to upload your own icon. That was a big feature we had but it was only for the JavaScript version. Well, now we have it for the web font. Web fonts is basically like a custom typography or custom font on your website but we're shoving icons into it instead of letters and numbers. So, we're able to dynamically build that with Rust. So, we've got Rust and Elixir is our other language we use running on our app server and then Cloudflare is in front of that caching those assets and serving that up for us and being the front end, the edge servers. So, we're just able to add more features and products and bring more things to our customers that provide value and kind of accomplish the mission of making it easy to put stuff on your site. Awesome. Fantastic. So, you mentioned before how you're plugging into our APIs, making sure that you always have a pulse on what's going on with your infrastructure. Has there been other automation or capabilities that you kind of built on top of our APIs to drive more intelligence, to drive more automation or awareness of what's going on to lessen the burden on you and your team? Yeah, we have. From the lessening of the burden point of view, that Terraform provider that I mentioned earlier where we have infrastructure as code, that was a big deal for us. For so long, and we're a small team, there's 13 of us today, we've been a small team trying to serve a lot of people and there's some challenges to that. We don't want to spend a lot of time on infrastructure. So, we're just now getting to the spot and we're large enough where some of the responsibilities we can take on ourselves, having our own services that monitor, having our own services that handle logging and that kind of thing. So, that Terraform provider and being able to automate the configuration of our origins and our load balancers and make sure those things are exactly like they need to be, that was a huge thing for us because before it was manual. And with manual things, you always have that issue of, did I mess something up? And now I've created another problem for myself. So, that was huge for us. The other thing is we do have a monitoring system and we've hooked into the analytics that Cloudflare gives us to give us a different view of our systems. We've got the app servers and the things that we wrote and we're managing on AWS. We've got a dashboard for that and we see how that works. But having that same dashboard for Cloudflare to see the requests coming in, what's our cash rate? Are there any 500 or 400 errors coming through that are unexpected? All that stuff is really important to maintaining high availability. We're able to get, and it's all about being proactive. So, can we get to something before it becomes a problem and impacts for customers? And if there is a problem, do we already have something built in place that we'll route around and kind of self-heal? Makes a lot of sense. That's incredible. I love hearing a lot of the uptake of Terraform, the uptake of more intelligence, more customization on top of what we built today to really empower you and your teams to be able to focus on your customers and not have to worry about, oh, we're just going to stay up. Can we take a proactive stance against these resiliency items? Make sure that you are the best and foremost provider of icons and assets to different customers around the world. So, this is incredible. I think it's an amazing story and going from having the product ready and struggling to release to being massively confident that every release is going to go fantastically and that you can make sure your customers have the absolute best experience day in and day out is transformational. So, I'm so happy to hear it. I guess the last thing I would ask on my end is, have you had to interact with Cloudflare support teams other than Chris? And if so, what has that experience been like? Yeah, we have. There's been a few issues that, well, there was one specifically was a DNS weirdness that we ran into. We were able to work with the enterprise, because we're an enterprise customer, work with the enterprise support team. And they were very responsive. They understood the issue. And importantly enough, our solutions engineer was still involved in that situation. So, he wasn't necessarily on call, but he was involved in helping the other support engineers that were working with us to try to figure out what was going on and make sure that we had a solution that was going to work for us. So, they were, and again, this is also a big deal because we've been at other places before and we didn't feel like a priority. We would get kind of the first level or the first tier tech support, people who are just trying to do their jobs, but they're asking us questions to the account of, have you restarted it? Have you turned it off and turned it back on again? And we're like, yeah, yeah, yeah. Let's get past this. We're down into the specific IP addresses. I know what this stuff is doing. I've got these curl commands that are messing up and I need to figure out why. So, with enterprise support, the level that Cloudflare has been much higher and we feel supported. I don't have any doubts that if I need to get ahold of somebody, I can get ahold of somebody and do it really quickly. So, to me, there's a level of confidence there that, again, we see this as a partnership with Cloudflare and just the experience of having someone to watch over you that's got your back, that's super important to us. Makes sense. The Internet does weird things and there's never a one size fits all solution as much as we always want it to be. We kind of just want to have our fingers and help things work. Even in the simplest implementations, not saying that this is simple at all, but there's always some weird gotchas that take place. And I think you hit it right on the head that having a partner, having someone that is there with you hand in hand, not just trying to lead you, but digging into the weeds whenever necessary goes a very, very long way. And like you said, feel supported, feel taken care of and doesn't feel like you're just there, another ticket or another number in their system that you are a person that has needs, that has serious impacts and that we want to get the time to resolution as fast as possible. So, I'm really happy to hear that. Kayla, I'll turn it back over to you. Thanks, Brian. And thank you so much for all that info, Rob. That was really great. And we're glad to hear that you're doing so well with load balancing. And that leads perfectly into my next question. Obviously, as somebody who's had so much success, we might have other viewers who are either just starting with load balancing or maybe aren't using it to its full extent. What are some best practices that you can share or what should others know about using load balancing that you found particularly helpful? I think for us, we waited too long to deal with this. We were in a spot as a company where this wasn't a focus and our reputation was suffering because of it. So, if I have one piece of advice to other folks is don't be afraid to reach out to Cloudflare and get engaged with them. If it's not a fit, I'm pretty sure you guys will tell potential customers if you don't have a solution form that will work, but there's a good chance that you do. And that I wish that we had done this a long time ago. I wish that ourselves, Austin, our sales guy had emailed this two years back because maybe we could have avoided some of this. So, and there's a hesitancy, especially with a small company to not want to take some of this on yourselves. You talk about monitoring and logging and all this stuff. It can be kind of overwhelming, especially for a small team. And we found that it wasn't quite as bad as we thought it was going to be. There's a lot of great tools out there today and you can do this yourself. And the result is a system that you control, you have confidence in, and somebody like Cloudflare partner that can back you up as the same goals, the same concern for you. That's awesome. Thank you. And while we just have a few minutes left, I'd really be interested to hear more about what's next for Font Awesome. And I know that you said you're focusing on your next release, but whether that's any projects you have with other Cloudflare products or just how you plan to help your company grow, I would love to hear more about that. Well, we have recently, we had an experience, we're trying to build our own API. So this is a GraphQL API that we have customers that are building tools in Applifont Awesome. And we had an issue recently where a funky release of WordPress caused our API to go down. And it was because there was so much traffic. We had a spike about 400 requests per second. We just didn't have the capacity on the server. It just wasn't that big. So we're growing in that direction. And what we did is we took Cloudflare and we put the workers in front of it. And those workers are now doing some caching for us, which if you work with GraphQL, the sucker is hard to cache. Everything's a post request and it's kind of a turkey. So you have to do special things. And we were able to accomplish that with workers in a single day. We had to do it in a single day because we were down, the API was down. So we had to kind of hurry and get that done. So another spot for Cloudflare helped us out. We're continuing to do things like that and launch services and we've got all sorts of good things planned down the road. Awesome. That's really exciting. There are so many new things coming with workers. We had a big developer week launch last week where we announced a lot of new features and products. So that's great to hear that you guys are following up on that. With our last minute left, do you have any final statements, anything that you want to share about your company or you that we might not have spoken about? Not really, but I actually have a question for Brian. So one of the things in load balancing I saw today, because I was looking through our setup is you've got this custom rules. That's a new feature. What does that thing do? Yeah. So I was actually going to bring it up because I was going to do the same. So we just launched to be generally available custom rules. And what that allows you to do is to implement your own custom logic, as the name says, onto your traffic steering or just selection decisions. So if you want to load balance by path, if you want to load balance by a certain header, a cookie, time of day, whatever it may be, you actually now have the capacity to create your own conditional statements. And then we actually flattened the API where you can take very granular segments of your load balance configuration where you want to route folks to a particular origin pool or maybe update the session TTL on your session affinity, whatever it may be, you actually now have the capacity to do that yourself, either through our rule builder or our expression editor, whatever workflow works best for you. And you can make some really, really complicated, intelligent rules by utilizing our and or or statements in a specific rule. So then you can really make sure that the exact functionality behavior you want your traffic to exhibit directly aligns with your company goals and what you want infrastructure goals to be. Well, we are off, but unless you have any other questions for Brian, Rob, I just want to say thank you again so much for joining us.
