Gleb Budman, CEO Backblaze & Usman Muzaffar Fireside Chat

Hello, all right and welcome to another Cloudflare TV session celebrating the 10th anniversary of Cloudflare. I'm thrilled to welcome my guest today who is Gleb Budman. Gleb is CEO and co-founder of Backblaze and I've had the pleasure of knowing him for almost a decade. We were just chatting a minute ago about how long we've known each other and the incredible journey that of the companies that we've been involved with have been taking. So Gleb, let's just start with you. Say a little bit about yourself and how you got into tech. Thank you so much for being on the show with us today. Yeah, thank you for having me. Thank you for having us and it's awesome to spend some time together, whether it's for Cloudflare's birthday week or just to catch up and chat. So that's right just to have this opportunity. Yeah and you know I was looking at your resume which you know I remember looking at ages ago and I always forget you were part of Excite which was my homepage for years before you know it all got changed to the big G. So tell me, you gotta have some great story from the Excite years. So tell us something. Yeah you know that was a wild and nutty time. So I ended up there because our company that we were building at the time back in 9.9 got acquired by Excite days before the whole dot-com crash and so I'm not sure if that was a better dot-com story other than like before my company got acquired before the crash. It would be nice if it were a cash for stock deal. It was just a nutty nutty time. I mean we ended up there and within days after we got acquired the people who had acquired us left and so we show up and the new people are like, hey guys so what do you want to do together? And we're like you just bought our company and we're just gonna have a brainstorm session about what to do and then it was just this roller coaster where Excite at home had the first cash flow positive quarter. Then the next quarter they were completely out of money. Then they raised 50 million dollars. Then they went bankrupt and were getting acquired. Then they weren't getting acquired. It was a very very wild and crazy time. How did you even get into tech? What's your background? How did you and what was your role in even at Excite and we'll talk about your role at Backblaze of course. So I think I got into tech I think probably the way a lot of people do which is I broke things. When I was a kid I just I loved trying to figure out like oh how is this made and so I took apart various things that my parents had much to their dismay. Most of them didn't get put back together and I was very interested in mechanical engineering for a long time. Like cars and boats and robots and things and I was just fascinated with robotics and I was the youngest member of the United States Robotic Society. Robotic Society of America I think it was called at the time. I think it was nine years old and I just loved that and and at some point in the late 90s I was actually looking to build a robotics company focused on the nuclear industry. I'm trying to automate a lot really. It was my first job out of college was actually to work at GE for in the nuclear industry and so after I left there I thought you know what's really missing here is all of these tasks seem very manual and we could and we could automate this with robots. So that was going to be my plan and then the Internet happened and I kind of got my Internet happened at the end. That's that's fine too. I was actually studying to be a physician to be I was in med school and then Internet happened and you know that's that's the rest of that story. That is even a larger jump. I went a lot of left turn conversations like of like so where did you start and where how did you get in? So yeah so let's let's skip forward though. So Backblaze has a very interesting sort of founder story. There's your co-founder but actually a relatively large founding team and you know tackles this. That's right and and you know it's it's online backup which is nothing you know that was that was already kind of you know a bunch of players in there and you become senior of this team. So tell us a little bit about those early days and what was the mission and why backup out you know out of all the projects that you could work on? Yeah so the the so there were five co-founders and of which I'm one and so all five of us had actually worked together at this point now for about 20 years. Back then when we started Backblaze which was 13 years ago we had already worked together for quite a while. Yeah and we one of our co-founders Brian Wilson he's our CTO and he does IT for friends and family the way that many of us do. Of course we're all we're all we're all on call for for for parents and cousins and yeah my email broke help and so he had a friend who had a laptop die and he and she called him up and said you gotta help fix it and he's like no problem we'll get you set up we'll get your new laptop where is your backup and she starts pounding the table look what I don't need now is a lecture what I need is for you to get me my data back. And I need you to conjure the data. What's that? I need you to conjure the data. Yes resurrected from the ground from the failed platter from the ashes yes so you know he was he had always been religious about backups I mean he was just super good about it and and so he was like what do you mean you don't have a backup how is that possible everyone has backups and then we started talking about it and realizing you know I'm not sure that everybody does. I think this is actually a gap in the in the personal computing. It may be a problem and so we just we went down this path of talking about that and realizing as we talked to more people that most people did not yeah and so we said you know everything's going digital everybody's going to have these laptops and all data is eventually going to vaporize off the face of the planet if it's not backed up and that seems like a problem that should be solved and so we didn't know how to solve that but we decided to try to work on that problem and uh we and whereas the two previous companies that we had worked on both went the traditional path of raise venture funding they won etc what we decided to do with Backblaze was uh we got five of us together and said we would commit to each other for one year without salary that would be the initial seed round and we get started we start working on the problem and so that's how that's how we got started with it. That's amazing I mean like the the egalitarianness of it and it was also very you know um uh like all of you represented different parts of the business it wasn't all engineering you were bringing a product uh sort of mindset to it and Brian was the technical leader uh and and I believe you were all literally working together physically as well right and this is uh this seems this seems astonishing now in a COVID world but you were you were all crammed together in in a in a in a single apartment if I remember correctly that story is yeah before before today the idea of all being crammed into one small space seems very scary but back then it seemed also very very lively so we we actually crammed together in Brian's one bedroom apartment to start uh two of the people that uh were starting with us were in San Jose two of us were in San Francisco Brian was in Palo Alto in the middle and he was open to having us sit in his living room and originally it was going to be the five of us for six months and then six months became a year and five people became nine and over time Brian said you know what you guys are obviously not leaving my apartment so I am and so he left moved out and we converted his bedroom into a conference room and we had to drag uh power from the bedroom and the bathroom and everything into the living room and we set up a tough shed in the backyard to experiment with um storage servers and the like so uh we ended up being crammed in there for quite a while and I want to get back to that storage server idea but one thing I want to talk about because I think it's a very interesting um aspect of Backblaze it's a very opinionated product it has a point of view on how backups should be done and one of the things that's notable about is it doesn't ask the users to make a lot of selections or controls and uh just talk for a second about that because that was very striking as this is a product built by very you know very strong like engineers with a very deep engineering focus and it builds a product that has almost no knobs or switches of any kind it's the opposite of giving the users a lot of control how did how did how did that come about usually engineers you put engineers together in a one bedroom apartment they're going to come up with something which it looks like you know a cockpit and this is the opposite of that yeah it's funny you know the when we started the company we said this is a problem we didn't exactly solve that and we started talking to everyone everybody just kept saying oh my god it's so hard it's so hard it's so hard like what is so hard and one thing that we realized was that people kept saying what was hard was that all of the solutions had them make all these choices yeah the answers to and a lot of it was I don't know where my files even are yeah I don't even know what you mean by funds what's what you mean by files that's yeah um but so it was just the idea was we can make it so that it's infinitely flexible yeah nobody will use it just like nobody had been using it and the and that means that what we build won't work for everyone but it will solve the problem for the mass groups and I think for all of our products and that's one of the things that as a group we had always focused a lot on usability ease of use design the full experience it's actually you know this but one of our five co-founders is a designer that's especially 13 years ago that was rare and we and a lot of people said why is one of your co-founders a designer you know we understand an engineer a cto a business person maybe a finance person but a designer it was it people looked at us askance but we said but of course usability is so key no matter whether you're selling to a consumer an enterprise it person anything like whatever that experience is you have to make it easy for them and so it's something that we cared about from the whole moment of the phrase consumerization of it but that was one of the things people then talked about like look look how much of a difference it makes when it's easy and I think you know that it sounds like an easy thing to nod your head to but one of my favorite FAQs of all time was long ago in the backblaze and I and I wish I had I even went on gleb I went on archive.org to see if I could pull up that original FAQ and I couldn't find it because it had some awesome things like I am a linux hacker and I run a pro script that r syncs my data to multiple different things like I don't understand how to use backblaze in this setup and the answer was we love people like you backblaze is not for you at this time and I thought that was such an honest straightforward answer to be like we recognize there is a level of sophistication that is not our sweet spot and we are going to focus on making sure that everybody can just turn this thing on and back up I want to go back to what you what you talked about that that shed in the background because one of the problems you had to solve was not just software but how do you get a gazillion hard drives connected together and you know now it's the story of legend there's this thing called the backlist storage pod but talk about that like where did that start yeah it you know it's now it seems like such a good idea at the time it was sheer insanity so when we started the team was a very software focused team yeah and our plan was we were going to build this really easy to use service and then we needed to put the data somewhere right and our original plan was to put the data on amazon s3 which was fairly new but it was well established and it and it seemed like great you know they'll take care of the storage for us and when we did the math on using amazon s3 we just said we're gonna lose money on every single take care of the profits for you too yeah it's you know and everybody said no no no amazon s3 it's so inexpensive it's only you know pennies per gigabyte per month but you know when you add that up across terabytes and petabytes it starts adding up and we did the math like it's not going to work and so we said okay that's not going to work what are our other options and we looked at buying equipment from a lot of the big vendors at the time for storage and it was all those systems in addition to the fact that they were millions of dollars up front which we didn't have because we were bootstrapping the company but it was also it didn't make sense they were uh on a per gigabyte basis they were still just very expensive they were 10 times the price of a hard drive yeah we looked at it like but if you think of the hard drive was what's storing the bits yeah and if you could just figure out some way to plug that hard drive into the Internet that's what we need and so we just started down that rat hole of how do we attach a hard drive into the Internet and we went down you know we we thought about well let's see the usb spec for hubs says you can attach 128 devices into one hub so maybe we can do that and just because the spec says that since nobody ever does that none of them actually work and so we went down these various different paths of attaching a hard drive to the Internet and one way that we had figured out to do it was to attach these direct attached storage devices that were kind of consumer prosumer devices yeah right and plug those nine of into one server and we got that to work but obviously completely unscalable in terms of having all these different boxes and cables and you know it's just it would have been a mess and so then we tore the boxes apart and took out the little physically enclosures physically tore them apart because all we needed was the the little cards that the that the drives would plug into yep and then we said okay well now we have that and we made a plywood box server it was very high tech very high tech uh i'm so sad because we threw it away at some point don't tell me that it's not going to be in a museum somewhere i thought for sure you would put this in fiberglass somewhere i so we have a couple photos of it and we actually have a storage pod now but it does not have the plywood one which i'm so sad about but yes it's we we actually took it made it work and put it into the data center and ran it to make sure it actually worked and then we iterated on that and made it an actual server design that we call the storage pod but the whole point was make it as just don't expect anything from the box it's a hard drive connector for the Internet that's it nothing else and i thought the last version i saw of it was like fire engine red it was almost like watching the transformation of the original the iron man that he built in the cave the cool one like it was it's extraordinary it's become real engineering and it's become a reference architecture that other companies actually talk about and it started off with a bunch of software engineers literally nailing plywood together because they were trying to save a buck it's a great innovation story i love that iron man analogy we're gonna forget iron man storage i want to i want to credit on that if you start using that and that that that increases sales um the other thing you talked about there was this business of um hey you know the only thing i care about is that hard drive so forget this enclosure forget this power supply and i love that you guys invented this this term hard drive shucking uh like like it's an oyster but this isn't just a joke this became a real problem at one point in your company's history talk about the great hard drive prices yeah the great hard drive prices so when we started we looked at what hard drives to buy and when we were looking at buying enterprise grade drives yeah and those were really expensive and then we said i wonder if we could just use the consumer drives and at the time a lot of the people that we spoke with said oh you can't use consumer drives they won't work in a data center they won't work in a raid environment they won't work for long they're not meant to run for a while all sorts of they won't won't won't yeah but they were also roughly one half to one third the price of the enterprise drives for the same storage capacity and when we looked at it we said our expectation is everything we're going to do is software we're we're a software company we're software team we're going to write all the redundancy the performance the availability we're going to write all that in software and so we don't need much from the hard drives we just need them to store a bit and give us the bit back and if the failure rate on the consumer drives is 50 percent higher than the enterprise drives that's fine because we'll just stick that in software and make the add the additional redundancy but we'll still save a lot of money because they're so much less expensive and and as it turned out as a side note the consumer drives are perfectly reliable they've been they've been great and worked perfectly well but was that a gamble or did you feel confident like what did that did you first of all did you actually model this out like failure rate redundancy how many extra drives we'll need all of this was that all a very thoughtful decision or was it like you know what we don't really have a choice we can't afford anything else let's just go for it so i'd say on the early days it was somewhere halfway between those two okay right which is we looked at it and said the the enterprise drives are so expensive that they're they're not going to work in this environment that these price points we're trying to offer a really cost-efficient service yeah and that math isn't going to work the consumer drives seem good and what we can do is we can make sure that there's reliability by having enough of them with them we can we can watch them and if the failure rates go up we keep we obviously swap the drives and we in time and we make sure that the durability is high but whether that was going to be affordable long term yeah was an open question at the very beginning and we pretty quickly found out that no it's fine because six months in they they were still perfectly good and in fact at one point we tested enterprise drives and thinking well it'd be good to know how much better they would be i mean they're going to be better yeah it ended up being worse no way now measurably measurably small sample size you know and they weren't like massively worse but they were a little bit worse and like these consumer drives are great so the there are valid reasons to buy enterprise drives sometimes if you need if you need a lot of performance out of a but for for our type of use case let's be clear what you know a lot of rights uh you know one-time rights and then a lot of reads reads when what you know it's it's a it's a very unusual usage profile it's not you know it's it's very different from say cloud first cap which is just hammering uh small rights back and forth uh for example and also just because we have we have uh a hundred thousand hard drives and and they're redundantly distributed so it because of that we don't you know a lot of times people ask us but which hard drive should i buy to make sure that i don't it doesn't die i'm like all hard drives die you can't count on that uh you need a backup or in our case you need redundancy so the on the hard drive shipping thing so what happened was we buy lots and we need a continuous flow of more hard drives and everybody in the world assumes that hard drives are like water you can just turn on the faucet and get get hard drives whenever you want and what happened was there was a flood in thailand which wasn't on anybody's radar because it's not like we had a lot of customers there or partners there or anything's happening there but it turned out that half the world's hard drives were made there or did you know that before the flood or was this like this was this was a shock to you as well like oh my god one geographic reason in the world my entire business is actually uh dependent on this was a shock to us and it was it's one of those where i think now about our approach to risk assessment and where we we consider lots of things in at different phases where you worry about and this was definitely an area where we said you know you you look at the big hard drive companies they've got global supply chains global factories all of it but this was this was a material and some of this has changed over time i think they have also adjusted with that but at the time hard drives all of a sudden became really unavailable it was really hard to get them and they were incredibly expensive and so what we need is the hard drive that you stick in a server and those were three times the price of normal and so we thought about different ways to deal with that but one of the things that we thought about was well these other hard drives that are external drives the ones that you as an individual consumer would buy and plug into my into my you know my laptop or my pc just to uh yeah just to store photos or videos or whatever yeah this is the the external would just plug it into your usb and call it done those were actually still pretty available and not too much more expensive i mean they increased in price but think 15 20 percent not 3x and and they were available at places like best buy and costco and others and we thought i wonder if we could just use those and so we we actually bought a handful of them and like you said we we cracked them open shucked them like an oyster and it got harder and and frankly for us they were they were as more valuable than oysters i mean they were they were the lifeblood of the business and so we tried them and they worked great and they were the same drives that we would buy for three times the price elsewhere and so we just started this whole process of uh buying lots and lots of these consumer drives from consumer stores like literally just giant shopping carts buying every hard drive you can on the shelf yeah giant shopping carts until until there was there it was a sign that said limit of two per person at which point it no longer was giant shopping carts it was lots and lots of trips that's great um let's point today where our our supply chain's a little more uh built up and resilient and uh and structured but uh this was eight years ago now roughly yeah that sounds right yeah i feel like it was right as uh when when you were advising my startup at the time we were we were we were still trying to uh trying trying to trying to look for people to admire and you were near the top of that list um one of the things that came out of this business of measuring other other all the drives and the consumer and the enterprise is that you started blogging about your results and that thing has become a hacker news tradition every time you post that thing it rockets to the top of the of the front page of hacker news it engenders all this vigorous debate uh i'm sure there are plenty of people who also come back at you going hey what did you put here and what is your methodology and um and what i love about it is that it's it is a data point a very articulate data point you don't make any any presumption to say that this is some something that needs to be true for everybody but it's like look for our purposes we have measured back out of a lot of different vendors and drives and here are results it's a great it's a great cultural thing and i just wanted you to talk a little bit about that like where did that come from what did the response surprise you was it obvious that you were going to make it a tradition and keep doing it so when we started the company i started writing blog posts yeah and frankly i had no idea what i was doing and the blog posts that i was writing were if i was lucky maybe my brother and my mom read them that was probably was worth it but you still felt it was worth it that's extraordinary i thought that the long-term value might be worth it but it was certainly unclear and it wasn't exactly obvious where it would head and then we wrote about the storage pod that was a blog post that we wrote about in 2009 where we actually open sourced that design yeah and that one blog post about a million people read it and it it blew us away that i mean for starters it blew us away that that many people care about a big red storage server come on it's cool but gleb i read it i'm not in the hard drive racket i mean you and i find it cool i just surprised that there are a million people like us and uh and so it got us thinking that you know that there are a lot of people that are hungry for information of certain kinds and if we have that information available to us yeah a lot of people will probably want it and the over time a lot of people proactively asked us for it they came and they said you know i use you for my storage you guys obviously know a lot about uh about hard drives and the liability tell me which one i should buy yep yeah i just got this question five minutes ago before we started so tell me which is the best product by you you're the expert and so we we you know after getting that question a bunch of times we started talking about uh whether we should compile our information and share it yeah nobody had right nobody publishes these reports and we were always wondering why doesn't anybody publish them we're not the only ones that have a lot of hard drives but in in a lot of people want the information and so at some point back in 2013 i believe we said let's write a report and we published one hard drive reliability report and we said here's the statistics and the data on all the hard drives that we have and we found that they followed a bathtub curve of reliability where at the early side they a bunch of them would fail then they would get into a steady state and they wouldn't fail often then toward the end of their life they would fail more and this is actually the bathtub curve is something that actually is a known statistical concept but it was interesting to see it in practice yeah so we published that and we thought of that as a one-shot deal we've published it we're done moving on and what happened was one it was wildly popular people loved it and then the other was people had all these questions they would say wait but can you slice the data this way can you tell me what happens if you are with temperature what happens with vibration what happens if you use it in a different version of your storage what happens if this that or the other and we just found that there was so much interest in it that what we decided to do was on a quarterly basis we would provide an update and so every year now for years we've been providing a quarterly update with whatever the latest hard drives that we have are yeah then we also have taken various slices and dices on that information yeah it's very it's just people love that it's it's so great for information you know i feel like i could talk to you about the the history of backblaze and and you know the startups that you've advised me on in the past but with with just a minute left i think you touched on something very important and it's a key part of Cloudflare you know if we're talking about how and Cloudflare and backblaze are roughly the same age they're sort of you know sort of born in that end of the previous decade one of the one of the one of the companies to watch during this uh during the during the tens and uh and is now now becoming a real player especially you know we haven't had time to explore but you know backblaze's b2 product is ahead on competitor s3 which is extraordinary uh that you know you're taking on um you know one of the giants of the cloud uh revolution but i wanted to end with just a few comments on this culture of openness and transparency and sharing everything with your users what's going on on the inside not just the technical specs but also the challenges the fact that you're you know that you that the heart that the thailand floods were affecting your business um you know sharing with your with your users look here's why we believe at five bucks a month we can sustain this indefinitely like talk about was that a deliberate decision is that something that evolved how did transparency become one of your touchstones so i think it's both i think it was inherently something inside of the company based on who we were and how we ran the company yeah and then we started doing things that would help it become an external thing so the publishing things like the blog posts around the storage pod and open sourcing that design and then frankly getting feedback and reinforcement from the community that hey we love this help build that culture and ensure that this isn't a bad thing it's actually a good thing and so we have there is a certain amount of building trust through transparency and a lot of the a lot of what we're going to get cut off here glad but i wanted to just say thank you so much for coming on