Originally aired on April 27, 2021 @ 11:00 AM - 11:30 AM EDT
Andrew "Bunnie" Huang is an open source hardware and an activist for digital rights and freedoms. He's the author of "Hacking the Xbox" about reverse engineering, "Essential Guide to Electronics in Shenzhen" and other books. He has a Ph.D. in EE from MIT and mentors hardware startups at the HAX accelerator program. https://en.wikipedia.org/wiki/Andrew_Huang_(hacker)
English
Fireside Chat
Interviews
Founder Focus
Welcome everyone. I'm your host Jade Wang and today our special fireside chat guest is Bunnie Huang. Welcome to the show Bunnie. Hi, it's good to be here. So Bunnie has authored a whole bunch of books on electronics and given lots of talks and also most of you probably know him as the original Xbox hacker. So Bunnie, you're currently living in Singapore these days, right? How's that been? What's living in Singapore like? Both like before the pandemic and now? Yeah, yeah. I mean, it's dark outside right now as you can tell. So it's nighttime over here. I've noticed that there's, when I finally catch up with friends from all around the world, I realize there's going to be a very different experience that a lot of us has had over the past year and a half or something like that in the pandemic. Here, like there are weeks we go by with zero COVID cases. Mass compliance is very high. They have a decent contact tracing program here and they have kind of basically have the virus on lockdown. And so when the government started like lifting restrictions and like allowing gatherings and stuff like this, people were of course willing to do it. But some of my friends in the US, I think it sounded like it was more of like an opinion, if you wanted to lock down or not, or if you want to wear a mask. And so early on, people had to convince themselves it was an okay thing to do. And so when the government started telling them, you can start getting together now, like, can I really do it? Or are they just saying I can do it? Is it like an okay thing to do now? And so there's a big, it seems like a lot of people having this existential crisis, which I didn't have at all. Like when the government's like, oh yeah, you can go ahead and have groups of eight or something like that. Or you can go out and do these things. And it was like, okay, fine, it's not a big deal. Because you just figured that's the thing that you would do. It's been a very different experience. I mean, Singapore is a small place, though. Like I haven't traveled more than like a 10 kilometer radius in about a year and a half, which is really unusual for me. I'm more typically used to, you know, darting across the globe and that sort of thing. So it's been, it's been interesting. I learned how to do like, you know, gardening and cooking and all these sorts of things. Wow. Did you do sourdough bread baking too? I didn't, I didn't join the sourdough trend. But like, we were like, I also wanted to do some hydroponics. And so we had some like, Oh, nice. Got a raspberry pie and control the pump and then put some lights on it. I had some spare LEDs and put together and then we started buying the chemicals for mixing the things. And I've learned all these things about like crazy things about plants I never knew before. It's like, it's been interesting. And I had this time lapse camera. What did you plant? Oh, we planted, we planted all sorts of things. Some of them successful, some not. We had like lettuce, we had basil, sage. We've had tomatoes, we've had thyme, a few of them, some of them did not grow well at all. Some of them are like, basil has been like gangbusters. We have so much fresh basil now all the time. So it's, it's, you know, some of them better, some of them worse. Nice. Oh, by the way, I totally forgot to do this earlier, but any viewers, we welcome audience questions. I will be taking them closer to the end of the show. The information for calling in or emailing in your question are also down below. Let me scroll down a few pixels. So, so let's, let's chat a little bit about some of the, the macro trends in the past decade, especially around, around hardware and hardware startups. So being from the software startup world myself, one of the things that I frequent, I have frequently heard from investors and also from other founders is, you know, hardware is just like really hard to do. And so there are investors who just like won't get into hardware at all, but it's still, it feels like there's some kind of phase transition that's been happening, right? Like it's not as hard as it used to be. Can you talk about some of the forces that are at play here? Yeah. You know, I, I feel like the funny part is that I feel like software is way harder than hardware. Like if you actually looked at, you know, on the first generation iPhone, I kind of knew a little more about what's going inside. I heard that they're the guys, like the design team for the circuits and stuff, it's like five people or something like this, but there's like 200 people on the software, which sounds about right in terms of like the ratio that you would expect for like a very complicated software product, like an iPhone versus like the hardware design effort. I mean, back then, of course, the iPhone was just putting together off the shelf commits. Now they have custom CPUs. I mean, they have a huge department, right? So it's, it's a different ballgame now, but you know, the success or failure of a product is typically more dictated by the software, right? And software has a tendency to grow complexity without bounds. And that in itself starts to become a really big problem. And then release management and all, you know, updates, all this sort of stuff. Hardware has a nice thing where basically once you get it working, it works right. Then you just have to manage the supply chain, put it together, make sure it keeps on working. You don't get mysterious like hyzen bugs and all these sorts of things that, that, that like software only happens on prime number of days. Yeah. Prime number of days or like, you know, rollover times or like, you know, someone decided to pull a node package and the world falls apart underneath you or something like that. Like if that doesn't, you know, someone doesn't say, Oh, we're going to just change Maxwell's equations today to a new version. Sorry. Like that doesn't happen in my world. So so I think the main reason why people have, I think said hardware is hard is because it's not as immediate as software. So software is a little bit like, Oh, I, I want to write a program that counts numbers. I'm just going to type on my computer and five minutes later compile something. And I have the, if I say I want to build a clock, okay, well I have to buy some parts and a little, some do some circuits and whatever it is. And then, you know, there's a, there's a little bit of a longer lag between building a hardware counter versus something in software. But once you, if you have the right ecosystem around you of, of, of services. And so the key to hardware is that you don't do everything yourself. Like I don't literally go and solder my boards by hand. I don't literally go and you know, the circuits, you know, or like run a CNC machine and take, you know, cases and stuff like this. Right. You know, you find partners who specialize and do things as a service, like circuits as a service, CNC as a service, like assembly as a service, you find all these basically, you know, as a service providers, you build a supply chain together. And then once you have all these service providers, it's a matter of just sort of like giving them enough time to do a good job of it. And, and, and it actually kind of runs pretty smoothly. And I think part of the problem was early on Silicon Valley had sold you on this idea that we would all have digital fab at home and we'd be able to 3d print our everything in the home shortly or something like this. I don't remember those, you know, 10 years ago, wherever's my magazine articles about how, you know, throw away the store. We're going to like digitally print everything and download all the lines, stuff like this that ever happened. So a lot of reasons, but I think, I think, I think people are now figuring out that once you have the right kind of service providers and the right modules you put together, actually hardware is more approachable. And combined with the fact that actually software has just gotten as far as the out of control complex, like, like, you know, also, it's not just that hardware got a little easier software also got like way more unmanageable as a, as a part of a product and all the VCs want to moat and they want something that makes different and all these different sorts of things. And even software in itself is no longer like a simple, like I'm going to compile on the computer, run it. Now, the first thing you do after you do is you have to push it to a Cloudflare, or you have to do it to an Amazon, or you have to like, you know, pull in something from GitHub. You have all these other service providers you're now dependent upon. So it's no longer like you, you have like this little, you know, for I equals one to 10, you know, I equals I plus one, I have a counter type of thing. It's like, I have huge amounts of mechanisms and vendors. And like, if I want to do ML, I'm renting really expensive things in Google Cloud for TPUs, like, you know, thousands of dollars an hour, like, it just, you know, and so now you look at that, hardware looks kind of cheap, actually. It's just you and the atoms in front of you, right? So let's, let's talk a little bit about the, the product development process and how that differs between hardware and software, right? Like, you know, usually we think we often think of software developers, you know, hacking together an MVP to get it in front of the potential user to get their feedback pretty quickly. How does that loop work in hardware? It's, it's similar. It's similar. What actually one of the key things to being successful in hardware is learning how to kind of sketch in hardware. So, and a lot of it comes from kind of not necessarily building full products, but even like, even before you build a product, you get like a lump of clay and you say, should my phone be about this big? How's it feel in the hand? Where are the buttons go? And you just carried on your, just a lump of play in your pocket for a while with the right shape. And you, and you learn a lot about the interaction with the product. You start with mock-up UIs and different types of things. You break it down into its component pieces. Okay. I want to use you know, some type of display, some kind of new display. You don't build a whole product around at first, right? The UI and then like, Hmm, that display has really bad contrast. I mean, like you don't buy the element, right? You put a piece of carpet around the bezel to hide all electronics. And then you just do like a big up between the two and you sort of continuously integrate your way through the different hardware features early on. Right. And then at some point in time, you do have to have the discipline to say like, this is, this is it. We've, we've, we've made a cut and we're not going to change things anymore. So that's one of the most, I think the most difficult things about software versus hardware is that software people love to push updates eternally for everything all the time. And so nothing's ever actually done because you can always push an update. You know, it's like, well, let's just pump it to the next release. It's fine. Who cares, right? Hardware is like when I, once I ship it, it's, it's there, like it's on your side of the world. If, if I made a mistake, I'm either going to your house and soldering and fixing it, or I'm paying a lot of money for the realist logistics. I mean, I don't know if people know, but like, I mean, like it's really, it's pretty easy to get stuff to your home through Amazon. But if I want to get one thing back that broke back to the factory, to my place, it's orders of magnitude more expensive. I mean, it's to the point where like back in the day when we used to work with like Best Buy, remember Best Buy, that thing, that store that you actually walked into and bought stuff from back in the day, right? One of the contracts they had was that they would actually make you build like 5% more product and just give it to them at the same price. Like it was built in. But what would happen is that when you, when a customer came and said, this didn't work, they would actually take the product that was broken, abandon it, and just give you one of that 5%. And it wasn't even worth sending it back to that. They just had this reserve they kept on the side because it was cheaper to build 5 % more product, keep it there for the reserve returns, and then throw away what you thought was going to, or you went to Best Buy, you saw the As Is pile, they always had this As Is pile. Like they actually very rarely sent things back all the way back to the manufacturer. If you thought, think about how hard it is in a big screen TV, if your big screen TV breaks, that does not get on a boat. There's a bajillion parts in there. It's not just a, it's huge. Have you tried shipping a package that big? Like even like across America, now think about sending it to like China or something like this, through customs, through all sorts of stuff, paying the taxes and the tariffs and duties and all sorts of stuff to get it back into the country. You're going to pay many times more than the value of the TV itself to get the return back. It just doesn't, and so it's like, so the point is once you ship a product to a user, you ship the product to a user. So you take the release process very seriously in hardware. Hardware has a very strong validation cycle and a very, like, you know, we're going to lock things down and ship a product. And then, and then of course there is sort of the update cycle, but it's more of like, oh, why does the iPhone 12 so much better than iPhone 11? I don't know. It's got, it's purple, something like this. Like, you know, but then you can roll in little bits of features incrementally and users for better or worse are willing to throw away gadgets and buy new ones. And that's sort of the release cycle for hardware is, is, is you sort of say, we, this, this feature didn't make the cut for this generation. We're going to put it into the next generation. Hopefully users will buy that thing and we have enough planned obsolescence or whatever it is to make that happen. But, but when you, when you actually say, okay, we've, we've got the feature set for this product set, like it's really set. You don't just say like, oh, we found this really cool thing. It would be awesome if we could just add another camera to this. And I know all the things were in a shelf, but let's pull them back and patch them. And it'll be, it doesn't happen. But that, that, that makes me think of something else, which is, so when you talk about planned obsolescence and, and the process of pushing updates, like how does the, how do people in the hardware industry think about the environmental impact of, of the manufacturing of these components and, and the disposal of the obsolete electronics afterwards? Yeah. And it's, it's now you're getting to where the sausages are made. I mean, I, so, so I think, I think, I think the problem at the end of the day is that I think, I think the, a lot of companies will say they have like these green programs or whatever it is, but because a lot of the costs of this are externalized and it's so beneficial to allow these vendors to sort of keep churning things out. Like there, there, there's, you know, there's, there's obviously the waste that you have, you know, at home, cause you're gonna throw something away, but I mean, God, like the amount of waste that just happens on the manufacturing side, like, you know, we, like, it's, it's like a drop in the bucket, right? Yeah. Like, like, you know, if you, if you have like an assembly, like I remember hearing this thing early on in one of the iPhone generations, they had the spec for like the glass bezel had to be something less than 30 microns gap between like the glass bezel and the outside because it's 70 microns is big enough to catch your hair, be held up to your, to your, and then there'll be a bad experience. And that's a really tight tolerance. And, and that mating happened at the very last step. And if it was off, you couldn't rework it. They just threw away those phones. It was just a mountain of phones that just, they just threw away, right. Because they didn't meet that spec, right. And that was like a not externalized cost. It was sort of just built into the product. It was part of the, it was part of the experience and users loved it. And it was wonderful, but you know, we never talked about the, the huge amount of stuff that went into those landfills. Right. I mean, and I, I mean, I've personally experienced this problem where we had some plastic cases that were being injection molded. They're like, they cost like 60 or 80 cents a piece or something like this. And there was like a little blue line on the outside that wasn't quite perfect on about 40% of them just threw them away. Right. Cause like the customers would have been angry because they bought like a hundred dollar gadget and they had like this mark on the front, right. For a 60 cent part. And so if you think about my incentive in this whole thing, it's not, you know, I'm not going to try and convince a customer to do their part for the environment and accept these pieces. Right. It's going to be like, it's, it's, you know, effectively 30 cents, but it was a mountain of scrap plastic, which did eventually get recycled and whatnot, but you know, it's not, it's not great. Right. But the thing is, is because all of that, all of that decision -making is externalized. Like customers are not willing to accept a product that's slightly worse, slightly marred, slightly miss, you know, or not even the newest product anymore. Everyone has to have the newest, best thing. Cause I deserve the best thing. And I paid money for it. And, and in a way, as long as we do that, we are willing to accept legally externalizing costs. Like some places like Europe, they actually force some manufacturers to build in the cost of taking like the refrig out of your house after they ship it to you. I think that's probably a really good way to sort of make sure they have a whole closed cycle ecosystem built into it. And then they have an incentive to actually think about how to take apart the machines, how to like recycle them efficiently, because now part of their costs, like the reverse logistics is part of the supply chain. But if the, if you have a system where I sold it to you free on board, like, you know, I don't have to care about him for now. I get to report profits to my, my shareholders. It's no surprise that we have these types of environmental problems, incentives. So that's a really a matter of incentives in the industry. And it's just the, you know, it's a problem we have to deal with it, but I don't think it's like, if you say what the attitude is of the harder manufacturers, it's like, you know, I mean, we've got to make a living. Yeah. I mean, the, the mountain of iPhones that are just, uh, slightly capable of catching a hair. I mean, like, do they get, is there any chance that a bunch of people will like, you know, jack them up, like take them and, you know, sell them as products on the, on the way to the landfill? I mean, where do you, where do you think those markets are? Okay. In a way, no part of the cargo is wasted in China, right? Right. Yeah. In a way it does. Eventually there's a very efficient, like, you know, I mean, the good news is like, you know, that mountain of plastic, there is like an IE in China who would take it and figure out a way to grind it up and sell it as lower grade plastic for another injection molding process. Right. There's, there is like a system that exists that as long as something has an economic value, it, there is some ability to recycle it, but it's very externalized. Right. It's not, it's not, you know, in, in, in a lot of that incentive really just comes from, you know, people just trying to improve their profit margin at the end of the day, locally within the China, like that boss who has now the mountain of plastic, you know, what China will do is, is they will like, they're like, you can't dump this plastic. They'll find them for dumping it. And so then he'll find a way to recycle it. So that's typically what China does to sort of cut back on the waste. They see people who are egregiously dumping and they find them and they find a way to recycle it. I mean, that's just, that's a very China way of solving that problem. So they, you know, leftovers for a new meal. Yeah. Yeah. Yeah. Yeah. They, they, they tend, you know, they have ways of solving the problem, but they're not, it's all extremely like distressingly informal and very decoupled. And, and, and I, and, you know, you can tell, I have a lot of frustration angst about the situation because, because I also, I, you know, I, I literally have to deal with customers who return stuff because there's fingerprints on their, on their product. Right. You know, you, you have that bar you have to meet. Right. And, and, and, and I told you how expensive it is to take that product back with the fingerprint on it. Right. It's got a fingerprint on it and I'm going to spend $200 to get it back to rub the fingerprints off and you couldn't rub them off. Right. When you open it up. But, but I mean, I think you can't get on the Internet and write, my customer was an idiot and they're unreasonable and they're causing all kinds of e-waste and this is, they're wrong. Like you say, no, of course you're right. You know, and now you go to the factory and say like, you know, we have the standard and if you don't need it, throw it away. Right. That's like the, the, the system, the incentives don't exist. The world isn't structured for me to fight that fight and put food on my table. Right. It's like, it's, it's very, it's a very tough place for someone like me to be, to, to interpret. I have to sort of, unfortunately I have to live with all of these cognitive dissonances in my head because we are in a, in a non-stable state as a, as a global economy. Well, let's, let's dive a little bit into, into the ecosystem of Sensai products, right? There's, so in the WIRED documentary, you described like what happens as a bunch of smart engineers at a company get kind of fed up with all the processes that they have to deal with. And then they, you know, they, they, they figure they can make something as good or better if they just quit and strike out on their own. And this, like the shape of that story is like the same shape of the story of the Fairchild Semiconductor story. Right. Like, is it really the, like, is it the same, is it the same thing that just keeps happening over and over again, but in a new location this time? A hundred percent, a hundred percent. Like history tells a story over and over. I mean, you can take it now back to like, you know Americans taking steel process from Europe in the industrial revolution and trains, right. You know, and then you can take it back to, you know, all the, you know, the Japanese with the cars in the eighties, right. And if you remember them sort of taking over, you know, GM and Ford and Chrysler and then Toyota coming up nowhere and all the, you know, bitching and moaning about them stealing their technology and processes, like, like, you know, every, you know, there, there is sort of like just this cycle that you see where, where you have, you know, some kind of established players, right. And then some upstarts come along and they benefit from, you know, of course learning a lot from the big players, but then they're not, it's actually very, I think one of the misnomers is that, you know, they do a straight copy, right. Like a lot of people say that, you know, they're slavishly copying exactly what I do, wherever it is. Like, you know, if you actually look on the inside of their devices, like a lot of the theory behind the, you know, the Shanzhai guys was like, look, we were working for Motorola or RIM or Nokia at the time. And we're like, why are we selling phones for like, you know, $200? We could sell them for $30 and get them into India or like, you know, places that can't afford these really expensive phones. And they just cut all that fat out based upon the same core design, but they, they've just actually kind of fundamentally built a very different product that Nokia wouldn't even put their name on because it's such a, the quality. It's fundamentally disruption, right? It's like attacking the down market position and then moving upward. Right. And in a way, if you were to say, okay, well, let's say we had perfect IP protection and Nokia could have prevented that from happening. I don't think Nokia would have an incentive to have created those low cost phones, to create those new markets that eventually created a different type of disruption in and of itself. It's actually sort of an untapped potential in a way, right? So, so there's a certain, there's a certain inefficiency in the, I mean, I, I have a very long rant about these types of things, but, you know, but when I'm not, I think there, there's definitely the IP system has strengths, but also has weaknesses. And one of the big problems is that patents have a 20 year life term, which in technology is like an eternity, right? Like, you know, I don't remember. That's like a generation of people's careers. Exactly. And to say that, you know, someone gets a monopoly on that for that long, is, is, you know, isn't, doesn't create a vibrant enough ecosystem. You need to have an incentive for people to move and innovate, change and, and do things. And so, and so, you know, some, I don't think we have to like trash the whole thing, but definitely there's like, there's, there's like a little, you know, the, the expansiveness and sort of the terms need to be reeled back a bit. We need to have a little more flexibility to play with this and figure out what the right balance is to go ahead and bring innovation back into the ecosystem, make it more limber, a little more competitive. Like another problem is that like, you know, if I want to file a patent, a proper patent, like not just like literally just pay the $300 filing fee, but actually pay a lawyer to draft it. So it actually holds some legs, right. You know, it's a lot of money. It's like 20 grand, like, you know, some huge amount of money. You know, I can, I can prototype a whole product for less than it costs to even file one patent on the product. And then the VCs also, you have to have a portfolio because like, you know, you know, all the things they need to, you know, whatever it is, you know, protect your product. And so that's actually a really big drag on, you know, not just the innovators time, cause you have to think about it. You have to process the papers, you have to, you know, get it through the system and you have to protect it and you have to then litigate over it. Right. You know, it's very expensive. You know, I could have reinvented that product five times over in that. And that's basically what you see going on in the Shanzhai ecosystem is like, okay, well, you know, maybe someone can copy our stuff. We put it out there, but you know, good luck. It's a distraction for you to copy my stuff. Cause while you're copying that, I'm making the next generation. And then I'm very happy for you to take over the previous production of that, because that just makes a bigger market base for my next generation product. Right. And we'll compete. Right. And that's a very like, you know, sort of Shanzhai. It's like the Shanzhai mind frame is almost like the software mind frame. It is. It's very, it's very much about the, you know, yeah, it's very much about a sort of, like I said earlier on the, the reason why hardware seems hard is when you don't have the right ecosystem to sort of continuously integrate your product in real time. And, and what the, what the Shanzhai have that is kind of missing in Silicon Valley is they have that ecosystem. Right. So they can very quickly iterate. So a lot of people, when you remember the, the hoverboard, those little bit, those two wheel scooters, everyone's like, who invested the billions of dollars in China to like create the factories that build them? Like overnight, there was like, you know, you know, several brands, it was like this big hodgepodge of stuff. And there's these articles of people like, you know, these people are obviously stealing someone's really rich. And what it turned out was actually no one invested any of it. They had the ecosystem, right? It wasn't the reason why you had so many of them is it wasn't hard to make them. The actual idea of the two wheel hoverboard was it's actually easy to build in the one wheeled unicycle, which was already being made in droves in China before the two wheel hover. It's easier to make a thing that balances on two wheels than one. Right. So when they saw, you know, the two wheel hoverboard on TV, they're like, oh, people want one of those. All right, let's make a bunch of them. And a whole bunch of factories tooled up. And within like a few weeks, they're like banging them out like gangbusters. Right. And so and because they had that means of production, they had the ecosystem, they had the flexibility to go ahead and essentially, you know, deploy in real time. It's sort of like being able to it's a it's a little bit I guess it feels a lot like deploying software when you have all the ecosystems of tools right around you at your fingertips. Yeah. It might even be slightly easier than software in some ways, particularly these. I mean, I'm not myself. Not I don't do big software ops. I'm a hardware guy. Right. But I have had to do some some a little bit of like website maintenance and deployment on, you know, with like different cloud tools and whatnot. I'm just like, oh, my God. Like how does this like. Yeah. That's a it's a Rube Goldberg device. Yeah. The kindest way I can put it. Yeah. I have like. Yeah. Oh, my God. I have like this diagram here that actually printed right here. I was like, I have to remember how this thing comes together. It's like I like like this is the process we're deploying an update. Right. I'm just like how how does anyone remember this stuff? Like I can't. It's not. I don't know. I feel like this is way harder than it is like soldering together a board and sending out like that. I had I but, you know, I'm a hardware guy. Maybe I'm just deficient in software. I feel like, well, yeah, I mean, you're you're proficient in the things that you're already experienced in. So and like that diagram that you drew, that Rube Goldberg device, imagine that like every couple of years, all of that changes, too. I know. No, I'm so afraid right now. It's like I keep getting emails from people like, oh, like, you know, this is a new way of API or this bucket is going away. You have to go to these new longer names. I'm like, well, what if I don't want to? I don't have time to deal with this. I just want it to work. Guys, just please. You know, when I when I go ahead and I buy copper ore from 10 years ago, it's still copper ore today. I can't get like, you know, a website to work 10 years later on. But that is one of the fundamental questions in in our industry. I don't know how you guys do it. It seems like insanity to me, but all right, you know, continuing education. So we are almost out of time, but I did want to ask you about some fun questions. So Burning Man, one of the things that's really struck me over the past period of time, decade or so, is, you know, the first time I went to Burning Man, I was surrounded by lots of things that are on fire and like glow sticks. And then over time, there was like a couple of years phase transition. It suddenly became LEDs everywhere, LEDs on the ground. It was like, you know, a hurricane had picked up a Shenzhen warehouse and like reassembled it into a city and dropped it in the desert. What happened? It's almost literally correct. Yeah, I mean, I think, I mean, there's a few, there's a few dimensions. I mean, first of all, obviously, LEDs became insanely cheap, like, like, you know, the production of three, five semiconductors. A lot of progress is made in that. Thank you so much. We're out of time. Thank you so much for coming on the show. It's been a great episode. Sure. Yep. Thanks. Thanks for having me. Thanks. You have a good night. You too. Bye.
