CIO Fireside Chat with Adam Healy

All right. Hello and welcome to Cloudflare TV and also welcome to CIO Week. Here at Cloudflare several weeks during the year, we take some time and celebrate a different theme and announce new products and new features that focus on a theme. And this week's focus has been on all of the different problems that not just CIOs have, but really any organizations have keeping themselves safe, keeping their team members productive, and just focusing on the mission behind their organization. And to that end, I'm really excited to have Adam Healy here on Cloudflare TV today. Adam, welcome. Thanks for having me. Great to be here and great to be doing this with you and Cloudflare. We appreciate having you on the air. We're doing this live today. We were originally going to prerecord this. So if we mumble or stumble on ourselves, forgive us. But just to get started, why don't you tell our audience who you are, what you do, and what brings you to this conversation? Yeah, absolutely. So as you mentioned, my name is Adam Healy. I'm the chief security officer at BlockFi. I've been in this role about 20 months now, and it's been quite a journey. At BlockFi, we provide both retail, so basically consumer and institutional financial products across a wide array of different offerings. Everything from our Visa branded BlockFi Bitcoin rewards credit card, where you can spend dollars and get back Bitcoin rewards instead of things like frequent flyer miles or reward points, etc. And a number of different account structures, yield earning accounts along those lines for retail. And then we also have different products that we offer our institutional clients, things like different types of complex financing arrangements and loans and things along those lines. And how long have you been there at BlockFi? Well, like I said, I've been here about 20 months. It's been quite a journey. When I arrived, we were managing about a billion dollars in assets on our platform. And that sounds great until you think about today and that 18, 20 months later, we're managing somewhere in the neighborhood of $15 billion in assets on our platform. So in some measures, we've about 15X the business in that year and a half period. And as one can appreciate, especially you and Cloudflare can appreciate, you don't hyperscale a business that quickly without accumulating some tech debt and having to make a lot of decisions really quickly. So the journey has been great. We really have invested heavily in partners like Cloudflare, but also building out a lot of our core engineering and security functions. For me, which maybe is a bit unique for a head of security, and partly why I'm here speaking with you today is I actually get to oversee a pretty large swath of our engineering organization. So I have all of our traditional kind of security programs. So red team, blue team, purple team, GRC, things along those lines, but also oversee our machine learning and data science team, oversee our cryptography and cryptoengineering team, oversee our corporate IT team, and also oversee our kind of all things cloud engineering. So DevOps, SRE, things along those lines. Wow, that is a huge portfolio. Within the security side, I guess, what had you interested in a career in security? Were you working in security before you joined BlockFi, or were you in engineering and this has been a new part of the role? How'd you find your way into this field? So that's a long story, and we probably don't have a lot of time for that. But the short version of it is, I think instinctively, I've always been focused on protecting. And I think my journey, my professional journey really started around 9-11, and joining the Marine Corps shortly after 9-11. And that kind of put me on a trajectory to spend a number of years in the government space, both as a Marine, then later as a contractor, and then even after that, as a civil servant, as a federal employee. During those kind of three very different roles, got to do a lot of things, got to work on some very cutting edge technology problems, I got to work on some very interesting intelligence problems, and also got to take all of my experiences and spend about three years deployed overseas in a couple different countries. So that time is very formative. And then from there, I was able to transition out to the private sector, and work at companies like Microsoft and work at companies like Palantir. And then kind of, as crypto is really heating up back in 2016-2017, was able to pull on a lot of those different skills that I had built over about 15 years, and kind of go full time into crypto. And here I am today. And those are some incredible organizations for the whole career. What about BlockFi's mission, when you were looking into the crypto space, intrigued you to come join that organization? So me joining BlockFi was, there's actually a bit of a funny story in there, by itself. It was, BlockFi was having some security challenges in the first half of 2020. And Zach Prince, our CEO, messaged me, and I had known him previously, he sent me an email. Basically, it was, hey, what are you up to? And from there, I had a couple of conversations with Zach and Flory and some others on the executive team and decided this was the right place for me. And I think beyond the personal relationships and working with great people, which is obviously always very important, some of the technical problems that we're that they're trying to solve were very interesting to me. How do you support a global client base? I think right now we have somewhere between 1.6 and 1.7 million account profiles on our retail account profiles on our platform. How do you support that spread across 50 countries? And how do you deliver content securely? And how do you make sure that you're providing high assurance, financial services to these to these individuals? And on the other side, we have a very different archetype of clients, we have institutions, and we have multiple hundreds of institutions, these are top tier institutions that that probably all of the listeners here are very familiar with household brand names, certainly in the financial sector. And it was a combination of those challenges, and understanding the scale story and the hyper growth story that BlockFi had been on, even prior to my arrival, and then where it was going, kind of brought all of those, all those different things that I'm interested in together. And then when you layer on crypto, just in general, it has a lot of very unique security problems, something I always tell security engineers, when I'm interviewing them, and I've, in the last 18 months, I've hired about 65 people. And it's a lot of interviewing, it's a lot of resumes, thousands of resumes. And one thing I always say is, we have every problem from a security perspective that a fast growing 1000 person fintech has, we have PII data, and PCI data, and credit data, and high value APIs, or third parties that were integrated with we have all these problems. But then we have $15 billion in Internet money that we also have to protect. And it creates a very complex risk profile, and requires a very sophisticated security program. And when you're in those interviews, and building out that team hiring 65 people, what are you what are you looking for, for a person joining the security organization at BlockFi? Really good memes on Twitter, usually. I was actually speaking at AWS reInvent, I guess, two weeks ago or something. And somebody asked if I was on Twitter, I was like, absolutely. I like crypto Twitter, the memes are amazing. We actually have a team internally that we call the meme team. It's great. But no, honestly, I have actually hired two people that have DMed me on Twitter, that were like, hey, I sent a resume, haven't heard back. Here's kind of my profile. What do you think? And I think with that, to me, I mean, they've actually worked out really well. And they're some of our top engineers. But what I primarily look for is aptitude and hustle, and not hustle in kind of the overused Silicon Valley use of the word like hustle culture, but really someone and how I would define that as an individual that will own the company's problems as their own. Someone who's willing to put their hand in the air, someone who's willing to think in a nonlinear manner about complex technical and complex organizational problems, and help provide solutions and not just surface needs. And we've, we've been really successful at that I've got to handpick this security organization. Really, since the first day I got here, when I arrived, there were about five folks on the security team. And now we have a fairly large and sophisticated global team running 24 seven security operations. So primarily, though, aptitude and hustle, smart people that are willing to work hard is probably the two most important things I look for. It's a good formula. And so once once folks are on the team, what's the structure like? Do you have people own different verticals things horizontally? What's the organization Yeah, so we spent a lot of time working on this. And you know, I wish I could say we got it 100% right every time. But when you're when you're growing an org this fast, we definitely have had to reshuffle some things and reevaluate where where we're structuring people. For me right now, I have a number of different VPs that report directly to me one covering specifically cybersecurity, one covering specifically some of our cryptography services and crypto engineering services and custodial services that we leverage third parties for. And a couple others focused on things like cloud and SRE DevOps and also then one that reports to me for things like internal services, GRC project management. So it's a fairly normal organizational structure, I would say. And it seems to work really well for us. And it's also something that is really easily explained to regulators to insurance underwriters to partners as we're looking to do more and more business across the spectrum. It really enables us to tell a pretty good story. So I think we've kind of taken a fairly traditional security organization structure and then slightly tweaked it under the covers to fit our unit name. That makes a lot of sense to me. And within that security organization, if you are just kind of looking to the rest of your the BlockFi team, what's the relationship like between security and product or security and it at BlockFi? Well, the good news is I own IT too. So it makes it pretty easy. And I actually think it's a model that works. I've seen it work really well at a number of other companies. So many of the problems that a security team deals with today do originate from external threats that are coming in through email. We all see the reports from Verizon and Microsoft and everyone else. Phishing is still a problem. And we have a lot of controls around that. But occasionally things still do get in and we have to respond to them accordingly. And so I think putting IT under security actually solves a lot of those problems. And it also creates a culture within the security team that you can't just be a department of no. And I think that's a very dangerous narrative for a security team to kind of fall victim to. You have to be a security organization that enables the business. You absolutely have to be partnering with the business to make thoughtful decisions. Something I say a lot is you have to be a little comfortable being uncomfortable. And what I mean by that is risk management is a real thing. And not everything is going to be 100% secure the way that the team would like it built. So partnering with IT is pretty easy for us. And it's worked out our model of having IT actually report into me actually has worked really, really well. Partnering with engineering and product. You know, I think every organization has a healthy tension or should have a healthy tension between those that are building product and want to ship it, and those that are safeguarding the organization. And in bad organizations, it's a tension, good organizations, healthy tension. And I think, you know, we have a, we have a fairly large engineering organization. And our head of product, at least on the retail side, or GM of product, has recently joined us over from PayPal, where he spent a number of years building product there. So right now, I think we have a very highly cohesive and very high functioning relationship across all those organizations. But at the end of the day, that's a start at the top, right? You know, that can say, and mandate and lead us to a certain direction, but kind of that next layer down at kind of the, the head of security, the head of product, the head of engineering levels, that is where we have to have good relationships, and then push down a culture of partnership. And there's something I say a lot is, you know, we're not a, we're not a security company. We're not a IT company. We're not, in many ways, we're not really a tech company, although we could talk about fintech and things like that. But what we need to be as security practitioners, as engineers, as, as product people, whatever that may be, as mechanisms to provide better financial services to our clients. And we're a financial services company, in many ways. And that is the goal of what all of this, you know, money that we spend on building technology should be aimed at it to helping run the business and, and build the business. Yeah, we, you know, Klausler, I think, has a very similar theory about how product and security should work together. We, you know, we build security products, it adds kind of an additional layer of, like you, like you say, in a good way, healthy tension. But we frequently we're building products that solve a need that our own security team came to us and said, Hey, you know, there's no tool out there that does this, or we're actually doing engineering with them, which is really exciting. I guess, kind of on to that end on the Klausler side, your organization uses Klausler, how did that start? Yeah, so, and I would actually just circle back to one point you made is, when I say we're not a security organization, we're not a security company, I do mean it. But at the same time, a core competency of ours, and any company that is in the crypto space, must be the safekeeping of client digital assets of client cryptocurrencies. And we've invested incredible effort, money, time into that as an organization, and we take it very seriously. So when I say we're not a security company, what I really mean is, we are that that isn't we're not selling security products, right? But we are a one of our core competencies as an organization is really that the safekeeping of client digital assets. So but but to your, to your question, how did the relationship with Klausler come to be? It was actually in the middle of an attack. And it was I often say that every problem BlockFi has ever had is thoroughly documented on Reddit and Twitter, because our clients are are amazing people and have have no worries about calling us out on social media. Yeah, so and I like it, it keeps us accountable. It keeps things transparent. Just, you know, myself, I've even engaged on social platforms directly with folks that have raised concerns. And I think it goes a lot to say, hey, our executive team is accessible. And we're willing to chime in. And, you know, Zach and others across the C suite, all do the same thing. So what what happened, though, and we had had a I had personally had a relationship with Klausler, my previous company, and had known many of the folks that actually have helped us get things off the ground here at BlockFi. And we were having a denial of service attack. And we recently published a case study with Klausler on this, I think it's a, it's a great case study. So we're in the middle of both the denial of service attack on a series of our Internet facing infrastructure. And also, a bad actor was abusing a particular API that we use to facilitate our signup workforce. And these things are happening at the same time. So presumably, same threat actor, but you know, threat actor attribution is difficult. And it was causing massive degradation of our platform and was also causing a very public but very public reputational incident because of the impact to our signup infrastructure. So we had been struggling with another vendor for a while to get things like bot management, WAF, other technologies in place. And with a phone call, and I woke up someone on the Cloudflare team, fairly senior in your organization pretty early, I think it was, you know, like a 6am email San Francisco time. And the response was, what can we do to help? And that's exactly what we're looking for. And I've been thrilled with the partnership that we've gotten, since we've since that relationship has started. And the what can we do to help turned into great understood, give me 15 minutes, and you're going to get a zoom link. And we joined a zoom call with, you know, some of the best engineers that Cloudflare could rally. And it was first thing in the morning for them, and they cleared their schedule. And we spent about, I want to say it was about six hours on the phone on and off with them. And between working, getting contracts in place and getting all of that stuff worked through, but then also doing all the technical migration. And by the end of the day, about six hours later, we, which definitely wasn't the end of our day, because we were still in the middle of it. But by the end of some days, we, we had Cloudflare deployed across our entire environment, we had done a number of DNS migrations. And we were behind an enterprise contract at Cloudflare, allowing us to access all the bells and whistles from bot management to WAF to CDN. And basically, the onboarding process, I don't think I've ever had a as fast of an onboarding process with with any, with any partner that we've worked with. But six hours, I think is the record. I, I've seen shorter and we can pride ourselves in making that efficient and easy, especially, you know, in the event that every minute, like in your circumstance, every minute counts, right? You're keeping an eye on that clock. Well, I mean, every minute, every minute costs money as well, right? And there's, there's this, every minute is, every minute is incrementing our reputational damage. And also incrementing our financial impact. And those two things, you can always recover, you know, conceptually, you can always recover from the financial impact. The, the reputational impact is very difficult, right? That's a client trust problem. And those things are very, very hard to solve for once, once you've had an issue. Absolutely. Like you mentioned earlier about, you know, how, how core security is to everything that you do, the trust that your customer, your clients place in your organization, that something that's hard earned and hard kept, right? Absolutely. Absolutely. I think, especially in crypto, clients, users of clients, and there's a huge spectrum across who's invested in crypto. And we have, you know, lots and lots of research and stats on that, as we think through marketing. But the reality is, a lot of the demographic that is invested in crypto are generally very tech savvy, and very security aware. You should see some of the messages I get on Twitter, around like very, very, very thorough and detailed questions asking about how safe their assets will be. And, and that also lends itself to a unique problem when you start to instrument reputational risk, and how to prevent it or how to recover from it, is these individuals are generally more tech savvy than just the, you know, than someone who is outside of crypto. And it creates a difficult challenge, a difficult needle to throw. And how do you to kind of make sure that your organization's keeping that, you know, keep keeping your commitment to that core competency of security? What do you use inside of your house to keep BlockFi safe? I know Cloudflare is part of it. What's kind of the tool suite look like? For the processes and programs? Yeah. Sure. So one of the challenges that we had over the last year and a half was we were not only scaling our external infrastructure to support massive client growth, but we were also scaling all of our internal infrastructure services. Meaning we went from somewhere in the neighborhood of 250, 275 employees, to about 1000 employees spread across 22 different countries, all working remote throughout the pandemic. So it was a challenge to scale up that infrastructure. But luckily, as the pandemic started, we were already pretty well positioned, you know, we weren't having to really make massive infrastructure changes. At that point, we were, you know, take your laptop, go home, and we'll ship you a monitor. Right. We'll see what at the time it was, and we'll see you in two weeks. Has changed quite a bit. And here we all are still still working from home. But we'll at least largely, I will be going into the office in a little bit. But the, the, I think the part for us that we really started to want to get serious on as we were using that as an opportunity was, okay, we're going to continue to hire at a pretty rapid pace. And I think right now we, I don't know off the top of my head, I think I have 17 open roles on my team. So if anyone's listening, and is wanting to work on hard data problems, hard cloud engineering problems, hard security problems, hard cryptography problems, you know, feel free to check us out. But there's probably, you know, somewhere in the neighborhood of 50 to 60 roles open across the entire company. So we continue to hire at a very rapid pace. And we still have a remote first work philosophy, such that people are still working in probably more than 22. But I think the last time I counted, it was 22 countries. And we needed a suite of tools that really allowed us to do that safely. So as you're aware, we decided after evaluating a number of different products that Cloudflare Teams was going to be a product that we were going to go all in on. And a combination of teams and how that works with the tunnel infrastructure, and how that works with the backend WAF and CDN, and how we're delivering content to our employees. All of that has worked incredibly well. Not only because the technology works well, but because it's snapped in very seamlessly to things like Okta, which we have built our entire SSO platform around. And then also YubiKey, which we have used to really lock down all of our internal services. So to me, it was kind of a no -brainer. We were already heavily invested in all of our Internet-facing infrastructure with Cloudflare. We did a fairly thorough assessment, and then really liked where we landed with Teams. And the fact that it just snapped into Okta and YubiKey work made a lot of sense for us. Some customers ask me, you know, hey, from all the customers you talked to, what's the single thing I could do to make my organization most secure or more secure with just one task? And every time I tell them, in our Zero Trust product, you can add to the audience, you can add a rule that requires people use a hard key as their auth mechanism, not just, you know, if they fall back to an SMS code or something like that, they still get blocked. And every time I tell a customer asking for this recommendation, that that's the one rule that will change, most change your security footprint. Everything else is fantastic, layered on, but that, like you mentioned, the YubiKey, the importance of that is how we run Cloudflare internally. Yeah, and I think companies that take security seriously and have security core competencies, whether they're a security product company, or they're a company that has a core competency around keeping client assets safe, like us, you have to, you have to invest in these things. You know, I do say, and I it's any organizations that may be listening, if you can't deploy YubiKey today, understand, like, that's not something that depending upon the legacy nature of your infrastructure, it's not thoroughly supported by all the by all apps that you may use. That's fine. But I would even like zoom out a little bit from what you said and say, the single biggest value add thing that an organization can deploy is two factor. And if you can only get to SMS, it's better than nothing. If you can only get to app based authenticator, TOTP, authenticator apps. Okay, that's better than better. And if you can then if you get to YubiKey, that's even better. And then if you can get to YubiKey layer on Zero Trust technologies, many of which we've built around our the team's product for us, from Cloudflare, if you can, if you can do that, then then it's even better. And then you layer in things like device trust, and all of these other things that kind of bolted all together. But you shouldn't be sacrificing something I actually was just talking yesterday with some folks about it was, you know, don't let perfection be the enemy of done. So get some stuff done. And then, you know, roll the version two of your multifactor strategy out, whenever it makes sense. And I'm sure, you know, in your role, and just as security teams in general, it's never done. But what, what's up next? Like, what are y'all focusing on as a team? Here? I guess right now, we're kind of approaching the end of the year, what what is the beginning of next year look like? Well, hopefully, the beginning of next year doesn't start with a DDoS attack like it did this year. But you know, I think we're I think we're well situated at this point. I think we're well taken care of. But for me, and I've been really lucky, and I've had a lot of support from Zach, our CEO, from our board, from Tony, our CFO, but a lot of support from them. And for me, I've been able to build out a team of direct reports to me that are really running a lot of the day to day program. And that's been really valuable for me. So when I think about what am I thinking about next year, I'm kind of I kind of take a slightly different view than somebody who might be running a SOC day to day, we might be running an IR team day to day. And from my viewpoint, it's better alignment with other teams. So better and more alignment with the business better and more alignment with engineering better and more alignment with product. I think we're great already, but we can always get better. And I think that's it's funny, a lot of people that work for me sometimes can get frustrated because I never give them like a 10 out of 10 performance review. You know, at best, it's a nine. And it's like, well, why? We can always get better. So that's something I've carried with me even from days as a young Marine, you know, doing fit reps on others. So that's one area. Another area is going to be mature. So we've spent a lot of time over the last year, getting infrastructure and tooling and things in place. You know, I've only been here a year and a half. You know, probably the first 90 days of my time here, we're trying to just figure out where everything was. And then what what was the scope of the problems that we were dealing with? And now we're a year and a half in and we've got a really good handle on things. We have a fairly sophisticated program. And now the focus will be how do we mature that up? How do we get better at all the things that we're good at? How do we get better from a capability from a technical capabilities perspective? Probably the third bucket would be probably tied for third, I would say, is getting better at I would say, like the soft security things. So we're already pretty good about with things like GRC, and pretty good about how we partner with, with regulators and auditors. And we do a lot of work on that front. But we can always we need to get a lot better on telling those stories. And you're getting a lot better around maturing our policies and procedures and the paperwork to catch up with the engineer. And then tied for that, I would say is professional development. And I would say we've hired a lot of people. And we've made some really good hiring decisions. And we're really going to be investing next year in our people training conferences, you know, SANS, Black Hat, all of the things that I think are really table stakes for a mature security program. We're going to be investing a lot more people. That's fantastic. Hopefully, a year from now, we're celebrating another CIO week. And we're talking about how, how successful those programs were. Is there any just kind of in the couple minutes we have left, if there's other CIOs or CSOs watching this program, any sort of people who want to go into that as part of their career? Any one bit of advice you'd share with them from your career? Yeah, so I would say there's probably two answers to that. The first one would be for folks that are maybe more junior in their career and wanting to head that direction. Don't lose focus on the technology. I think everyone can always kind of lose sight and want that next role and want the office, want the title, want the money, whatever it might be. But what I have found is the best technical leaders come from an engineering background, and are able to engage at a deep level with the engineers that work for them. And that is a way that not only builds respect across the team, but also it will help the leader as they progress through their career, to understand how to separate signal from noise, and being able to know when to engage and when to dive deep on particular problems that they might be hearing about. And then for folks that are maybe already in these roles, it's hard to be prescriptive. I never try to make any absolute statements. We're all kind of in this fight together. I think something that I would like to see, specifically around the crypto space, as we become more and more material part of the financial services sector at large, would be a better focus on things like data sharing and threat intelligence sharing, and leveraging things like FSISAC and others that the traditional financial space has kind of done and figured out pretty well. Very cool. Well, it's been a pleasure chatting with you today. And I know we love working with your team as a customer. So thanks for the trust y'all put in us and helping out your organization. And thank you as well for your time. Absolutely. Great to be here and look forward to doing this again sometime. Fantastic. All right, Adam. Have a good day. Thank you. All right. Bye bye.