Latest from Product and Engineering

Hi, I'm Jen Taylor. And I'm Usman Muzaffar, Head of Engineering at Cloudflare. Hi Jen. Hey Usman. Who's our special guest this week? Our special guest is Garrett Galow. Garrett, can you introduce yourself, please? Hello everyone. My name is Garrett. I'm a Director of Product here at Cloudflare. I've been with the company for about three and a half years now. I started actually a couple months before Jen did. Not long after I did, right between us actually. Yeah, I think it was like equidistant. We're about six months apart and I was right in between. I remember actually thinking it was when you'd been there for a lot longer than you had when I first met you. He's good at faking it. Apparently, yeah. But yeah, been here since. Been a great run so far. What part of the system are you responsible for, Garrett? Yeah, I wish I had a very brief way of describing it. But the easiest way is, I'm sure you all have talked about our edge and our core. And if not, you've seen that, talked about it in our blogs and stuff. Most of the teams that I work with work on core products. Everything from our UI and API platform, account and user services, trust and safety, and legal abuse processing, and then a lot of our pay-go growth. So a lot of core stuff, making sure that customers can really easily use our platform and scale well with it. Great. Garrett, what brought you to Cloudflare? Yeah, so it's funny because when I interviewed PM Kennan and stuff, they'll often ask me this kind of question. And so before I was at Cloudflare, I was at Microsoft working in Azure, and there were two things that I was looking for. One was the ability to just have bigger impact, bigger scope, do more. The other was actually a little bit more customer -facing. I really liked the work I did at Azure. It was really exciting, but it was a little detached from the everyday person using the platform. And so I took the risk. I don't think it was much of a risk, came to Cloudflare hoping to find that, and very quickly found both of those things, and still do to this day. Cool. That's great. So as we're getting ready for this conversation, and I was checking my email, and I got an alert that we're doing DDoS alerts. And I know that the alerts platform is something your team has been responsible for building out the foundations of. Can you talk to us a little bit about what is it we're trying to solve with alerts? Yeah. So this actually is probably a feature request probably longer than I've been at the company. And it totally makes sense. One of the things we do, we protect websites. One of those things is DDoS attacks. And that's great. We tell customers we'll block DDoS attacks for them, but they aren't sure. They buy our services, they use our services. How do they know? Are they even being attacked? If we're doing our job, they won't know, but they kind of know that we're doing our job. And so they're kind of like, still want to know just for knowledge purposes, or maybe they might need to do some extra things. And so it's been a longstanding ask, like, hey, if we get attacked, can you tell us? And there's other things customers want to know, like, if my origin is down, please tell me, these kinds of things. So we built the alert platform to let it give customers the power to configure alerts so that they can receive those kinds of notifications wherever they use them, whether it's email or other systems. And so we started at the beginning of this year, we supported email as a delivery mechanism, and we supported origin monitoring. So if your origin goes down, and we also support billing alerts. So if your bill is going to be really high for the month, we can tell you so you can maybe do something about it. And very excitingly, just in this last quarter, we started supporting pager duty as a mechanism. So now you can hook up these alerts to go to your pager duty. And so you're, you know, your team, your engineering team can be alerted following their rotations for escalation paths. And then, as you just mentioned, today, we launched L7 DDoS alerts. So now if you're getting L7 attacked, you can get notifications either by email or pager duty. And for the customers using our layer three and four products, we launched those last week, actually. This is really cool, because it's like you mentioned at the top of the hour how this is a edge and the core. The edge, of course, is where we actually make the Internet faster, safer, more reliable by, you know, being the proxy, being the thing that stands in between our customers, customers and the origins, their websites. And Garrett, you start off by saying that you were in charge of, you know, a lot of the teams you work with are in the core, which is basically telling the edge what to do. But this is alerting, which is sort of crosses that boundary, right? Because obviously, it's alerting on something that it noticed at the edge. How else would it do this? And yet, how would it know where to send that alert, unless someone had a priori configured that in the core? And so, you know, I think there's some interesting things going on here. And how is it that there's multiple kinds of alerts we want, and then multiple ways we want to alert our customers? So what is some of the interesting challenges we have to solve here to make it so that customers can sign up for alerts being delivered to them in a way that makes sense to them? But the average edge engineering team doesn't need to know anything about that, and can just report what they are noticing, what they're blocking, what they're making more available, you know, and that something, someone, someone like Garrett is somehow tying all this together. Yeah, it's funny you mention it, Greg, because a lot of time when we talk about the core and the edge, it's, you know, you, customers make changes, and we put those in the core and then push them to the edge, right? Yeah. But this is sort of like, it's like analytics, it's the inverse, right? Things are in the other direction, we want to bring that back into the core, analyze it. And so it actually, you know, analytics is, you know, the base layer of this, right? We need the data to be able to even, you know, make these decisions, right? But then we need another layer on top of that data to let us say, okay, I want to look for, you know, certain patterns or thresholds for a specific customer, and do that. Yeah. And you mentioned, you know, how do we make this scalable for teams, right? Like, you know, the DDoS team, they should just worry about blocking DDoS attacks, detecting DDoS attacks, and all of that stuff, right? They don't need to care about, you know, what emails do I need to alert to, you know, is it pager duty, is it email, is it one, is it 10 people, right? And so we build a system where, you know, a team like the DDoS team, if they can detect these alerts, and they basically fire an event into our system that basically says, hey, there was a DDoS attack, here's the customer that it's affecting. And then on our side, and the alert notification side, what we actually do is we listen for those events. Every time we see one of those events, we'll say, okay, I got this event, you know, let me check that customer, have they conferred any alerts for this, right? And we'll match that up and say, okay, so they have alerts for L7 DDoS attack, okay, now let's look up where do they go? Okay, we need to send an email to these three emails, and we need to alert this pager duty service, right? And so by disconnecting sort of the upstream, which is actually detecting that things are happening, and then the downstream, which is actually sending those events to customers, right, make that really easy. So, you know, it's sort of a plug and play mentality where, you know, service teams plug in on one side of the equation, and then our team in the application services team, we sort of build out the outputs, right? And so when we, you know, add a new destination, you know, the DDoS team won't have to do anything for that. That's awesome. All of a sudden, you can get, right, yeah, you can get L7 DDoS alerts in this new mechanism that we now support, right? And so it sort of scales both ways, right? That's super cool. Yeah, that's amazing. You know, one of the things I thought a lot about as we're talking about and building up a platform for alerts is, you know, another area that you have a lot of responsibility for, where we do a lot of detection and sort of mitigation is actually some of our own internal trust and safety tools. I think, you know, it's amazing if you step back and you think about it, you know, we service 27 million Internet properties across pretty much every country on the globe where you're allowed to service people. And, you know, we need to find a way to kind of keep a scalable kind of bird's eye view of what's going on, you know, across the network. What are some of the ways that we've approached that sort of observation and the living? Yeah, so this is actually, you know, it's really tricky, right? Because there's so much data going through Cloudflare. And so, you know, how do we possibly keep up with what's happening? And I think, you know, for a lot of cloud first history, we've been maybe a bit more reactive about how do we respond to these things, right? We have ways for people to, you know, tell us that abuse is happening on our network, whether it's copyright, or, you know, child sexual abuse material, things like that. And then we're able to respond to those kinds of notifications from customers or from other people that see this on the Internet. You know, we work with a lot of partners, NGOs, and other aspects, other organizations in order to make sure that, you know, we're following best practices, you know, we're communicating with them and aiding, you know, in investigations as necessary. But, you know, one of the things we wanted to do is start to take a little bit more proactive approach, right? You know, why do we have to wait for people to tell us when we can, you know, maybe detect some of this stuff ourself? So one of the things we've been working on is, can we build a tool that can help us proactively detect certain kinds of content, like child sexual abuse material, and actually proactively do things, you know, alert customers, block that content, and then alert the correct, you know, partners that we work with about these events. So it's something that, you know, we announced, I think it was late last year, something we were going to be building, you know, we sort of had a, people could sign up saying that they want access to this. We've been working tirelessly all year on this. It's a extremely hard problem. And, you know, in the last few months, we've been sort of, you know, taking the first steps of rolling out to a few customers to see how, you know, the service performs. Yeah, it's really interesting. The hard problem is the right, right, right, is exactly right. Remember, because oftentimes, for massively distributed company, companies and technology bases, like Cloudflare, the story is, this is, this is no big deal at the hello world level. This gets hard when you have to do it 20 million times a second, or at scale, or whatever. And here's an example of a problem that's hard at all levels. It's hard even to get it right, even the hello world is hard to actually identify something correctly, watch for the false positives, make sure that you're gonna send it to the right pipelines, make sure that it integrates all people. And then to do that at scale, is yet another whole level of magnitude. So it's sort of like the Zen squared problem of how tricky this is to do right. But it's so important. And so, so, you know, so proud that we're working on that. Yeah, the interesting thing about that, right is, is, you know, one of the things we learned is that we can't really solve one problem at a time here. Because, you know, we don't like this is the kind of content you don't want to have, right? It's actually illegal to have this content, right. And so we're trying to, you know, validate things like false positive rates, you know, using as we learn as we scale the service, and we're also trying to learn does the service scale with, you know, the code we've written, we've had some really, really smart people at clubs are working on really efficient algorithms to sort of process and match this data, you can imagine, how do you verify that an image is like another image? How do you do that efficiently? How do you that correct? How do you sort of prevent minor variations that, you know, it's still the same image at heart, but it might not, you know, be a string mash. Yeah, and so, you know, we're having to sort of solve these problems simultaneously as we go, which, you know, it poses a ton of challenges. But given the given how important this work is, you know, it makes it it's really inspiring. This is really at the heart of, you know, building a better Internet. Yeah, absolutely. Actually, on one of the one of the the next thing I wanted to ask you about, I'm going to unless Jen was going to ask you, but I'm good. My dog, my dog might ask you a new a new visitor player for has entered the game, folks. But the, you know, you mentioned talking, working with partners, and, you know, so much of what the control plane has to do is connect with other partners. And coffee's got a great history of working with partners. There's, you know, one of the great stories when I first joined Cloudflare was how many people were Cloudflare customers, didn't even really realize they were Cloudflare customers, because they were very non technical users signed up in a very easy to use sort of hosting provider, there was a checkbox there, which was something like, let Cloudflare help protect and accelerate your site, you press that button, and the hosting provider itself, provision the cloud for account, signed it up, and, you know, did did, you know, sort of orange plotted those zones and help people get the benefits of the Cloudflare platform without ever having to lift a finger. But that was a long time ago. And now our partnerships are far more sophisticated. Talk a little bit about what the partner platform is now and what our relationships with things like some of the things we've announced on our blog, the partnership with JV Cloud, the partnership with IBM's Bluemix, like what is what does that mean now? What does that look like? Yeah, I mean, this is very near and dear. When I actually started at Cloudflare, I was RPM for partnerships. That was my first role. That was expanded. Yeah, it's expanded a little bit since then. Yeah. And yeah, you know, we for I don't even know how many years worked with lots of hosting partners, making sure that you know, they can protect their customers. And it was actually when we started talking with IBM about approaching a partnership with them that we sort of looked at, okay, how can we really take this to the next level? How can we really, you know, that platform at the time was probably four or so years old. And we really were looking at how do we modernize it to work for the future. And so we sort of built from the ground up a new way for partners to have a provision with us. We standard everything, standardize everything on the same kind of API as we give our customers. We modernize actually, the a lot of the billing technology we have today around subscription management actually came out of that project. The first customer that used subscriptions, just the way we track, you know, what a customer has purchased, that IBM was the first customer to use that. And that eventually became the de facto way that all, you know, purchases and, you know, usage of products, it's now tracked in Cloudflare. And so we built that because we want it to be a lot more scalable. And we did that with IBM, that was a couple years ago now. It was a whirlwind project, I think we shipped the first version of that platform in about three months. And we just expanded it since then. And so, you know, we have more and more partners that are able to build on top of that, provide a bigger, you know, you know, more of our services to their customers and manage them in a much easier way. And then that's sort of led to sort of the partnership we're have announced now with JD Cloud, which is sort of the almost the next level of that, where it's not only sort of using those APIs and the services we built for provisioning of customers and managing of them, but actually, you know, deploying another network inside of China, that customers will have access to. And so both customers in China who, you know, are customers of JD Cloud, right, we'll get those benefits of Cloudflare in China. And then our customers in, you know, what we call the rest of the world, our global network, right, we'll be able to get the benefits of Cloudflare inside China for their customers. I think it's really interesting that our customers now have the word cloud in their title, that they are cloud providers, that then are signing up SaaS companies that are service providers for other customers who then have eyeballs. Like, I mean, like, you look at the graph of people who are leveraging and benefiting from Cloudflare technology, it's just incredible. One of the things I loved about the IBM story was now you, even things that seem relatively simple, like, oh, we need to get analytics. Of course, you need to get analytics. Yeah, but we don't need to get analytics for one zone, or even one account. We need to get analytics for hundreds of accounts continuously. And so, like, this thing that was working just fine, all of a sudden is now being looked at through a very different lens. And it's all the right information, but it's got to work at scale. And so we re-engineer, we come up with new endpoints, new batch ways of processing, you know, analytics and logs, and sending them to our partners so they can then expose them, you know, effectively in their own control panels and their own control points. And it's a really great partnership with us. It just makes everything mature and, you know, grow up so much faster. Yeah, it's really funny because, you know, they, you know, we worked on a lot of things and they pushed us to a lot of places that, you know, there were other enterprise customers we had that actually were working very similar. They just sort of were using Cloudflare as, you know, the, I'm one, a much smaller enterprise customer. And so we're actually working with some of those customers who we don't necessarily always think of as partners in the same light. We're actually working with them and seeing how they can move onto this platform because it's much more scalable. It's much easier for both Cloudflare and the partner that that customer of ours to manage. And so we sort of, you know, there were things we had done in the past sort of accommodate use cases that we weren't really built for. We didn't have technology to solve. It's like the wrong word almost, right? It's just, it's, it's, it's, you know, next, next level of the service. Yeah. Yeah. And, and, and, you know, I, I work, you know, I've worked really closely with the folks at IBM cloud over the past two years and it's been, you know, quite excellent. They've been, you know, super patient, super understanding, super helpful, really helping us. Yeah. Yeah. In the right way, right. Demanding in the right way. Yeah. Right. And really, really helped us up level, not only, you know, the product for, for them and other customers like them, but just the product overall. Yeah. Yeah. I think that like that, that's the world power of a great partnership. Like when we have with IBM, the one we're starting to have with JD is that like, you know, it pushes our platform kind of step functions above where it is now. And it's just been so fascinating to work with you, Garrett, over the course of the last three plus years. And just to, to be a part of the growth that you've seen and you've created and you've driven in our platform, things that were sort of like, well, we should try this with, you know, one partner, you know, that we then build on and internalize and make available at scale. It's just, it's great to watch the way in which you've really, you know, taken some of these underlying services and really fundamentally made them platforms that other teams at Cloudflare can build on as well as, as third parties. Yeah. Yeah. I appreciate that. I actually think I've had a lot of thoughts recently about, you know, the idea of like how services platform inside a company and really support other teams in a certain way. And how do you sort of, you know, remove or, you know, make simpler the interfaces between parts of, you know, teams and stuff, because that helps things move a lot faster, more smoothly. That's true outside, obviously, because you don't have the same coordination, but even inside a company, it's super important. One of the lines we like to throw around at Cloudflare is one plus one equals three, right? So like, you build a messaging platform, you build a notification platform, and all of a sudden, 50 products have the ability to notify on PagerDuty when they didn't even know PagerDuty was one of the integrations that you were working on. It's, that's fantastic. Yeah. You know, as we were starting to think about, you know, this conversation with you, I said it was one of the things that, like, you were responsible for that, like, I absolutely love and I love nerding out on is actually our off platform, because it is kind of so critical and so integral and so powerful, and yet also at some level, kind of so nuanced. If I could just note to engineering leaders who are interviewing heads of PM, if the candidate says, I really am, I like nerding out on off, you found the right person. You know, Gary, you know, I know that in particular, as our platforms and our services have matured, one of the areas where we've made a lot of investment over the course of the last three years has been off. Can you talk a little bit about, like, that journey and what are some of the use cases and things that are really driving that innovation? Yeah. So, you know, I think, you know, a good way of painting the picture is Cloudflare, you know, in its earlier days, right, was like a free service, and we've slowly moved up market, right? We started having paid plans, and we started having sort of enterprise contract level plans, right? And we moved up and up the market. And, you know, one of the things that as you move, you know, up market more is things like off, right? How do you control access to things, resources and stuff in a system? And so, you know, I appreciate all the, you know, the words of the work we've done, we've done a lot of good work, but it, you know, it's the kind of thing that always feels like we're trying to keep up because, you know, as we go up market, we get more and more demands. And they're absolutely the right demands, right? They're the things that, you know, customers need to, to be able to secure and feel good about how access, you know, they can control access to the product. When I was pretty early when I was here that we, you know, first launched shared account access, it was the first time we let, we sort of had a, I feel like I can say it now, extremely hacky way of letting our, you know, contract customers share account access. And we sort of rebuilt that from the ground up, and then also gave our, you know, pay as you go customers, some of that, some of that control as well. And since then, it's sort of just being, you know, refining that process. So last year, we launched, I think it was last year, two years ago, I can't keep up with time, we launched API tokens, which, you know, really foundationally changed how you can create API access to Cloudflare, and really give you that binding control of, you know, I only want this API token to be able to update, you know, DNS records on one zone, right? It's just my staging zone, not my prod zone, right, which is super, super critical when you're running these automated systems, because you don't want to risk, you know, something happens, and a server gets, you know, compromised, you don't want to risk someone having full access to your account. So I was super informed, we built that platform, you know, we're, I don't know if I can say this, but I'm going to go with it, we're currently working on extending that kind of control into kind of user access, right? So making it easier for, you know, granting specific access inside of an account to people, and doing it that way. And then, you know, last year, we launched SSO, which was pretty, you know, we had to meet some customer demands that were pretty, pretty aggressive on timeline, we really pulled it together to get something out to market for our customers. And, you know, there's absolutely stuff we need to improve there. But I still feel really good, because if you use SSO, and you go to Cloudflare dashboard, as soon as you type in that email address, we recognize that you're using SSO, you don't have to do anything else. Every time I go to any other software, I have to go, I have to go to another link, or to go to a specific URL, it just kills me, because it's like, this is this horrible user experience. I remember that was the thing I cared the most about, I was like, I don't care how we solve this problem, we have to solve this problem. Because if we make customers do something weird or special or different, it's going to be a failure, especially on login, right? That's the last place you want to be thinking that something different or weird. But I want to go back to something you just said a second ago, because it really resonates, especially just as as anybody who's responsible for large operations, you know, mistakes happen. And you can talk about, well, we should have fixed that bug, or that script shouldn't have done this, or why wasn't it? But the better question is, what would have guarded against this from happening in the first place? Like, how was it? You know, how do we how do we build defenses where even if the mistake happens, the blast radius of the problem is so limited. And that always comes down to control and auth, right? Because you want to make sure like, look, I was just trying to write a script that was supposed to, you know, scrape audit logs. It wasn't supposed to be able to delete my DNS records. So why isn't that script only allowed to do read only operations. And that way it gives it's everything gets better. It makes your security team feel more comfortable. It means the developers can have that confidence, like, I'm not going to cause damage here, because I am not even I don't even have a key that lets let's do that. And as we get as we talk to bigger and bigger customers, you know, they're not even in the same continent, we're talking about different divisions of multinational corporations, they don't know each other, this is not a small group of people who all know each other. As far as they're concerned, that's like a different company that owns a different part of that subdomain. So it's got to be separate. And yet it has to be one account. Otherwise, you know, we lost the single pane of glass. So it's such an important problem to solve. And it's one that becomes more important as you go up market. And I'm just really proud of how you do it. Because if you do it wrong, you wind up with a performance problem. That's the other classic way that you can mess up the engineering on this is the code becomes littered with a gazillion of statements, because you're not allowed to can't do anything without checking some giant rules database for us. And then everything goes slow to a crawl. I had that in the last previous thing. Yeah, the, you know, the other thing I remember trying to build this, and, you know, from my experience and other people's experience using sort of these kinds of off control systems and other other services, right? Like, the other problem you run up against is complexity that, you know, customer can't understand what they're supposed to do, right? Because like, you can have all the control in the world. But if someone can't figure out how to say, go, I only want to read audit logs, right? And if they can't figure out how to get something that actually only has that scope, you've also failed, right? Because they'll make mistakes, they'll have extra access, or they'll say, but like, NTFS files, permission still confused the heck out of like, you know, like, it is possible to get create so much rope and so much flexibility that nobody understands how the rules work anymore. Yeah. And I remember we spent, you know, this was we often do, you know, design sprints when we're kind of in the early stages of a product to understand, you know, we kind of know the requirements about what sort of things we need, but we might not still be sure about, you know, how it's all going to come together, what it's going to look like to a customer. This is one of those projects we spent a few designs on. And, you know, it was, I remember like halfway through the week being really worried, we weren't going to come out with a design, like we just kept hitting our heads against the wall. And like, how do we show this, we knew what we could do, right? We knew what the system was capable of at that point. How do we show it to a customer so that they can, they can get it right. And, you know, some people may have feedback that there's still improvements to be made. And I'm sure that there are, but you know, I need to think we've reached the end of innovation. Yeah, but I remember the point when someone on the team suggested, Hey, have we thought about doing this and drew something on the board. And as soon as I was like, that's it. That's what that this is how this is going to look because that makes sense. Well, that's the thing I really love about the way that we work that we have that kind of very highly collaborative process and like the great ideas really come, come from anywhere. Kind of the one more super nerdy off thing that you guys have done in the last year that I have been really excited about is the, the workflow and the simplicity that you've brought into 2FA, right? I mean, a big part of what we need to do with auth is, is build great systems and have great power. But we also need to make them easy enough to use an easy enough to recover from in the event that something goes awry. You know, why was actually enabling our customers to do 2FA themselves? Why, like, why was that hard? Yeah, 2FA is, you know, kind of the center point of a bunch of disparate problems. There's obviously like security, right? How do you keep a customer's account safe? But there's also like simplicity, right? Like you can make it really a customer's really sick or their account really safe, but at some point it's too hard for them to get into their own account, right? Or something goes wrong, they can't recover. And so, you know, we, we supported, we had supported basic, you know, 2FA, TOTP for, for a long time. But, you know, there's some glaring issues of, you know, if you lost access to that, right? You were, you were basically posed, you had no real option. And then, you know, 2FA had evolved, there are things like hardware keys, which, you know, present a much, much better user experience that we need to do. So, yeah, thanks to, you know, a lot of hard work of folks on the team, we were able to sort of, you know, kind of solve multiple problems together by bringing, you know, recovery code so you could actually get, and, you know, we thought a lot about how do we make this, you know, the right balance of security, right? Because if, if they're too easy to get to, right, you risk, you know, that being the way someone hacks into your account, right? But if they're too hard or too unuseful, then no one's ever going to have them when you, you know, making it easier to have multiple, multiple types of 2FA. So that way, you know, you can have a backup, and if something goes wrong, you have another one. And then hardware keys, being, you know, sort of the best, the best of the best currently for, for access, and also being the most usable. I use it on my personal cloud player account, and it's so much easier just hitting the side of my laptop than trying to like open up my phone, go find a Google Authenticator app to like put in a code. So yeah, it's been a, it's been a long journey. It's been, it's been really fun. And, you know, it's, it's also the balance of like, you know, do you force customers with 2FA? Do you have an option? Do certain customers need to use it because their accounts are more valuable? Really, really tough questions that, you know, don't have one shot answers, right? A lot of it, when we talked to, you know, we talked to people that worked at other companies. And, you know, a lot of it was, you know, well, we made these decisions for these reasons, because this is the price of, you know, someone's account being taken over, right? You can imagine that, like someone at LastPass, right? LastPass being, you know, that's the most important thing, right? It has the keys to the kingdom, right? Maybe one of the most important things to protect versus there are certain other services that, you know, it may not be as big of a deal. Great. As, as always, getting, getting one of our senior product leaders on the, on the show means that we, we hit the 30 minute mark within minutes. I was just like, I can't, I'm just, we're just getting started. I can talk to Garrett about like 80 other projects over the last three years. Totally. Thanks so much for joining us and talking about alerts and, and CSAM and all the partners and all this great stuff. And I can't wait to have you on here again soon to talk about all the cool new stuff you're working on. There's not a lot to talk about yet. Yeah, I appreciate it. Yeah. A lot of things in the hopper that should be coming out in the next, well, some stuff in the next week and some stuff in the next few months. So really looking forward to, to all of that. I appreciate y'all having me on. Well, great. Thanks for joining us.