Co-engineering the future: How Uber & Cloudflare engineer a global platform

Welcome, I'm Steve Goldsmith. I'm joined here by Mattie Toia from Uber. I was just about to say the wrong company. Yeah, don't say the wrong company. From Uber, and we're starting a new series here for Cloudflare. We're trying to talk basically about with technical leaders who are customers of ours, how do they use the platform and get into some real detail, so we can understand collectively how different global brands like Uber are stretching, how Cloudflare runs, but also how you might learn from how they're using these platforms and otherwise. Do you want to do a quick introduction, Mattie? Sure, yeah. So I'm Mattie Toia. I'm the Vice President of Infrastructure at Uber. I've been there about five months now, so relatively new. So maybe I would forgive you if you got the wrong company. I know it's a change for me. But I've been a long time Cloudflare customer and really excited to talk about the partnership, particularly for large enterprises and for infrastructure scale. Well, let's start with that. So five, six months in, what's your first impressions of how this platform relies on Cloudflare or doesn't, and sort of interesting historical choices that have been made around how this is working? Yeah, it's been really interesting to learn about a new business and the new way that the infrastructure kind of comes together. Obviously, Uber is a very high-scale business. We are mobile first, so a lot of the traffic is coming from mobile devices. We have multiple apps, so a lot of the traffic is coming from those apps themselves. And, you know, coming from a background with a lot of web experience, this was sort of a new thing for me, and I got the chance to learn a lot more about the nuances of dealing with traffic coming from devices. And also, it's a lot of real-time data as well. So it's not just like you render a web page and you're done. A lot of the data is constantly coming in, telemetry, information about a trip, information about earners on the platform, and so on. And as the business has grown, we've seen the kind of diversity of the requests that Uber serves has also grown. And having Cloudflare be a significant part of what our front door looks like has been a development that's really helped us reach the scale we had, and I think also deal with the modern state of the Internet today. So if you start thinking about, like, tracing a request through the system, it's your first day, and you're like, I open my phone, I've used Uber before, right? But as you start to unpack that, how do you, can you walk us through a little bit about, like, learning how the architecture of these systems work and sort of, like, the connect-the-dots version of how things are flowing? Sure, yeah. So obviously, with all good infrastructure, there's many layers, right? You always give, anything can be solved with one more layer, is usually what people say. And it really does start with the device itself, the SDK that's in the application. We do a lot of work on the client side to try to make sure that the requests are low latency, that they are using, you know, the right HTTP, the right set of parameters to make those connections to our front door. And then once the request comes from there, it hits Cloudflare. Then at Cloudflare's layer, we have our DDoS protection, the dynamic WAF all set up, which is a big part of how we make sure that we secure our network and deal with, you know, the threats of, you know, rogue scrapers or large-scale traffic hitting one of our endpoints on one of our properties. And then also provides us with a good amount of telemetry. One of my actual secret favorite features, I think, of Cloudflare is log push. I think there's a ton of great data that Cloudflare provides to you about the traffic coming over the network. And in addition to sort of the way that the tools work online, you can actually get that data into your systems, which is a big part of what kind of comes in the second layer at Uber. After Cloudflare, it comes to one of our data centers. We have a mixture of on-prem and cloud locations where we run our workloads, and there is a Envoy proxy that sits at that layer. This unpacks and takes a look at what the message is to try to figure out where the right backend inside of Uber is to go to, and then also sends off a bunch of the fingerprinting for that particular message so we can know a little bit about where it's coming from, who the client is, the versions of the client connecting, and then also the data that we get from Cloudflare about things like the job for fingerprint and so on, which is gonna become important later down in the stack as well. And so that gets forked off to Kafka where it goes into a locking pipeline, and then the request itself gets forward on to our service mesh. So we run our stateless workloads inside of Kubernetes. We have an internally developed service mesh that then routes those requests through down to the individual's backends that provide the actual support for the particular request that's happening. And then that actually does some rate limiting. So we have some internal rate limiting at that point. So if we do see traffic spikes or behavior that's undesirable or a system under load, we can either route around it or push back as needed. And then finally, it gets to the service, whether it's the thing that's providing you with an estimate for your fare on Uber or potentially information about a merchant that you're looking at where you wanna order some food for dinner, whatever it is, it gets to that ultimate service at the end. And then we provide back the response upstream from there. And in that particular background, there is also a couple other safety measures we have layered in where it can do on the service side, some pushback or connection termination if there is an overload situation, and then the error propagates and gets handled on the way back up. That's an excellent description. So if you think about which dimension matters most operationally in terms of scaling the system, it's the number of clients, it's the number of requests coming through. It's like, which one do you worry the most about? Yeah, we think about this a lot. And one of the interesting things that I've come to learn about Uber's business is that we just have a lot of different products that kind of scale differently. We have our core rides side of our business, which really we're thinking a lot about, okay, how many trips are ongoing at a particular time? Where are the errors on the platform? What's the data that's coming in? If you've used the app recently, you might note that sometimes there's an option to record your trip. So it could be like video or audio data that's coming in, support requests, et cetera. And so that's like a lot of data exhaust that's happening during the trip in real time. And there's also sort of this other side of the business, which is our delivery side, where there's a lot more browsing and selection and learning about what products are offered to you before you even start your purchase. You're constructing a basket and then ordering a delivery from that way. Or even if it's not food, it could be a retail thing as well, where you're getting a local delivery from a local business. Those have fundamentally different dimensions, right? We think a lot about the concurrent trips, but we also think about visits, which is much more traditionally, I think where Cloudflare comes in and as the business has shifted since COVID, we've seen that sort of overlap in the way you think about Internet traffic really converge a lot more with the technologies that Cloudflare has been providing. I mean, that's really interesting to think about this collection of different use cases and how the answer varies on different. When you think about, I guess the next year or two of your roadmap, are you consolidating those into certain kind of patterns or is that diversity likely to remain? So there's always gonna be diversity and probably even more diversity over time. I think one of the kind of powers that Uber provides is just a lot of different services to bring these products that are accessible to people like on their phone and accessible to people who are earning on our platform and then connecting them in this marketplace. And there's always gonna be an evolution of that. So I expect that that's gonna diversify. I think what continues to get refined and continues to get more interesting is, I just talked about all that telemetry data. We can use all that data to create the forecasting models to really drive the rhythm of our business. So we know what to expect come say Halloween or something like that, where there's gonna be a lot of activity on our platform or how many folks here watched the World Cup this year? World Cup's a big thing for delivery around the world and an interesting impact on supply when everyone's watching the exciting game. It's like suddenly no one's driving cars anymore. So understanding the dynamics of those events along with the historical trends in our data from the traffic coming in, really allows us to prepare from the edge in, in terms of how we provision, how we ensure that our capacity is safe and then how do we make sure that our systems are efficient and performant in order to deliver the best results. Interesting. So Cloudflare is a source of continual innovation. You'll see it around here for the next few days. One of the piece of feedback I get from lots of customers is you guys give us too many things and it's hard to figure out where to use them, use safely or new whatever. So do you have a point of view on how you guide your teams to try our new products or just generally products from other vendors and which part of the stack and where you're willing to take more or less risk? Yeah. Well, I think that it's important to always have some budget and appetite for risk. I think if you don't take, especially in infrastructure, I think traditionally, if you're working in infrastructure, a lot of people look at you as like the people who say no. And I say, we have to be the people who sometimes say no. We can't always say no. But there's always an opportunity to innovate on how you deliver your services and how you incorporate new technology into your stack. Like we've done a lot of work on DDoS with Cloudflare and we still run a core set of DDoS protections internal to our systems for defense in depth. But we also recognize that there's a lot of efficiency we gain by being able to do DDoS protection at the Cloudflare level. And additionally, there's a lot of additional intelligence and insight that Cloudflare can layer on in terms of detection for some of the very large scale botnets that might be coming in. And so it took a program to sort of work our way up to that. We also have the ability to sort of split workloads, right? Where we can take a particular part of the market or a particular percentage of traffic and experiment with that independently of the rest. And that's a really important way for us to gain insights into how new products are being developed and also how new technology from an infrastructure perspective can evolve. You know- I think you built that with us, didn't you? Yeah, we did. Actually, I think there's a talk later, there's a plug that some of my folks, I think are gonna do a talk about config deployment, things like that. But yeah, having the ability to do a change where you don't impact the global world is just such an important fundamental construct when it comes to reliability and safety when you're dealing with high-scale infrastructure. And having that built into the platform itself really has enabled us to do more and move faster with the kind of work that we're doing. So I have two kind of related follow-ups. Everything breaks all the time, right? Yeah, always. This is the reality. Yeah, breakage is real. I think there's multiple incidents happening on my phone right now. Yeah. It wouldn't be a conference without- Hopefully not serious ones. Not serious ones, but there's been a couple where- We're still here, I guess. We're still here. If it really starts breaking, you'll know. So just graceful degradation and how you think about failure modes of the system and at what point does fail, is Uber really broken versus degraded? And how do you manage that through the whole pipeline? Yeah, so obviously it does start at the edge, right? No one could reach the website, then we're super broken. And so we do have some contingency plans for if there's a major outage. And often, a major outage might not be that Cloudflare's having a problem. Sometimes it's like, I think I was in New York a couple of weeks ago and T-Mobile just had this massive outage and I was standing outside with my phone. I'm like, well, okay, this is a brick. So how does that happen and what can we do about it? There's other paths to get to people. If it's over Wi-Fi, it's over a different provider, things like that. And sometimes we can use those to work around a network problem, but getting them at their front door. Once you're inside of our network, we do run redundant systems. We run active, active setups for all of our critical workloads. And then on top of that, we have a tiering system where we think about different workloads, like which are ones that absolutely must run, which are ones that we can break for a period of time. And so if there is a failure, what we'll typically do is we'll bring all the services online in one of the other active regions and locations, and then take down some of the lower tier capacity to make room for it, and then slowly fill back in and replace those lower tier services with new capacity as we can begin to scale behind it. And this is kind of a good balance, I think, of like having a good resiliency plan, when you can drill, we drill this regularly. Usually once a week or so, we have a small drill associated with this, but also control costs, which I think is on everyone's mind, right? You can't, if you're just running tons of infrastructure all the time at max capacity, you're gonna have a hard time with your business, I think. So then the inverse of that is performance testing and scale. It's not degraded, you just wanna make it faster. And for complex systems like this, that's one of the hardest things to do is figure out where it's slow and then how to make it faster. How do y'all think about solving that problem, Saeed? Yeah, we've been doing a lot of work actually on like squeezing out every ounce of latency, reducing the spinner time. Like if you've ever tried to get your car and then you see a spinner while you're waiting to get home on like a cold day, the frustration grows very rapidly. So reducing that time is a very big, important part of how we try to think about the quality of the app, the quality of the experience for the people who are using it, the earners on the platform and the merchants in the marketplace. And it's an end-to-end problem, right? Like it's as much about what's in the client, as much about what's the network transit to the edge and then getting through Cloudflare, through the smart routing to our servers, like every leg of that can have room for variability. And then we're dealing with millions and millions, I think something like tens of millions, like 40 million or something like that trips per day or being heavy on the platform or something like that. And these things, you will see the tail, like you will have those tail experiences. We're a global company operating in parts of the world that don't have as great Internet as San Francisco. And so those things do have an impact on that experience, but if you can't see those details, if you can't section the data, then it's difficult for you to begin to optimize and to talk about like what can be done, like what can we do to reduce the tail latency to get to the front door, right? Like we know very well how fast the backends are, so we know that we can actually deliver to the network a really fast response. But then how do we get that back to the end user who's on their phone someplace? Because that's ultimately what drives that experience of the spinner. Or there's some example projects you could share that you've been working on to address those kind of upstream legs. Oh, let's see what we can talk about. We've been doing a bunch of work on the specifics of the RPCs I've been using in the SDK. I don't wanna talk too much about it mostly because like I don't have all the details on my fingertips, so if my team is here, they'll call me on it if I say. But really, like you're trying to use like all the new quick parameters, like really get the traffic off the phone and to the backend with as limited handshakes as possible. And then also taking a look at the behavior inside of the client to make sure we're using connections and caching things locally as much as possible. We've also done just a pretty big migration just in the past few weeks to get all of our static content to be available on Cloudflare's CDN, which is also a huge help because there's a lot that can be done with like cache shielding and like the way that you can deliver the content even if it's not just yet available locally for that particular user, we can get it from a nearby region and then push it out that way. So it's slightly increasing the cache hit rate of our static assets. Like those things each add up and when we're talking about a P99 .9 latency, it really matters to do those types of things. That's fascinating. So I'm gonna tee it up for a little bit of like Cloudflare specific stuff. Of course, yeah. That's what we're here for, right? How do you decide when we are the right partner to solve these problems and when we're not? And then I have a bunch of follow-up questions. Yeah, I think a lot of times it's not a zero sum question, right? I think that a lot of times we look at the capabilities that Cloudflare has and it's a lot of the discussion is around like how can we incorporate the best without giving up what we already like? And I think that there's a lot of times where you can engineer yourself into a place where you actually can have your cake and eat it too with some of these things. So I talked a little bit about the way that our DDoS protection works with WAF, right? We knew that we needed to help with this sort of volumetric attack type problem because we were seeing, I think there was a, if you saw the keynote, there was this great slide of like traffic go like this, revenue go like this on the Internet, right? And I mean, like if you're here, you probably are aware of this trend, right? And you start to see this sort of rise of non-human traffic on your platform, you need to have solutions for it and Cloudflare has some really great solutions for it. But we also know that we have like a core set of identity associated with your Uber account, a lot of telemetry associated with your behavior in an app, et cetera, which is not necessarily present on the Cloudflare platform. But we've been using that for years to actually protect our platform and had a number of services that actually depend on that data to build it, right? And so that's the layered approach where we can say, okay, we build our WAF that uses all the tools that Cloudflare provides, we keep our system that maintains our understanding of the nuance of traffic with our internal data, and then we can stitch that data together in our pipeline to the feedback loop to both systems, a configuration that represents our intent in terms of blocking traffic that we believe is illegitimate or useless on the platform. And so that layered approach kind of is this sort of Goldilocks zone of being able to do both at the same time. Similarly, when we talk about the way we think about some of the more compute and storage products that Cloudflare has, like we have our own data centers, we have a number of Cloud footprints, so we have a lot of resources from that perspective. But there are still like use cases where like R2 is a good solution for certain types of data to store where we're wanting to either pull it from Cloudflare or make it accessible on the edge and the web. And so having options, I think, is a really powerful thing. And so when we talk about building infrastructure for the future, having the ability to maintain a level set of the complexity of your infrastructure while still being able to try new things is kind of the name of the game. And that's where sort of this internal platform really comes into play and where I think that you can drive a lot of value from your engineering team focusing on that commonality and what's the substrate you use to manage all of this. It's actually a good segue to my next question. If you think about a partner like us, is it you use Cloudflare or you use an abstraction that's powered by Cloudflare, like paved roads or like the actual native solution? That's an interesting question. Like I would, I mean, I think it would be, it would be weird to say we don't use Cloudflare. I think that like we definitely use Cloudflare, but one of the reasons why working with Cloudflare has been great is because you do provide a lot of APIs. You provide a lot of configuration primitives where you don't need to click ops your way to success. You can if you want to, but like ultimately when you're a large enterprise running large infrastructure, that's not necessarily the approach that you wanna take to driving your business. And so having the ability to build the commonality in that like API layer, and so you're like, okay, yeah, there's three different APIs. Some are internal, some are provided by vendors, but I have one system that I actually use to push out into it, that makes a big difference. And so that to me is sort of the way that I see us moving forward. And it's been kind of interesting with the way that agents have been developed over the past couple of years now that we're seeing those become much more richer interfaces because we wanna make them accessible for agents. But like, it's just generally good and useful for any type of automation that you might have, whether it's traditional, whether it's an orchestration layer, whether it's something where there's an agenda experience associated with it. In any of those cases, ultimately you need those hooks and having those hooks allows you to build what you need for your business. What is the biggest product gap that we have that we have yet to fill? Oh, I'm gonna complain about my least favorite thing. It's gonna sound petty, but I'm just gonna bring it up because it just drives me crazy. So a lot of us are cloud customers. We have big cloud footprints, things like that. And as part of the sort of evolution of what it means to run infrastructure on the Internet, we've all been developing this idea of like FinOps and like how we manage the costs when you're dealing with cloud services and SaaS and things like that. One of the hardest things I have to deal with is the billing with Cloudflare. The billing data and managing it is just a seriously complicated thing. As I mentioned, we do a lot of complex forecasting. We do a lot of forward -looking planning. And so being able to really understand that aspect of our business in the same way we think about our cloud spend, et cetera, is a powerful thing. And I know there's a lot of people working on it, so I don't wanna pretend like that's not happening, but it is a place where I'd love to see more just because whenever you're doing an investment with a partner like this, cost is always a part of it, right? The commercial side of it has always gotta be part of the equation because ultimately you have to make your business make sense. We've seen tremendous value, which is why I'm here on the stage. It's easy to talk about things that are showing a lot of value, but we also wanna measure and make sure that we're planning properly for the way the system is running and the way that the traffic is evolving over time. Right. That's very fair feedback and something we've heard before. Yes. A couple people in this room are in a pile of a new billing platform that hopefully you're aware of. Good, okay. Well, maybe I should try to get myself lined up with it. If you know somebody, he'll cook you up. So I'm gonna switch gears a little bit in this. This is sort of like based on the products you offer the customers. Sure. Switching gears, talking about managing engineers and managing engineering teams and building things these days. Yeah. What's the most interesting thing you've seen in your first five months of maybe how Uber's doing something differently about building products than they would have built before? Yeah, I think one of the things I really appreciate is that despite having both a diverse set of infrastructure platforms as well as a wide variety of microservices, like Uber's kind of a little bit famous for having a lot of microservices, the team has done a really amazing job at figuring out ways to manage that diversity of workloads and platforms. And I think that that has actually been kind of the key on lock as to why this is sustainable. How do you actually run across so many different clouds? How do you actually orchestrate so many different teams across the company who are building their own services and kind of have like full ownership of them in a way that then still allows them to do platform things like, we're changing your front door to the Internet, we're changing your service match behind the scenes. Those types of changes are pretty difficult if you don't have the right layers and hooks. And at Uber, we really think about that as like a first principle part of the problem. And bringing that investment in those foundations has been a big part of the success that's got Uber here. And I think it's a big part about what we're gonna need going forward as we think about the realities of what it means to have capacity in 2026, looking ahead to 2027, what it means to continue to integrate with the new and innovative companies that are out there. So I think that like those platforms are a really big part about what makes infrastructure great. Some of the things that we're working on around percent rollouts and safe changes and stuff like that, we've partnered with you on, but you wanna talk a little bit more about how you're thinking about using a partner like us to do percent, like I wanna change it just for this one region, this one slice of traffic. Yeah, actually, just to sort of start with, I think I'd like to say that one of the things that we've really appreciated is the willingness to like work and partner together to do these types of things. I think when you're making a big investment with a provider, then you really wanna have someone that not just is available for help, but like actually wants to do that sort of co-engineering type things for these like specific types of programs. And then also I think the sort of values around scale, latency and performance, like we feel there's a shared set of values there when we talk to Cloudflare in terms of how we think of our own infrastructure. So it's been really great to have those kinds of things. But about config rollout, I think that it started out as a conversation around safety. Like really we were looking at all of our systems in terms of like, what is the global blast radius, right? Because I spent many years as an SRE and if something was a global rollout, it wasn't like, when is it gonna blow up? Not if it's gonna blow up. It's like eventually someone's gonna take down everything because it's there, right? It's just, it's there for you. So being able to do these types of rollouts and these types of changes, which affect uber.com, which is kind of a big deal is scary if you don't have the ability to slice the traffic. And the sort of techniques that we were able to work with your team to deploy to really have our configurations focus on like slices of traffic in regions or based on header information or based on other parameters that we could set allows us to do similar sort of progressive rollout that we're doing inside of our environment on the edge as well. So that was just a really big unlock to allow us to be safe. But then also now we have a conversation about like, okay, we wanna make this change to our configuration to adopt a new feature or there's a deprecation for a configuration format. We wanna go to this new one. Great, we have a way to do it such that we're not gonna immediately take down all of Uber just because we do this migration. And that's actually, I think a really important tool to have no matter what your system is, but especially for your edge. Cause again, like no front door, like no one's coming over, right? So that's a big deal. So if you think about like just the places beyond billing where you're working around Cloudflare, right? What is your biggest workaround right now that you say, well, we have to do this differently because Cloudflare made a choice that's frustrating to us. Yeah, I think one of the challenges is always the sort of like, the global Internet's kind of a complicated place. And I think that we do see that in certain countries in certain regions, connectivity is not as good as we'd like it to be for certain providers. And like, we can, because we have this like amazing distributed kingdom, quote unquote kingdom type setup here where like it's mobile phones all around the world. We get to see a lot of like what's happening globally at any given time. And you can see that like there are places where the connectivity between Cloudflare and this provider is not as good as like other paths that we might be able to take. And so we have worked around just in like specific regions in order to increase reliability. We've actually put in like a sort of fail fast method where we can like, you know, instantly flip between, you know, one path or another if we need to just to work around those without, you know, much interruption to someone who's using the app. On the client? On the client side, yeah, yeah. Being able to own the client has been new for me. It's been a really powerful thing to be able to actually solve some of these problems. You know, Chrome won't do that for you, right? So this is a really useful tool in the toolkit, but like ultimately, you know, I like to think that Cloudflare is a company that thinks big. It'd be interesting to think about like what does the Internet look like in the future? Like, is there other ways for the ISPs and the mobile networks to integrate better so that these aren't just persistent problems like the way that they currently are today? It does feel very, I don't know, I'll say like early 2000s Internet. Sometimes I'm just like, okay, I guess that someone didn't plug a cable in, but it's real. It's kind of how it works, right? There's still switches and routers and things like that, but there's a latency penalty when it happens and then potentially, depending on the specifics, you may or may not have an alternate path for that particular city or region or whatever it is that's being impacted at that moment. We do have some work on the H3 quick congestion control, that last leg. Yes. But what it's missing is a partner who has the client-side telemetry to tell me if it's working. I wonder who might have that, yeah, yeah, yeah. You see, I didn't realize. Yeah, folks are nodding their heads. So yeah, I mean, yeah, yeah, this is good, yeah. Okay, last question and then we'll turn it over to the audience. If you had one magic wand to have us build one thing on Cloudflare, what was the one thing you would just say? It doesn't exist today. I wish you guys did this. If there's one thing, I think this is a really good question. Okay, so I talked about log push being like an awesome feature. I think there's like a real interesting challenge because getting the logs out of Cloudflare means that you have to have some log system on your side that actually can ingest it. And I think that's actually, we are doing it, we're a big company, but like I really do think that having long horizon aggregate statistics about your traffic is like super useful. And I think that like having more of that provided like inside the box in Cloudflare could be a really awesome feature because there are a lot of problems that you just can't see unless you look at specifically where your traffic is coming from and what are the headers, what are the response codes, like how are the clients behaving. That kind of, even in aggregate, that kind of thing back seven days can be a big difference. Like you might not detect a small problem for a week, but when you wanna go look at those logs, if you don't have them already, then you're kind of in trouble. You know, if that's something that could be in the box, that would be really great to have. So that'd be my push. That's a good one. My log push. Your log push. My log push, yeah. We did announce a log transform product earlier this year where you can do like on the fly coalescing and transformation of formats from our format to yours for that ingestion process. Yeah, I think that ingestion is its own like hard ball of wax to deal with. And I don't know, I'd say that you could enable a lot of people if that could be a thing. Awesome. Thank you. Thanks very much. Thanks, Maddie, very much for being here. So much. This was wonderful. Yeah. Thanks a lot. Thank you.