Canary to cloud: Building a flexible traffic strategy

Thank you all for coming to our breakout session, Canary to Cloud, Building a Flexible Traffic Strategy. By way of introduction, my name is Noah Crouch. I am a Senior Product Manager at Cloudflare, primarily working on our load balancing product, but also a few other things here and there. I'm joined today by Oscar Moreno Garza, who is a Staff Software Engineer at Uber working on the Internet Traffic team. I think I got that one right, Internet Traffic. Yes. Cool. As you may have noticed from the title track, today we are here to talk to you a little bit about traffic management. I'm going to walk through a little bit about what we're seeing in the industry today, where what we can call rudimentary traffic steering algorithms may break down, and how a robust system can overcome some of the challenges that arise in hybrid type environments. Then, I'll hand it over to Oscar and he will go through how Uber scaled to over a million requests per second, or about a million requests per second from the Cloudflare platform, encountered said challenges that I will have described, and then how they solved those and led to a great success story. So, to get started, let's kick it off with a statistic because everybody loves a good stat. Seventy-three percent, that is the number of estimated organizations that are running a hybrid Cloud setup today. Hybrid Cloud is a great buzzword, but what does it actually mean? It's not very descriptive. So, included in this number, 73 percent are organizations that are running any combination of multi-Cloud, on -prem, or private Cloud. So, organizations that are running in AWS, Azure, GCP, included in this number. But more importantly, the type of organization that we're going to talk about today is the customer who may be running in GCP, but then also has compute and applications running in a private data center on-prem, as well. So, this number, 73 percent, has stayed mostly stable for the past five years. So, why do customers and organizations run in a hybrid Cloud environment? There are a variety of reasons. I spent a whole talk just going through those reasons, but ultimately it boils down to two reasons in particular. It's the fact that running on a hybrid Cloud environment gives you unparalleled flexibility and resiliency. Cloud One has an issue. If you're only in that Cloud, you're kind of out of luck. You're down until they get back up, which is not great for you, not great for your customers. Multi-Cloud environment, hybrid Cloud environment, take the traffic that goes to that one, shift it to the other, and then you're good to go. Or, God forbid, both of those Cloud environments are having a bad day. The Internet's just not happy today. You can take some of that traffic, shift it over to your on-prem environment, and you're still somewhat up and running, giving your customers a better experience than they would have otherwise. So, operating in the hybrid environment gives businesses immense amount of agility, but also introduces some very fundamental infrastructure challenges that make intelligent traffic management very critical and increasingly critical as we see more and more businesses adopt this type of operating. So, let's take a look at some of those challenges. Challenge one is going to be heterogeneous hardware and environments. So, your backends are running in different environments. They are necessarily going to have different amounts of capacity. It's true even when you're running in separate Clouds, it's going to be different behavior. Not every Cloud is going to operate exactly the same. But more specifically, this is going to be especially true for the organization that has applications running in on-prem as well. I can absolutely promise you that the server that you have running in the dusty broom closet is going to have infinitely less scalability and compute than the application that's running in Cloud environment. Second is the active -active type setups are becoming increasingly minimal and mandatory. In years past, you may have run active -passive. Let's send all of my traffic to one data center, wait for a disaster to hit, we'll be down for a few minutes, route traffic over to the other data center. However, with active-active, typically find that organizations have better resiliency and global performance overall. It's much easier to route traffic to a new data center that's already accepting traffic as opposed to trying to spin one up, do a DR and send traffic to a cold site. Then third challenge, failures can cascade very, very quickly. So if environments have differing amounts of capacity, there's a very real risk of a thundering for type problem that leads to cascading failures. So we'll go back to the example we gave a little bit earlier. Imagine you have an application running in the Cloud. Some of those nodes go down, become unhealthy. We'll take that traffic, shift it elsewhere. Except we've just shifted so much traffic into our on-prem environment and now our on-prem environment is also unhealthy. So now we have to take all of the traffic that was going to our on-prem environment, shift that back to the Cloud, except that Cloud is busy tearing down unhealthy nodes and spinning up new nodes. So then you just have a thundering herd, unhealthy, unhealthy, unhealthy, unhealthy, so on, so forth. Yeah, not great for anybody. So when a team's attempted to solve these challenges in bridge these hybrid environments, which are called rudimentary or typical traffic management techniques, the architecture inevitably fractures against and across a variety of failure modes. We're going to focus on three particular failure modes today. First one is under or over-utilization because of blind round-robin traffic steering. That is a mouthful. Round-robin is the simplest steering algorithm that you're going to find out there. Let's take request one, send it to node one, quest two, send it to node two, so on and so forth. It's still around because it's incredibly effective. Except it's incredibly effective in very specific circumstances. Those circumstances being, you're operating in an environment where all of your different environments have a homogenous amount of capacity. We're literally just taking the load, shifting it out as evenly as possible about across all of our different centers. But this begins to break down in heterogeneous environments with differing amounts of capacity, in which case you're either going to under-provision and your lower-capacity sites are going to become overloaded, shift traffic elsewhere, or you're going to over-provision and then you're just going to be wasting money because you've over-provisioned and you're doing blind round-robin. So then surely a potential solution here is that we want to maybe route by latency. Yeah, it's got to be the solution. Unfortunately, again, probably not. Latency-based steering can work great or it's great in homogenous environments, can also work great in some multi-cloud environments because the latency that you're going to find from the end-user to any of the big public clouds is typically going to be somewhere around the same. But as soon as you introduce on -prem hardware into the equation, the math just completely changes. So you'll have your cloud-based application will attract all of the traffic because it's going to be responding most quickly unless your users are directly co-located, which typically is not going to be the case. And then you have two outcomes. Outcome one, your on-prem is just never going to see any traffic whatsoever. You're going to be wasting money because you have servers that are sitting idle. Or outcome two, you have issues in the cloud, you try to shift traffic onto the on-prem, and then we have that same thundering curve problem that we talked about earlier. The third failure mode is configuration sprawl. So say you want to do something as simple as manage a canary environment. Ordinarily, that's going to be as simple as configure a few routing rules, you know, query string, header, configure those routing rules, boom, you've got a canary environment. When you're doing hybrid environments, you then have to take that configuration, replicate it from one spot to the next spot. It's still not great, even though those may be different configurations. But what happens when you scale that to 10 applications, and then to 50 applications, 100 applications, 1,000 applications, as we'll see here in a little bit? The management simply becomes untenable. Yeah, there's ways to automate it, but they're not good solutions. So to prevent these failures, we have to move past blunt routing rules and adopt a more intelligent, capacity-aware traffic management strategy. A intelligent traffic management strategy should be able to accomplish at least these three things, probably more. So capacity-aware distribution, instead of doing something like proximity steering, blind round robin, you need to be able to direct your traffic on something more closely resembling the actual capacity of your backends. You need to be able to have programmable and easily manageable edge rules to take care of those edge cases. The canary use cases, things like that. Importantly, they need to be easily manageable. And then finally, you need uninterrupted resilience. We mentioned earlier that active-active is becoming the norm for both performance and cost reasons. An intelligent traffic management platform needs to be able to fail over seamlessly, near real time, and take traffic that was destined for one location and shift it in a capacity-aware manner to other locations. Now, up until now, this may have sounded all boringly theoretical. I promise you it's not. This is the exact production architecture that Uber has deployed to scale on their Cloudflare platform from zero requests, as of 2023, up to over a million requests per second today. So I will invite Oscar up on the stage. As he's making his way up here, we'll take a quick, brief look at the timeframe of Uber's on-ramping. Welcome, Oscar. Starts in 2023, we got onboarding. Fast forward six months, we are live in production. Six months later, we're doing active-active with Cloudflare. Feature parity is at 99% and 30% of traffic's on Cloudflare. Fast forward a little bit before today, we're at 50% of core trips, 100% of critical web apps, 100% of employee traffic. So with that, I'll hand it to Oscar to talk through the problems they encountered in this journey and how they solved them. Thank you, Noah. Yeah, so my name is Oscar Marino Garza. I'm a software engineer on the Internet Traffic Team which, you know, you can probably guess from the name, we handle the Internet traffic coming into Uber. Just going to talk about some of the problems we've run into as we scale the systems. And I think, like, you know, organizations are going to run into these problems sooner or later as you'll scale your systems. I also want to call out, like, these particular solutions make a ton of sense for Uber, but they might not be the best fit for your organization. So, you know, your mileage may vary. So, yeah, these are not silver bullets. Okay, yeah, so big picture, right? H2023, Uber started with a single cloud provider, which was the FastPath, and then we had on-prem data centers that we could use as a backup. Fast forward to 2026, we now have Active -Active with both GCP and Cloudflare as the FastPaths, but also we have the, we retain the data center on-prem capacity, but we honestly, and I don't use it that much. Now, this doesn't mean, like, oh, it took us, like, three years to onboard into Cloudflare. No, it actually took us, like, around six months to serve, like, a decent amount of traffic. But we are continuing to work with Cloudflare on multiple different things. You know, HTTP3, decentralization is one thing that comes to mind. So there's, like, an ongoing effort going on. Yeah, and then just to clarify, like, that MVP, six months, rider traffic. So if you ever requested an Uber, Uber Eats, that sort of stuff. Yeah, so basically, at the core, we have DNS that handles the, kind of like, we have, like, a weighted DNS setup for Active -Active, which we will send you to Cloudflare or GCP in this case. But for this, though, we're gonna, like, focus on the Cloudflare configuration, which is pretty straightforward. We basically have a single zone for each top-level domain. So one for Uber.com, Uber Eats, so on and so forth. Right now, we have around, like, 120 to 115, I wanna say. They keep growing. Yeah, but in general, for each particular website we have, we actually have around 6,000 configurations. So overall, across all of those aggregated, 6 ,000 different websites configurations. And what do I mean by configuration? It's, like, such an overloaded term, right? So basically, like, NGINX and our proxy configurations later down the stream. So where do I route to on the Uber service match? What is, like, the authentication level for this request? And then there's also, like, things like, okay, what kind of parameters does this map against our DDoS efforts? So a lot of those, it's a pretty big configuration. There's 6,000 of them. It's a lot to manage. So obviously, we don't wanna create 6,000 cloud-friend configurations for this. That's why we only create the top zones and then manage the more, like, Uber-specific logic inside our Envoy proxy and NGINX services. I also wanna say, like, in general, we are currently handling right now around, like, 2.2 million requests per second. So half of that traffic goes to GCP. Half of that traffic goes to Cloudflare. But also, during important dates, such as Halloween, Fourth of July, that traffic cannot double, right? So we have to also keep that in mind when we are designing our systems. Now, this doesn't mean, like, okay, you mentioned there's a lot of Uber -specific logic later down the stack. This does not mean, like, Cloudflare is just, like, a simple proxy. We just send all of the traffic. No, no, it handles, like, a lot of really important configurations and, like, most notably, like, TLS, MTLS for some users, DDoS, volumetric protection, but also the first hop in our load-balancing scheme, right? So I'm going to talk a little bit about the Uber kind of, like, architecture. At its core, we have, like, a regional topology, and we have two main regions, DCA and Phoenix, and they are named after the closest data centers. So there's one in Washington, one closer to Arizona. And then inside those regions, we have zones. And, you know, for the example for Phoenix, we have, like, Phoenix 7, 8, and 9. These are kind of, like, what we look like a couple of months ago, where all of those zones are on-prem. So they look pretty much the same, even though if you've read, like, blogs about Uber, our compute kind of, like, cluster is heterogeneous. We are kind of, like, a special team. We get, like, special preferential treatment, and all of our hardware looks the same to help us make sure that we're able to route traffic safely to the back-end services. So for this particular configuration, everything looks the same. Yes. Yeah, I also want to say, like, the way that this is designed as it is is because whenever there is, like, an issue on a region or a particular zone, we are able to quickly just flip a switch and enable or disable one of the pools or the back-ends, the endpoints, and it just takes, like, a couple of seconds, which is amazing. I don't know what kind of magic Cloudflare is running there, but it's so good to know that you can drain in, like, one or two seconds. Now, here's kind of, like, the first problem is what happens when your assumptions around homogeneous capacity, but also latency, are broken, right? So there's this big push to migrate to Cloud, and we set up a Cloud zone. Everything's perfect, everything works. But the moment we start sending traffic to it, we realize, okay, there's two problems here. One is the new Cloudflare-backed zones have, like, a much better latency, right? Usually, Cloud providers are much better at latency than if you go and build your own data center, right? They just have those intercontinental cables in the C, right? But also, the other problem is not only the latency's better, but the throughput for these new servers is actually a lot better, too, right? So now you have, like, those two problems that you have to figure out, right? So, yeah, I mean, if you just go and use one of, like, the default of balancing algorithms, like, that might work out, like, perfectly okay, right? Specifically, if you're, like, just focusing on latency, because latency will encode things like capacity by default. But we care not only about latency, but also about the utilization, right? Like, we want to make sure that if we are paying for a specific server, we actually get, like, the full use out of it, so we can pass, you know, the savings to the customer. So how do we become explicit and kind of, like, encode this sort of capacity slash latency on the backends? Yeah, so luckily for us, there is a very well-known kind of, like, algorithm that Cloudflare already has on a traffic stack. It's called weighted round-robin, and it, you know, it's, like, super simple. You just kind of set a weight for each backend inside your pool, and then you can kind of play around with the capacity and latency and a bunch of, like, different factors, right? And then for our Phoenix 50, so on, which is just, these are mostly real numbers, we just kind of put a 2x factor on it. So that gets, like, 50% of the inferred traffic, and then the other legacy on-prem servers get 25 and 25% each. So now everything's, like, perfectly balanced, right? Yeah, I know, like, weighted round-robin is probably not, like, super groundbreaking, but honestly, I kind of love it. I feel like you have to be, like, a genius to come up with the algorithm in itself because actually you can, like, encode so many different things into that particular number. So for us, we care about the latency. There's, like, some piece of code somewhere running on the overstack that updates this on the fly based on capacity, throughput, and also, like, the latency. Yeah, I wanna kind of, like, have a little pause here and give a bit more context about some of the Uber architecture before we go into the second problem. And basically, the main thing you need to know is the mobile application actually uses, like, different host names, like DNS host names, in order to run different sort of experiments, right? We mentioned, like, HTTP3. There's a specific host name we are reserving to do some testing on HTTP3. We have this service called Dispatch Center, which, you know, at its core, I mean, it's a very complicated service, but at its core, it's a key-value data store, and you can map a request to a specific host name and then map that particular host name to an experimental group, right? And then an experimental group has, like, some, like, routing, some Uber-specific routing characteristics that will push you to a specific instances on our kind of, like, Kubernetes cluster. So using this Dispatch Center service is how we've implemented canary routing for Uber. So, yeah, actually, I guess, like, surprise announcement also for Uber employees. We've been running this for a couple of months now. We never made it kind of public, but if you are an Uber employee, you are already, like, pinned to canary right now. If you open your mobile application and just look at the network logs, you will see, like, CN Canary, CN Canary. But, yeah, just a little bit more context. CN.Uber.com is our main API. CN Canary is, like, our new canary host name. So, okay, here's a bit more context. We have this system that kind of rolls out in a sequential manner to different zones. And inside our stack, we reserve DCA20, the zone specifically for canary. So this is, like, a very small zone. It's only supposed to serve at max 2% of the global traffic. Yeah, so here's the problem. How do you make different routing decisions for different subdomains, right? So we have CN.Uber.com, which is our main configuration. We are perfectly happy with it, right? But now we want to say, okay, for CN Canary.Uber.com, we want to change the routing. Now we want to include DCA20 and maybe isolate DCA20 so that all of the canary traffic is, like, isolated into that particular zone, which all of the Uber services for canary are running in, and it gets, like, scrutinized for, like, metrics and logs and observability. So, yeah, there's kind of, like, the naive solution. And, I mean, I think this kind of works if your scale is not, like, super big or you don't care to manage 6,000 zones and probably duplicating that. But you can just say, okay, let's just create a cloud-first zone for CN Canary.Uber.com, right? We talked about earlier how we don't do that, and that is because then our configuration will grow, like, exponentially. And, I mean, we just don't manage that, right? Like, that's a nightmare to manage. So, if only there's, like, a way in which, you know, you can define an exception for a configuration without actually taking extra complexity in either one exploring your configuration or the other kind of thing that we thought about doing is, okay, we have this system that pushes configuration to Cloudflare. What if we add, like, a little if branch in there? If you are CNCanary.Uber.com, then just create that particular zone and then do something else for the rest. Yeah, none of those are particularly interesting for us. Like, they're not very something we want to manage. Add additional if branching or manage, like, a huge configuration set. So, yeah, turns out there is. Already a thing that we're leveraging, and I want to give a shout-out to Chris Martinelli because we can engage with him trying to come to a solution to this problem. You know, the suggestion was, okay, just use the load balancer custom rules. And, I mean, yeah, I mean, I think this makes sense, right? If you're not familiar with the custom rules, they basically allow you to map a particular request to an action, right? And in the case of CNCanary.Uber .com, like, it might be unsurprising that we map that particular host name and then we set up an override pool action, and we only have DCA20 as a primary pool. Yeah, we have the rest of them for backup in case DCA20 has, like, an issue. But in normal operation, everything is isolated to that particular zone. There's, like, a couple of things that I really like about this approach. One is, like, you don't have to create 6,000 zones, right? 6,000 Cloudflare zones. That would be madness. And then the other one is, even if you were to create the 6,000 zones, you have to think about, like, okay, am I going to create, like, duplicate load balancers for each zone? Am I going to create, like, duplicate pools? Are the pools going to share health checks? Are they going to duplicate the health checks? Yeah, health checks alone are, like, a significant amount of traffic for us. So, actually, this solution just allows us to say, okay, we just have, like, one extra pool that we can share across all of the zones, just one extra health check that we are able to kind of, like, ramp up and be a little bit more quick on that health check and failover faster. And then the other thing that I really like is that, actually, like, if you look at the configuration in itself, it's the same across every zone, right? Yeah, I mean, sure, you have, like, that kind of, like, branch on the rules itself. But if you look at the configuration and what you need to understand as a human, everything is the same for all of the zones. So, yeah, that's what we end up using. And I think that's it. Noah, throw it back to you. Please. Thank you. Thank you, Oskar. So, everything we heard Oskar talk about, think about the weighted round robin specifically, I wanted to give a quick preview of something that's going to be coming very soon to Cloudflare load balancing. It's called dynamic load shedding. You may hear this referred to in other contexts as feedback-based load balancing. With something like weighted round robin, it's still on you to intuit what the capacity of a particular area, particular application is. With feedback -based load balancing, we'll allow you to return a load metric for a particular origin. This one's a 10%, 20%, 30%. That can be CPU, that can be an arbitrary metric, whatever it is that you decide, to our active health checks. Then within Cloudflare, you set a target, and then once we begin to meet and exceed that target, we will actively shed traffic away from that endpoint and pool to prevent it from becoming overloaded in the first place. So rather than having to guess that, yeah, 30%, 50%, whatever works, we're actually load balancing based on the capacity that you're returning to us. Then a little bit later down the line, dynamic origin registration. We're exploring ways that we can integrate with the dynamic scaling nature of a lot of workloads today so that rather than having to run API calls, we can just autonomously know that you're scaling up and down, add those new endpoints to pools, and then everyone's happy because you don't have to manage scripts and terraforms and things like that. Thank you.