⚡️ Speed Week: Signed Exchanges, Early Hints, and Tiered Caching

Hello and welcome to our Cloudflare TV session today on the Speed Week.

So we had a few really nice features launching today and I'm very happy to have David and Alex here today.

My name is Marc.

I'm a product manager for Cloudflare's Emerging Technologies and Incubation team and I'm joined by David and Alex, who can introduce themselves.

I'll go first.

Hi, my name is Alex Krivit. I'm the product manager for the caching team and content delivery team here at Cloudflare.

Hey, I'm Tubes. I'm the product manager for the network availability and Argo teams at Cloudflare.

Very good.

Yeah, let's maybe start with Alex today. So we have launched a few new products that improve performance like for everyone.

Let's maybe start with tiered caching.

What is tiered caching and why is it great for customers to have it? Yeah, I think that's a good question.

A good place to start. So as I think probably most viewers and most people watching this may be aware, Cloudflare is a very, very large network.

This network is comprised of data centers and these data centers are all connected.

They all talk to each other. And essentially the strategy for building out and getting new data centers is we want to place these data centers very, very close to the end user because this helps with things like performance.

The requests and responses from the end user don't have to go long distances over ISPs and connections and networking incidents that may have congestion and other things in order for this end user to receive a good performance like content that they want.

Tiered cache sort of refers to a concept where we take this massive global network of data centers in all of these different cities all around the world and we organize them into a hierarchy of lower tiers and upper tiers.

And so generally, as I was saying, these lower tiers are ones that are close to an end user.

They may be getting a lot of these requests.

When that request comes in, the data center looks to see if content is cached there.

If content is cached there, it can respond.

Everything's great. Performance is great. But if the content is not cached there, then the lower tier has to ask an upper tier data center for the content to see if it has it cached.

And if it does, it can be filtered back through and recached at the lower tier.

And these upper tiers are larger data centers. They have more capacity.

They're generally fairly close to common origin pools such that when an upper tier may not have cached content, then the upper tier can ask the origin for it.

And so by limiting the subset of all of our data centers that can ask an origin for content, you really allow for the limitation of egress.

So you're saving money for data transfer between your origin and Cloudflare.

You're improving performance by again caching these things close to users and transiting through Cloudflare links between data centers, which are generally faster than between Cloudflare and non-Cloudflare data centers.

And so it's sort of a good experience all the way around is this tiered cache.

And it was formerly a paid product as part of the Argo suite of products, which includes smart routing and some other things.

And so today, we're announcing the availability of this product to be used by anybody free of charge.

Just go to your dashboard and turn it on and save money, get better performance.

Great. And we are not only launching tiered caching for free to our customers, we are also launching something called Orpheus.

Correct. What is Orpheus? And why is it great for everyone to get it for free?

I think I'll let Tubes answer that one. Yeah, thanks, Alex. So yeah, so we have this product called Argo.

And as Alex mentioned, an Argo's job is to kind of navigate the Internet and find the best, fastest path for our customers.

And so if you pay for Argo, you get, we find the fastest path back to your origin for dynamic traffic or non-cached content.

Argo is really good at kind of dynamically routing around problems on the Internet, right?

Like if there is a link between Cloudflare and the origin that's showing sporadic packet loss, Argo's really good at detecting that and routing around it.

And that functionality isn't just about being faster, it's actually about being more available and being more up.

Argo allows us to kind of help make a statement that says, if your origin is up, then if one Cloudflare data center can reach it, every Cloudflare data center can reach it.

And that's because Argo will route through different Cloudflare data centers over different transits over whatever to find the fastest path back to your origin and packet loss makes you slower.

Well, in kind of hidden in that statement, packet loss makes you slower is kind of a larger statement, which is packet loss makes you less available.

If there's packet loss back to your origin, your requests are going to time out and take a really long time, but they're also going to fail.

And so failure is just not an option for customer origins or even Cloudflare, right?

Cloudflare availability is kind of table stakes to Cloudflare's existence. If we can't deliver packets back to your origin, we're not really doing a good job of delivering our value proposition.

So we've released this thing called Orpheus and Orpheus basically takes the availability part of Argo and gives that away for free to everybody.

Basically, what it says is if we detect packet loss to the point at which your requests would fail, we'll route around it and we'll route it through different Cloudflare colos.

And Orpheus actually kind of behaves and thinks about the world a little bit differently than Argo, whereas Argo optimizes for time to first byte and making you faster and bringing latency down.

Orpheus doesn't really care too much about latency. If Orpheus is engaged, then latency is not a problem because you're already going to be really slow and to the point of failing.

Orpheus's job is to kind of pull you back in from the brink and steer you around trouble so that you may see like a latency penalty, but you won't fail your traffic.

And it will actually probably help increase your performance because requests that had previously failed due to timeouts will no longer fail due to timeouts.

They'll succeed. And that kind of mentality is really what Orpheus is about.

It's kind of saving your traffic and reachability and all of your traffic needs to get where it's supposed to go.

That's great. Do we have any numbers on how much it actually improved reachability or like how much it improved the error rates on traffic that we routed in a way that it wasn't reaching its goal?

That's a really good question.

And the answer is yes, obviously we do. So we've rolled out Orpheus to all plans.

The biggest source of data that we have is when we rolled this out to free customers.

And free customers have origins. Free customers have kind of two different groups of origins.

And the first group is, I'm going to talk about it a little bit, but it's not super relevant.

And that's basically origins that are unreliable themselves.

So maybe like, you know, you're me and you host a server in your office, and that is hosted on Cloudflare.

And the origin may go down because your cat steps on the power button and restarts the computer or something like that.

That origin is not very reliable. And there's something that happens quite often, and that's why the store is closed.

But the other side of that is an origin that's, let's say, hosted on one VM in a cloud provider, right?

Like the cat's not going to step on the power button, but it's one location being serviced by the entire world.

So if you're in, let's say, the central United States and out of Des Moines, then, you know, if a user is connecting to you from, let's say, Singapore, or, you know, Malaysia, or if Singapore is in Malaysia, if you're in, say, Singapore, or Thailand, or the Philippines, or even, you know, the Middle East, it has to travel a really long way to get back to your origin.

And if that over and the longer you have to travel to get back to your origin, the more likely you are to see packet loss.

And that's actually part of the reason why tiered cache exists as a product, right?

Like you want to minimize that time. But if you can't, then the best thing you can do is order your routes in terms of reducing packet loss.

And Orpheus does that. And when we did this for free traffic, we saw that the origin reachability or essentially the number of times that Cloudflare was able to reach the origin without receiving or some sort of packet loss timeout was about 0.13%.

And when we added Orpheus, when we rolled out Orpheus, that number went up to that number went down to kind of 0.1% packet loss, or 0.1% failures due to packet loss.

And so that's an improvement of about 23%. And we're pretty happy with that.

And we're pretty happy with that as kind of a first pass.

We definitely want to bring that number down, down, down, down, down. And Orpheus is just like all of our other products to ever growing.

It's going to become beefier and it's going to become more resilient as we go.

But that 23% gain in origin reachability is definitely something that's really going to help our free customers and all of our customers, to be honest with you.

That's great. Do we have any numbers we can share on tier caching?

How much it actually improves our cache hit rate?

Yeah, I think that's a good question. Obviously, one of the major benefits of caching and particularly tier caching is the potential improvements for increasing that cache hit ratio, reducing the miss ratio, things like that.

They're obviously very dependent upon customer's traffic profile.

But with sort of the initial tests that we've observed and seen, this can improve cache hit ratios from somebody who doesn't have tiered cache.

It can reduce the miss ratios by over 60%, which is pretty substantial and can definitely help improve the economics of running a website because there's frequently less egress data transfer costs that the website operator needs to use and pay for because of us serving things from cache.

So that's great and definitely helps the website operator run their website in a scalable cost effective way.

Cool. I would say we'll move to our next announcement, which was all about Argo 2 .0.

We already kind of touched Argo slightly, but haven't really fully introduced it.

Tubes, do you want to give it a go on letting everyone know what Argo actually is?

Yeah. So Argo, the way that we like to talk about it is it's the Waze of the Internet.

But what is Waze? So Waze is an app in the United States.

It's similar to Google Maps. So basically, you're in your car and you plug in where you need to go.

And then Waze and actually Google Maps does the same thing.

If let's say you're driving along and you see a car accident and you're hauled up in traffic by a car accident, you can go on Google Maps and say, hey, there's a car accident where I am.

And if a lot of people do that, then the next person who logs into Google Maps to get directions, they will get directions that steer them around the car accident.

That basically says, like, you shouldn't get held up in traffic because a bunch of other people are held up in traffic.

And that's kind of what Google Maps and Waze do when they do navigation.

And that's exactly what Argo does. Basically, Argo says, I need to get from point A to point B.

I want to find the fastest path. So I will report my latencies up to Cloudflare, how long it took me to get to where I needed to go.

And basically, that allows us to compute a table and basically say, if there's a faster path somewhere else in the world, we'll move all of the traffic over down that fastest path so that you get the best possible performance.

And typically, that's resulted in about a 33% reduction in time to first byte, which is pretty substantial.

But the things that we've been rolling out recently, and so as we've been going with Argo, we started at layer 7 with HTTP.

And then we started moving down the stack to Argo for protocols, so Argo for TCP and RTMP and lots of other layer 4 protocols.

And that added additional latency reductions. However, one of the things that we really haven't tackled is the last mile.

And basically, if you talk about the leg of a request and where Cloudflare sits, when you talk about from application stacks out, you think about the last mile as the last leg of the request from the perspective of a cloud service.

So let's say that you're connecting to, again, your theoretical origin hosted in Des Moines.

That path, if we think about it from Des Moines out to the user, the path from Des Moines to the cloud provider's edge is going to be the first mile.

So your data lives on a VM. But once it gets into the cloud provider, it has to travel some distance to get to the VM.

That's the first mile. Then you think about the cloud provider to Cloudflare, and that's the middle mile.

So basically, Cloudflare's Argo optimizations, that's where that lives.

And that's where we optimize finding the fastest path.

And then you think about Cloudflare out to the user, and that's really the last mile.

And we think about the last mile because it's the farthest away from where your data lives.

And Argo and a lot of cloud providers and a lot of services don't touch the last mile.

And we'll talk about that a little bit later on eventually.

But suffice it to say that Argo is starting to optimize a little bit more for last mile latencies.

And one of the key improvements that we've done is we've started to optimize last mile paths for our Argo customers.

So if you buy Argo today, or if you're an existing Argo customer, you should see your latencies drop by an additional 5% because we are optimizing for your last mile around trip time.

So basically, we want to make sure you get on Cloudflare super fast, even faster than before.

And we do that through our traffic management and just basically in our Anycast network and basically making sure you get where you need to be fastest.

And that gives us an extra 5% latency bump, or 5% latency reductions end to end.

And it helps reduce last mile around trip times by about 40%. So it's nothing to sneeze at, some big numbers that we've brought with that.

And that's one of the things that we've started to do.

And as Alex and we just talked about, tiered cash is now free for everybody.

So we want our Argo customers, we want to show them some love, we want to show we still care about Argo, we still care about optimizing performance, we still care about producing stuff.

And so these new last mile reductions with Argo actually go really nicely with tiered cash.

Because we're optimizing last mile, your latencies with tiered cash, if you buy Argo, will actually go down even more, because your lower tier will be closer to you.

And that last mile around trip time will be smaller.

And so then that's just one of the things that we're doing with Argo.

The second thing that we're doing is we talked about before about how Argo started at layer seven, then we moved to layer four.

The other big announcement that we have is we're actually moving down to layer three now.

So basically, Argo works at the IP layer to optimize one-way latencies throughout the Cloudflare network.

And for Magic Transit customers, what that means is if you're a Magic Transit customer, and you have been thinking about, if you're a Magic Transit customer, and you want Argo, and we've previously told you we don't support that yet, now we do.

Congratulations. That's the big, really big announcement.

And the other big announcement is that Argo Smart Routing will eventually work for everybody, regardless of whether or not you're a Magic Transit customer.

And this has a lot of really great applications in finance, in gaming, and anybody who wants to be part of the Cloudflare one suite, and they want even greater performance optimizations, or even if they're not part of the Cloudflare one suite, and they just want even greater performance optimizations, optimizing these one-way latencies definitely allow us to get about another 10% latency reduction on top of the other latency reductions we've seen.

So we're super excited to be delivering this and really kind of a new landmark or a new chapter in the Argo story.

That's amazing. What would you say is kind of, what's your favorite thing about Argo that we're pushing out or that we have worked on over the last month that is really impacting our customers the most?

So I'm definitely going to say Layer 3 Smart Routing is great, and the last mile latencies are great.

And we definitely have a lot of great data on that, and we're super excited to get more customers.

I personally am super excited about Orpheus.

Orpheus has been rolled out for a bit, and it's already saved a bunch of customers some pain due to things that happened on the network.

Watching it engage and steer traffic away during times when the Internet is not behaving the way that we would expect the Internet to behave is really, really cool.

And in the coming couple of months, we're super excited to be able to give those metrics to customers so that they can see when their traffic is being saved by Orpheus.

But that's really the cool thing about these things is that they just kind of work, and they just kind of work out of the covers.

That's really, I think, one of the value...

We talked about reliability and your performance is kind of the value propositions of Cloudflare.

And the metrics for me are the coolest part, because you turn on Argo, you turn on Orpheus, you turn on tiered cash, you turn on any of these things, and they just kind of work for you, right?

They don't actually...

You just set it and forget it. And checking the metrics is a really good way to be able to say, oh, yeah, look at what I've been...

Look at all of the stuff that Cloudflare has been providing me.

Look at all of the performance. Look at all the times that the Internet was messed up, but my traffic got saved is really cool.

It's definitely got that kind of vibe of like, hey, it's raining outside, but I get to sit inside with a good book and look out the window at all the other people who are getting wet.

Yeah, no. And I think that to sort of echo, I guess, what Tubbs is saying, that's, I think, one of the best, most powerful things from both a customer perspective and then a product perspective, being able to see in sort of like real time, like, oh, this is what would have happened had Orpheus not been able to route around this problem, had tiered cash not been able to save this person a bunch of their egress costs.

And so it's definitely these metrics and the benefits are good both for the customers, they can see the benefit of the product immediately.

And then also from a product perspective, being able to see like the power of like, oh, there's like a big issue on this link somewhere and we've been able to serve everybody safely, successfully, nothing broke.

You just, from a user perspective, you pushed a button and then Cloudflare magic sort of kicked in and did all the rest.

And so that's, I think, both powerful from both perspectives.

Cool. Thank you very much. There's one last feature released today. I am proud to talk about that one, which is Signed Exchanges.

So we launched Signed Exchanges as a collaboration together with the Google team.

What it does in short, it actually makes your website available super fast when you come from Google search.

So the idea from Signed Exchanges is to make a website available in the browser cache that can kind of be surfaced from there more or less immediately when you click on a search result.

To do that, you actually need to make sure that this website you develop from a browser cache, this package, is actually originally coming from the actual origin and is like the actual website or the actual assets that you wanted to have.

And to do that, like the Signed Exchanges technology comes in and what you need to do or what you need to do is you package up the website that you want to send, you sign it, you make sure that it's actually the real website, the real thing, and then you make it available on Signed Exchange.

And when the Google search crawler comes and finds a website, they actually see, hey, there's the Signed Exchange package.

I'm making it available on the Google cache, but also making it available for prefetching from the browser.

We do see that it kind of leads to quite a significant drop in Largest Contentful Paint time, which is one of the core web vitals.

That means the customer sees the website a lot faster, but it also means that if your LCP, so the Largest Contentful Paint, goes down, that also means that your search result ranking goes up because a lot of the search result rankings and the SEO depends on your website being fast.

So with something that is as fast as this, this really helps to also boost the website in terms of availability or ranking and search results.

One more thing, it also helps on availability on when the origin is not there, when there's, as you are able to cache everything also through the journey, it does add not only the latency advantage, but it also adds the redundancy of the data that you're making available.

It's great at the moment.

It's available for a Google search only. It's an open standard.

So we are hoping, and Matthew put out a great blog post today about how he sees the future of the Internet, that there's a lot more applications for technologies like this to make the Internet faster, but also more accessible for everyone by using technologies like the signed exchange.

Mark, can you talk a little bit about what problem the signed exchanges are trying to solve?

What are we doing here with signed exchanges?

Why is this a big deal? There's two things. There's one big deal, why it is amazing for users, because you're not waiting for a website.

You're just clicking on a search result and it's there. As you load all the static assets, prefetch them, it is just there when you click on it, which is amazing.

And it's kind of something that we expect to be a completely new feeling on how you browse the web or how you kind of react to search results.

I think the other thing is why isn't that available for everyone and why can't just anyone do it?

I think that's the other thing that is quite amazing because it is quite complex.

The whole packaging and certificate handling and signing is something that's not super easily done.

And that's why we kind of took it on as Cloudflare to make that available for our customers super easily.

So our customers just need to click a button, they just need to make it activated and we handle all the rest.

We care about the certificates, we care about the signing, we'll make it available, we encrypt it.

So there's no actual work that needs to be done on our customer side.

And this is something that where we are really happy to make this available to a broad set of customers by just clicking a button and using the product.

Yeah, I mean, that sounds incredible. It sounds like, I mean, this is going to be super easy from a customer perspective.

You sort of just push a button and then everything else is just kind of handled behind the scenes.

Have we seen any sort of data or metrics that tell us the improvements that signed exchanges could have for a website yet?

We have seen some examples of it working and we got that due to the collaboration with Google, where it really had a significant improvement.

When it's fully prefetched and it's a completely static website, it increases by, the load time goes down like more or less by a hundred percent because it's there immediately.

Of course, it always depends. And that's why we're launching it in the beta is to kind of make sure we are now testing it together with customers and making sure we configure it in the best possible way to make exactly the assets available via the signed exchange that the users need to make the website as fast as possible.

And that's something we'll for sure be able to report back on how the actual impact is on our customers.

So what's the, how does this from like kind of a functional perspective, if I'm a customer and I have, you know, like a webpage, why should I use signed exchanges over something like, you know, Google's AMP project or Facebook's equivalent?

Well, the big advantage like Google or like signed exchanges is generally available for everything.

It's like an open standard.

It's not linked to like the AMP technology or something like that.

So you can just, you could just use it. And it's like works on every tech stack that you have.

So it doesn't rely on any of those. It just makes stuff available.

And I think it's that like this being agnostic to a lot of, a lot of things of technologies that you use, but also in terms of being an open standard that everyone could adapt.

This is something that makes it a lot easier for customers to actually use the standard and kind of make their website, make their website faster through this one.

Oh, that's great. So yeah, that's super awesome.

We're super excited for signed exchanges. Yeah. I mean, it sounds like it'll have a big impact and that's great.

It looks like somebody just said that we have about a minute left.

So I just wanted to, I guess, say, you know, that all of this information is available on the Cloudflare blog and there should be directions for how to enable all of these features.

And we look forward to seeing what sorts of improvements you all find for your websites when you enable these things.

And please, you know, tweet us and let us know if you're seeing improvements and yeah, super exciting.

For everybody here. Thanks and have a great day. Thank you, everyone.

Bye-bye.