How to measure real-world network quality for end users

I'm Lai Yi, and this is Doug, and we're going to talk about how to measure real -world network quality for end-users. Oh, we should probably get through them. Yeah, thanks. I'll start with it. Okay, great. Okay, so quick, last session, crowd participation is mandatory. Who here has taken a speed test? Okay, great. So, chances are, the speed test that you took, Doug or I had something to do with it, so sorry. So, I'm Lai Yi, I'm the head of radar at Cloudflare. You might know about radar from our outage reporting. We also report on bots activity, AI trends, but we also report on Internet quality, and that's kind of my background. I came from Measurement Lab, which runs the speed test embedded in Google Search, so whenever you Google, how fast is my Internet, it's the one that pops up. And so, a lot of what I'll talk about today and what both of us will talk about today is our shared experience with that, and we'll let you introduce yourself. Hi, I'm Doug Suttles, most known as the founder of Ookla, one of the creators of speed test. It was the first thing in Google Search until they decided to do that, and it became the second. I always like to point out that if you look on Internet archive at Ookla.com, you'll see a picture of my cat from 2005 and prior, before we started the business. We named it Ookla because of the cat, because I own the domain, and it was just a catchy domain to use for that purpose. I also like to point out this italicized quote here, where I think I'm probably as guilty as anyone for convincing the world that throughput or max bandwidth is equivalent to user experience on the Internet. That's just not the case. Today, we apologize for our standards. Yes, yes. I think that when I first started chatting with IE, I said this company, Orb, that I've started now, that's part of my apology tour for all the years that I spent not lying, because in the early days, as we know, I'm just jumping to my slides, we had this heavy focus on speed. Back then, actually, my next slide's about that, so let me make sure I get in the right order here and don't ramble too much, but ISPs have always priced things, fixed broadband by bandwidth tiers. They needed something to differentiate getting more money or extracting more dollars out of you. This whole question of, am I getting what I'm paying for, is common in anything that a consumer purchases, but for the Internet, they needed something to do exactly that, and speedtest.net was born out of that, but what it fed into was this flywheel of marketing. I like to say the biggest component of Ookla's and the speedtest business is getting all this data to then allow an ISP to use it to say, hey, we're the fastest, we're the best, something along those lines. The perception, again, of what is network quality has just, it's long been, go take a speedtest and see what the number says. Graphic here, I don't know how many of you know, Iron Man 3, I always like to point out when he, I got a call from Marvel one morning, and I thought it was, I thought I was being pranked, and they said, we have an upcoming Marvel film, we'd like to film speedtest.net for it, and I said, oh no, not with all those ads I put on there, that's awful, and they let me make a custom render for them, which got dropped into the movie. I believe that the phrasing in the movie, I didn't get the, I got a two second clip with no audio and no effects, and so we just built the little thing that dropped into the laptop screen. I'm at the movie theater, and all of a sudden you get Tony Stark saying, we need to adjust the ISDNs, is what I believe the phrasing was, I'm like, what, okay, anyways, times have changed significantly. I left Ookla three years ago, you know, it's owned by a public company, I sold it a while ago, and our ability to innovate and do something new was shackled by, you know, a lot of typical public company needs of growth, revenue, all of that. But nowadays, as I like to say, you know, the available bandwidth that you can get with connectivity often exceeds what you need to do most things, and the chart at the bottom here, this only goes through 23, but it's 20 years it's covered. I like to say that since the beginning of speed test, you know, measured throughput has gone 100x. I think the only application, aside from like massive file transfers, if you're into like big media or something, the only application that really needs this is speed tech. And it's, you know, I was on an event, a cable show event once, and one of the operators, CEOs up there bragging about their new gigabit over DOCSIS, you know, and that was the question from the audience, like, what's it for? And he said, it's for speed testing. But you know, the thing at Ookla and the thing that we did with speed test that was hugely advantageous to them was basically built their own CDN of sorts. The speed test server network is a bunch of infrastructure that is sitting inside of ISPs managed by Ookla, but it's only for that purpose. And it's, you know, can lead to some questionable things as I would say, prioritization of traffic, you know, you name it, it's not always the answer that you're seeking. It's always the optimal result. So we, oh, I got to make this point. It's my favorite analogy these days is, you know, the car analogy of, do you care more about top speed or every other thing about the car? You know, if you say top speed, okay, maybe you're one of those people that transfers large media files all the time as well. So we need something that aligns, you know, aligns with the real world experience. Orb is that for me. I took all the things that were asked for in the Ookla days that I just couldn't do, and said, let's build that. Let's build a software probe type orchestration solution that lets you measure any leg of a network, measure anything. But you know, we wanted it to be available to the layman as well as the hardcore nerd and be equally useful. And so Orb is a, it's a three-tiered measurement, you know, solutions we'd call it where we measure responsiveness, which we think is the most important component. Reliability is responsiveness over time. That's the other thing with Orb is it's always measuring. You know, we got out of like the spot checking type solutions that we had in the past and then speed, we still do speed because it is expected. And so Orb can run on almost anything. I even run it on my unified doorbell at my house as an example. You know, it is a lightweight software sensor that is constantly measuring lag. And so let's talk about lag. You know, lag is application layer latency. You know, this isn't ICMP. It's technically running over H3 and anything that happens to the connection, packet loss, you name it, you see it in a single metric with lag. It has no impact on the connection itself. Since we're running with basically an empty calorie set of packets that are flowing to these endpoints, you can do this all the time with no impact, which is another thing. You take a speed test, anyone else on your network feels it. And the last point here, I said it in the prior slide as well, but, you know, that network, you know, when ISPs are chasing to be, you know, to get a marketing message, they'll do whatever they need to, to make sure that it's delivering mass transit. And so what I was going to do here, I made a video, but instead of showing a video, I like doing demos. So I was going to hop out to actually show what Orb is. And so as you might've seen before, there is an application, Orb runs as, and by the way, it's all free for consumers. You know, I'm a big believer in this play. It's a brand play. They just give it away, differences. We're not a data business. We don't mind your data. You can even turn that off. You can send it locally to some local endpoint. We have push and pull APIs from every Orb. It is meant for even being able to run air gapped off the Internet. So it is all the things that we weren't before. Data sovereignty is a big deal to us. And like I said, runs on almost anything. So this is my Orb account, you know, and I've opened the Orb app on my Mac. And the big Orb here in the top left is the local network. That's the one running in, it's a sensor running inside of the app on this laptop. And so if I switch back, there's a couple of views here. This is the default layman view, as I would call it. And so optimal scoring, it's 96. That's probably about as good as you can get over Wi-Fi. You have to be, you know, all of our stuff has been architected with a lot of key people involved that, you know, I think our scoring is probably about as good as it, as good as anyone's out there. But let's do a speed test. And this is testing the cloud player. This is what Laie will talk about predominantly next. But if I, you know, wrap this up, we had to have a new UI, I was tired of speedometers and whatnot. It will measure full capacity. We know how to do this. My business partner that I co-founded Ookla with, the technical guy that did all the engine work behind it, he's working on this with me. So this is 100% people that used to work at Ookla, work at Orb, but have a very different purpose and a different mission. That test is finished. But now what I was going to point out is we don't automate that test. We don't want to, because like I said, it thrashes the network. It consumes so much data. We just, we just use two gigs of data to run that one test. I mean, it's just ridiculous what you have to do in order to measure capacity in the old fashioned way. And so what we do is, as you can see something here called check score, we run speed checks on a scheduled basis. And these are just 10 megabyte transfers in each direction to give you a baseline of how, you know, how, how are things going? And you can easily see if you have a problem, if something changes based on that baseline. And so the history here, I'll do a couple of speed checks, it'll add another 10 megs on each side of the up and down in the calculations here. And then I'm going to shift into what I call, ooh, I had a little interesting impact on the local network. Because the other point I was going to make is you can measure from any orb to any orb. We also measure to local router. That is an ICMP measurement to the local router, which is why I think it's not very happy with me and not behaving, but data flowing through it is still working. But now let's shift into here. This is nerd view, as we call it. Everything imaginable that we would have wanted to collect, also all available in the local API or in our cloud API, you can get to all of this data, every little detail that we're collecting. We just want to make it easy and make it so you can build your own things and do your own stuff with it. Notice the router lag really got impacted, but even the Internet lag during the speed test portion, I can highlight this by floating over this section here. It is so brutal to a network to shove enough data through it to measure capacity. So the two things we're trying to do is get people out of that mindset. You can see if something's wrong with your connection, just looking at lag, just looking at the responsiveness metric. If it's always running, if you notice something a little off with your connection, you want to go see it, just go back and look at it. You actually have the data collected already without having to go take a speed test. Again, we always still give you that capability, but wow, look at all the, yeah, the router here is, or the Wi-Fi access point or router at the other end is basically not very, it's deprioritizing ICMP like crazy and probably is not even wanting to respond to us. That's why we see that there. But these little two highlights here, my last point before I hand this over to Lai, there's a couple of speed checks that went on here and they have almost no impact to the connection. It's like we can run them all the time. We run them on a scheduled basis, again, to Cloudflare to give you that baseline to know if something went wrong. That's how we score your connection. With that all said, I'm going to hand this over to Lai, and I can come back to the question later, but yeah, orb .net, check it out, and I'm not here to sell, it's all free, so. Thank you. Let's go to the next one. Yeah. Okay. There we go. So Doug mentioned a couple of times, orb.net uses Cloudflare, so I'll talk a bit about ... Sorry. Oh my God, the end of the world, okay. So how does Cloudflare play into this, but also just how, you know, Doug also talked about sort of how Ookla informed the journey at orb, and I'm going to talk about how mLab and just speed test measurement in general has informed how I've developed products at Cloudflare. So, you know, the definition of speed is actually a really hard thing to define. And Doug talked about how, you know, I certainly think that the use of it has been largely driven by the marketing sort of potential from it and the way that it's caught on. It's really intuitive also, it makes sense, right? Like when you're trying to load a page, speed is what comes to mind. But that use of that term, I also attribute to that the actual act of figuring out how to measure network quality is actually a really difficult problem to solve. So, you know, when you're thinking about how to define network quality and define a speed test or define any sort of Internet measurement, a lot of what Doug talked through and what orb is thinking about is the actual methodology. So how are you pulling the metrics? How are you calculating from them? How are you aggregating them? What are you showing the user? And I think of this as a measurement metrics themselves. But there's also the question of where are the vantage points? So when you're measuring end user quality, that's typically pretty easy. On the client side, you're going to be measuring from a mobile device, a desktop, a laptop. But where is it measuring to? This gets into a really existential question when you really think about it, which is what is the Internet? I want to measure my Internet connection. Where is that? What do I use to represent that? So various speed tests make different choices about how to answer this. Ookla runs an infrastructure largely either in or close to access networks, meaning the ISPs that you and I subscribe to. Measurement Lab answered this by putting servers in interconnection points. And the idea there is to measure the degree to which peering impacts the end user performance. But the point being that no matter what, as a speed test provider or anyone sort of devising a speed test measurement, you have to think about what it is you're measuring to and how representative that is of the user experience. So this is something that I thought about a lot at Measurement Lab. Again, we measured from the interconnection point, which was a really informative measurement for what we were trying to measure. But when thinking about what is the most representative of end user performance, and I'm not just saying this, even before I joined, I would often think about measuring to CDNs and specifically Cloudflare. You know, Cloudflare represents 20% of the Internet or facilitates 20% of the world's traffic. So in many ways, connecting to Cloudflare, you can think of as a proxy for connecting to the Internet, or at least that's my premise or proposition. So when I'm thinking about what is the most representative way to measure end user connectivity, measuring to Cloudflare, a lot of the time that is what you're actually accessing as a user. So measuring to Cloudflare can be a proxy for that experience of connecting to the Internet. And so this is the approach that speed .Cloudflare takes. And you know, when I go to speed.Cloudflare.com, we can run a test, you are just connecting essentially to the closest Cloudflare POP using our Anycast network. And you know, we run the typical speed test measurements. Our particular methodology is running basically increasingly larger sizes of data. Ah, you have the reverse scroll and that really messes me up. And so we also then use latency measurements. So idle and loaded latency, the difference being, are you actively doing something with your connection or not? So you know, the latency can really differ depending on what your connection is trying to do. So you measure both, jitter, etc. This speed test itself is all done on the client side. So it's all client side measurements. This is a browser essentially doing these calculations. And this is open source and will remain so. This is our open source speed test that you can look at on GitHub, look at the calculations. And then the last bit that I'll mention too is that we have a network quality score that basically translates these metrics into use cases. So how does my connection work for streaming? How well does it work for gaming and video calls, which are things that people are typically trying to understand when they take a speed test. So this is all, again, open source and going to continue to be. But the thing I want to point out is going back to that question of, you know, what is the Internet and what are we going to use to represent it? The way that speed .Cloudflare does this is it tests again to the Cloudflare network. And so over the years, this was something that I really wished I could do as another speed test provider. And what I'm excited to share today is that we're making that possible. So introducing Cloudflare's network quality API. So we're introducing this today in private beta. And what essentially that means is that any speed test can now use Cloudflare's entire network to facilitate their test on the server side. So again, legally obligated to show this slide in a Cloudflare presentation. And but, you know, the point being that you get access to this entire global network when you're using Cloudflare, any Cloudflare product, but specifically the network quality API. So you when I, you know, am wanting to create a new speed test and or use someone else's but run on devices, and I have then the question of, okay, but where am I going to test to? Now Cloudflare is an option. So we'll talk a bit about how it works and just the features on, you know, one of the core features is that you can measure network quality any way you want. So Orb has a solution, Cloudflare has our solution. But the something that I always come back to in Internet measurement work is I think the more methodologies, the better. Of course, you want to be able to compare over time and, you know, between different data collections. So, you know, longitudinal data is helpful for that standardization is helpful for that. But I do think that when you're thinking about how to measure network quality, having different measurements from different ways of thinking about from different definitions can be really useful. So the premise and what we're trying to create here, again, largely based off of my previous experience is thinking about, you know, when you have a speed test or when you want to use a speed testing or networking network quality product, often you will have to use both their client and their server. This is separating the two and meaning that you can test to Cloudflare as your server, but test in any way that you want. So you could also, so you can test in any way that you want, and you can also test in multiple ways. And all of this would just use our API to facilitate the server side of the measurements. Usage tracking. So now we make it really easy to, I don't have to walk over here, we can track in Dash. So you can track your API or your, you know, API usage directly in Dash alongside all of your other Cloudflare products. And, you know, you can look at the number of bytes transferred. And so, you know, in the Orb case, looking at all of the different Orb devices and how much they're transferring data, you know, egress and ingress, and then also where is the traffic coming from and how many requests. All of this makes it really easy, essentially, to facilitate this kind of measurement at scale, which is the other sort of core feature. You know, again, if I today just wanted to go ahead and set up my own speed testing platform, I would have to make that choice again about where I'm measuring to. But even say, let's say I figure that out, I would, if not using something like this Cloudflare API, I would have to maintain all that infrastructure myself. This is a really big challenge in the Internet measurement community and academic research is, you know, you have a question, you have a thing you want to measure, but where are you going to measure to and how are you going to maintain all of those servers? So we're trying to make that as easy as possible. And again, doing this at scale. So, you know, being able to test around the entire world and all of these different locations, that's now just done for you instead of having to, you know, sort of build up a platform one by one as I previously did. And then the last core feature I would say is just to really drive this home, that when you are measuring to our or using our API, you're essentially measuring to Cloudflare. And that's, I think, the biggest kind of feature is that you're able to measure to this entire representative, entire network that is representative of user experience. So how does it work? You know, fairly simple at the end of the day of really what you're doing is you're getting an API token and then with that token, you will be able to give your users. So, you know, in Orb's case, so let's say that Doug's apartment in Chelsea wants to request a speed test, then that will get a short-lived token that will then authorize that client or that device to test to Cloudflare. And then using that, it will measure the connection. Potential users were thinking this will be really helpful network quality monitoring services, but also potentially mesh network device providers. And then a lot of my background comes from measuring for digital inclusion and broadband mapping purposes. So, you know, trying to understand the landscape of who has proper connectivity and who doesn't. So I think this solution could be helpful for any of those use cases, but also really wondering who else. So if this is, you know, something that is of interest and you're not kind of in any of these categories, I would love to know about that. And, you know, just to say, why do we care? Why is Cloudflare interested in this? Why is this something that we're pursuing? The agentic Internet, as we know, I've been talking about is here and it only makes Internet connectivity that much more important. Only more and more of our lives are becoming dependent on this critical infrastructure. And so, you know, with that, then we have to think about, OK, who are then we helping advance into that brave new world and who are we leaving behind? And, you know, I cite the FCC here because it's widely known to be a very optimistic measurement and is also neither one of our former employers. But according to the FCC, 10.6 million Americans lack access to even 100 upload or 100 download, 20 upload. And so, you know, when marketing plans are pushing us to bigger and bigger plans, 1G, 5G, 10G, and there's still this critical mass of people in this country that are not able to access even the bare minimum to do the things that they need to work remotely, go to school online, etc., then I think these types of measurements are that much more critical to understand where the gaps are and what it is to really have 100 percent connectivity, not only in this country, but elsewhere. And so we want to make it possible to get those measurements and to understand the landscape of Internet quality, you know, in all regions by supporting customers like Orb and other providers of these measurements so that we can truly measure network quality meaningfully at scale. So just a preview of what's on our API roadmap. We're launching today in private beta. And again, I would love to know if you're interested. I'll have my contact in a second. But what's coming down the roadmap is being able to access finer grained server side metrics. So right now, what the API is primarily returning is just timing information, which is, you know, more or less what you need to run a network quality test. But thinking about what are the metrics on the lower level kernel side, so TCP info, we could imagine congestion control metrics as well. And so giving more fine grained access to that on the server side is something that we're interested in. Enable measurements of higher throughputs, because even though we said everything we just said, people still will always ask, you know, can I measure up to 1G? Can I measure up to 10G? And we want to be able to say, yes, talk about all the things we all really align on that. That's not the most important thing to measure. But we want to ensure that Cloudflare is never the bottleneck. And then finally, we're interested also in being able to analyze and monitor your user results inside of Dash. So basically creating an interface that's as informative to you as possible about all of your different users and the people using your measurements without kind of, you know, having to go outside of the Dash interface. So with that, thank you so much.