🔬 Two Years of Cloudflare Research

Awesome. It's 9am and welcome to Cloudflare TV. My name is Wesley. I'm the Product Manager for the Research Team at Cloudflare.

Joining me today is Nick Sullivan, Cloudflare's Director of Research, and welcome to the Research Blog Takeover Week, where the Research Team here at Cloudflare is going to be announcing a whole slew of stuff all week long.

Starting off yesterday, in fact, Nick actually posted our two-year update blog.

Nick, welcome to the show. Thanks, Wesley.

Cool. So we're going to talk about the two years of research. So why don't we just do a little bit of backgrounds first.

Why don't you introduce yourself and I'll get my quick background.

Yeah, so I'm Nick. I lead the Cloudflare Research Team. Cloudflare Research was launched two years ago to help pursue some long-term goals around computer science research that Cloudflare has.

As a growing company that deals with increasingly complex challenges from the technical side as well as the policy side and the customer side, as Cloudflare grows, we need to build deeper expertise in lots of different areas.

And Cloudflare Research is here to go deep on the technology questions that are three to five years from reaching scale or new or particularly challenging or ones for which we can connect with folks from academia or folks from the rest of the industry to help create new standards to make the Internet better as we go forward.

Because Cloudflare's mission is to help build a better Internet.

And to do that, we have to do it together. I know. I love the mission so much.

And I love how research helps drive the mission so much here at Cloudflare.

For context, hi, I'm Wesley. I'm the Product Manager for the Research Team here at Cloudflare.

Broadly speaking, I work with Nick, the rest of our Research Team leads, to help figure out how we're taking our innovations in fundamental computer science, next-generation Internet protocols, privacy technology, post -quantum cryptography, the distributed web, and turn it into usable bits and bobs that the engineering ETI teams can all pick up and deploy and run at scale.

So research and innovation go hand-in-hand here at Cloudflare, and there's a really big innovation pipeline.

I'm also teasing my blog post later this week that you should go read.

It's going to be really interesting to talk about how it's not just computer science and research here, but it's really fundamental applied research and how we take things forward.

And I think that's what Nick and I want to talk a little bit about today too.

Before we jump into the principles that we've been working on for a while, Nick, why don't you tell us about how we even decided to get into research?

I know you mentioned growing bigger network, increasing customers, but the crypto team was around at Cloudflare for a long time before it was the research team.

Yeah, that's right. So Cloudflare had a cryptography team for a long time.

So I joined Cloudflare over eight years ago. I started as the first security engineer, founded the security engineering team, as well as from that team spun out a specialized team focused on cryptography.

And so the reason that a SaaS company, that's building a cloud service, would need a cryptography team is that we were innovating in very deep ways around the cryptography that helps secure the Internet.

And so Cloudflare launched Universal SSL back in 20...

It was actually a birthday week. Cloudflare has a birthday every year and launches a lot of things back in 2014.

And Universal SSL allowed every single one of Cloudflare's customers, even the ones who weren't paying, to get encryption via HTTPS for their websites for free.

And to get that to work at scale back then when most of the Internet was actually not using HTTPS, if you can believe it.

Right now we're somewhere around 90% of sites that use it, but back then it was a luxury.

It was something that was difficult to do. We had to make some interesting kind of fundamental choices that were not built into the open source tools that existed at the time.

And ones that required kind of deeper engineering thought as well as research.

And so as we innovated in that space, as we started building new products and services that relied on cryptography, we needed a team that was able to go deep into it and to sort of figure out what was going on next.

So the cryptography team did a number of interesting projects in this space that a engineering team might not have the wherewithal to do or the very kind of long-term projects that took a long time to do in a way that it wasn't necessarily immediately impactful on the business, but five years down the line was very impactful in the business.

So the cryptography team tackled a couple different things, one of which was getting involved in the development of the TLS encryption standard.

This is the protocol that makes HTTPS websites secure, adds the little lock icon.

And at the time TLS 1.2 was the state of the art. This is what all the different websites and web services were using.

Well, a lot of them were still using TLS 1.0 at the time, but with TLS 1 .2, there were certain fixes that helped make the Internet more secure.

And at the time academics throughout industry, this was right after the Edward Snowden revelations decided, hey, this might be an area of interesting research.

This protocol was developed by the standards community and it wasn't created closely in conjunction with the academic community.

And so you ended up seeing that TLS 1.2 actually had quite a few flaws.

And these flaws were turned into papers published at venues like USENIX, like the Oakland conference, IEEE security and privacy.

And there were half a dozen to a tax against TLS.

So at the time the IETF decided, okay, well let's try to address these issues and let's try to make TLS stronger.

So TLS 1.3 was something that was starting to be developed.

So one thing Cloudflare decided to do at the time with the cryptography team is get involved with TLS 1.3.

And so we went to several IETF meetings, helped contribute to the standard inspect, were the first major service to deploy TLS 1.3, a beta version for all of our customers for free, and helped make sure that TLS 1.3 got deployed.

And this took upwards of five years to get to the point where TLS 1.3 is close to the majority of all encryption technologies used online.

And we explored a couple other things like certificate transparency, which is an effort to help make this system that encrypts websites and web services more secure even further by providing an auditing system for the certificates that help guarantee that a site is who they say they are.

So certificate transparency is something that requires a bunch of different parties to work together and provide these shared databases that you can use to see what's going on.

And if a certificate authority, the companies that issue these certificates, misbehave, you have this public log.

And folks can watch these logs and gossip about what certificates they see and be able to affect change and hold this system of trust, which really underpins the entire Internet, accountable.

And so with these two projects and several of the other underlying research things we did, including this research project with folks from University of Michigan about detecting middle boxes, it became really clear that this model was a valuable one for the company to pursue.

The model of jumping in on projects early, several years before they're going to affect the Internet at scale.

And rather than be kind of pushed in whatever direction, the folks who study these technologies or help develop these technologies, we could get in on the ground floor and use our deep expertise and things like cryptography and protocols and networking in order to understand and honestly, to help fix some of the problems that other people developing these protocols might not have seen because they may not have the type of network that Cleffler has, which is a massive global network.

So we see so much and our data feeds into our understanding of the technology.

And this is something that turned out to really be helpful with TLS 1.3 and with certificate transparency.

And so two years ago, we had this small cryptography team that was focused on long -term research, and we effectively expanded the scope of what the team could do from cryptography to a lot of different things, whether it's distributed systems, whether it was privacy technology.

Privacy is a really big and important part of the Internet going forward and something that if you think of security in the early days of the Internet, it wasn't built to be secure and it definitely wasn't built to be private.

And so as an industry, we've been fixing security issues and now we're fixing privacy issues.

So a lot of these areas kind of came under the bucket of what Cleffler research was scoped to do.

And so for the last two years, we've been growing the team.

We've been hiring really great specialists.

We've been figuring out how to work more closely with different groups in academia, different standards bodies, and how best to work with other groups within Cloudflare.

And so how to take some of the fundamental technologies that we're helping build and understand and integrate that knowledge with the rest of the company.

And this is sort of Wesley, where you come in and take a lot of the pieces that we're working on that are either product-shaped or near product -shaped or relating to different Cloudflare products and make sure that our expertise helps make the company as it is right now, even stronger.

Totally. I think what's so great about what you just said, Nick, is that there's this really great long arc of sort of three factors that come together.

One is this idea that research takes time and building the Internet actually takes time, right?

It's not just like we ship one piece of code and suddenly everything changes.

You have to work with our partners, right?

We're helping to build a better Internet. We're not building a better Internet, right?

This is a community operation. The second thing too is that network scale, right?

We talk about TLS 1.3, but we can look at when Cloudflare got involved in 1.3 and we can see the ramp period because of our network scale and how much faster 1.3 got deployed because we were involved, right?

That's a really big contribution point.

I think that final network scale piece that ties into this third track that I think is really interesting is that we have this innovation pipeline and that it's not just we're working on stuff in small contained boxes.

We're taking this long arc of time, this ability to work at network scale, and then also combining with the fact that we can work with everyone inside Cloudflare and all of our academic partners and all of our standards party partners and all of our industry partners.

We can all go together to help build something greater than the sum of its parts.

And I think we've looked at the last two years of work and the body of work that we're going to be announcing this week.

And I think you did a really great job along with the rest of us at helping distill these down to like really strong operating principles.

I sort of want to go through those with you because I think they're really key to what we are actually doing here.

And we can talk about the case studies too, but let's start with the principles, right?

We'll just go with number one.

It's like innovation comes from all places. How do we get to that?

Yeah. So I think innovation is kind of a fuzzy word, right? People use it to mean a lot of different things, but fundamentally it's about bringing new things into the world that make a dent, that really change things.

And as a company and or as an organization, it's really difficult to be innovative if you don't actually take advantage of the fact that everybody at your company has different expertise and everybody has a different role and everybody has a different perspective that they can bring to the table.

And so our idea of innovation comes from all places as a principle for the company is exemplified by the fact that there's not just one innovation team at Cloudflare.

There's not like a group of people in the corner who are coming up with like brand new things and then they just hand it off to someone else to run with.

Innovation is about new ideas and it's about curiosity and it's about an entire culture.

So when innovation comes from all places, Cloudflare Research's position here and role is to help facilitate the innovation that happens throughout the whole company.

And so working with different teams, whether it's customer support, whether it's user experience, whether it's some of the deeper networking teams or some of the user-facing teams or customer-facing teams, there can be good ideas that come from everywhere.

And these ideas have to be fostered. And if it's something that we think could potentially be a big deal in three to five years or could relate to a challenge the company's going to face, Cloudflare Research is there to help facilitate the development of these ideas and for Cloudflare to continue to be an innovative company.

And Cloudflare is an extremely innovative company. I think if you follow what we've been doing and the pace of products that we've been launching over the last several years, it is mind -blowing how fast Cloudflare is putting out new ideas and really changing the world.

And part of this has to do with the fact that it's all of us working together.

Yeah, no, I love that. And not just in the sense of like, oh my God, it's so nice to work with all your coworkers, but it really is true that it's a team-based approach.

You don't get to this level of innovation, this level of speed.

And I think the speed is such a word. You know, just think about this.

We've had so many innovation weeks so far this year between developer week and birthday week and privacy and compliance week earlier last year.

I mean, they don't stop. And we have so many more coming in the can too, just a constant drumbeat of innovation.

And it's so, what's rewarding for me as a product manager is the fact that I get to work with so many different teams and see so many different problems and help so many different people.

You know, I love being able to go to product and engineering, going to ETI, going to other groups and helping them solve their problems and vice versa.

It's a really great collaborative environment.

And it leads to the next principle that's personally my favorite on the list, which is, you know, a question -oriented approach leads to better products.

As someone, I have a very distinct memory when I was an undergrad of my favorite professor sort of sitting me down and being like, I think it was an ethnographic methods class actually, just like, look, we're going to talk methods all semester.

We're going to do different types of writing styles or types of interview styles.

The most important thing you're going to get from this class is how to ask an important research question.

That is going to be the departure point for all your work, whether it's here at Hampshire or else elsewhere out in the world.

And I think it's so key to the work that we do here, because it's not like you and I sit down with a nice shiny 24 month roadmap and say, we're going to build X, Y, and Z thing on X, Y, and Z dates.

You know, I think we're much more driven by the questions that are both interesting to us and interesting to Cloudflare and that we think are going to have really big long -term impacts about the world, about, you know, what's the nature of post-quantum cryptography?

How is that going to be different in 10 years?

What's the nature of privacy and security technology? How is the world changing?

You know, I think what's so fascinating is that we work so closely with policy, legal, product leadership to try to ask and answer a lot of these things.

And it really influences our roadmap, right? Yeah, absolutely. And if you are going to be building products, there's several approaches you can take.

One is let's look at what's out on the market and see what we can kind of take from this and learn from it.

And we do have a lot of people at the company that are working on this and Cloudflare's products innovate in so many different ways in terms of cost, in terms of usability, in terms of developer usability, as well as the fact that from a cost structure, it's very appealing to use Cloudflare.

There's a free tier, there's ways that you can get familiar with our products while moving up the scale.

And so, some of the ideas around, okay, there are businesses that have known problems.

We have excellent, brilliant minds working on this.

But in order to keep the long -term train of where the company's going within the context of a rapidly evolving technology landscape, sometimes you have to ask the questions of what do we not know?

We have the knowns, we know what products we have and what our customers' needs are, but in five years, what will the customers' needs be?

What sort of changes on the Internet will exist? And so, by structuring the team and really focusing on asking good long-term questions, we can come up with completely out-of-the-box answers.

And those answers might not end up being the long-term trend of where Internet technology is going, but sometimes it can be.

And given the right resources and the right leadership and the right amount of critical thought and analysis, we can help shape the industry through asking these questions.

You mentioned post-quantum cryptography. This is something that is an unknown.

It's something that could happen within two years.

It could happen in 20 years. Some people say it might not happen at all. But by asking the question of what is the impact of this, then we actually have looked at every single one of our systems to see where the cryptography lives.

Where are the weak points?

What security risks do we have in some of the systems that we're currently using that we maybe didn't think of looking into before?

And so, there's all these secondary effects that come from taking an approach that is focused on long-term questions and applying it to what the company looks like today and what the products look like today and what the world looks like today and comparing it with potential futures.

Totally. And I think what's so interesting too is that we really have the motto, at least in my opinion, to know is not enough.

We actually go out and build the tooling to do a lot of these things. It reads really nice in the next principle, which is build the tools today to deal with the issues of tomorrow.

I think so much of what we think through this implied word, it's applied research.

We actually build stuff. Yeah, that's right. And it's great to write a paper or to have a theoretical analysis or some type of thought leadership document or thesis about where a certain industry or technology is going to go.

But just like with anything that's an abstract description, the real world is much more detailed and much more complicated than any idealized version that you can put together.

And the only way to really understand where theory hits reality is to build, is to actually put things out there in the world and see what happens.

And Kloeffler's in an amazingly privileged position here because a large percentage of the Internet's traffic goes through our network.

And so what people do online is affected by what we do. And so if we want to experiment with things like what will a post-quantum cryptography world look like, we can change our code.

We can change some of our services and have millions of people living in the future, in a potential future.

And we can see the downstream effects of making these changes.

And from an abstract perspective, thinking, okay, well, we have one cryptography protocol and we want to replace it with another cryptography protocol, that should be fine.

And this is another lesson that we helped, well, that we learned with TLS 1.3 is that any change to the Internet is not as simple as you think it's going to be.

The participants of the Internet, like in whole, you think of the Internet as there's all these networks that are connected to each other.

And every person can connect to any service and people can chat with each other on a peer-to-peer basis.

But all of that evenness of the connectivity of the Internet relies on the fact that every single network that's connected to the Internet applies the same rules and does things in the same way.

And so there are technologies like security technology boxes that do network inspection.

There are different types of routers, different types of switches, different types of computers really that are hooked up to the network that affect how traffic goes around the world.

So if we're able to take a technology like post-quantum cryptography or a new encryption protocol and put it out there, we get to see where it breaks.

What are the underlying assumptions that folks are making technologically about the future of the Internet?

Is everyone going to assume that we're using the Internet? I mean, we've experimented with a lot of different technologies where we think that there's like an IP address functions in one way.

And people have made underlying assumptions about the function of that over time.

And we've seen how that's skewed from some of the original specs and RFCs.

It's also really interesting. To tie that all back, it goes into the going forward together principle, right?

I mean, it's not just us.

It's working with all of our standards partners, all our community partners, and also contributing to the open source.

I think there's a really big culture here at Cloudflare Research of contributing to the open source and making sure that it's not just as much as we're an industry lab, we're also an academic lab, and we're also a public contribution lab.

And we want to give back in all of those areas.

Yeah, so any change that we make to that's going to affect the Internet is actually one that we can't make alone, speaking long term.

Because as there are different networks that connect to each other on the kind of lower layer of the Internet that provide the substrate for all these brilliant multimedia applications and communication tools that are built on top of it, if both sides of a communication tunnel are using different pieces of software, then those two pieces of software need to interoperate.

They need to communicate in a way that one person's speaking French, one person's speaking German.

If you don't have a translation layer, you really are going to be speaking past each other.

And so Cloudflare can't unilaterally make changes to the Internet, because for the most part, we are protecting and accelerating and securing servers and the server side of the equation.

So we need to work with browsers, and what the languages and the protocols that browsers speak, we need to speak as well.

And if we want to evolve these, we have to evolve these together with all the other participants that make up the Internet.

And it's impossible to do that without open standards and even open source software.

Contributing back our version of different standards is a really good way for different pieces of software that are sort of run by smaller groups or individuals to be able to take these new advancements and integrate them into their products.

So one great example there is we've been working on this new protocol called Oblivious DNS over HTTPS.

We open sourced all of our code there. We have code that works in Cloudflare Workers.

We have Rust code. We have Go code. And we've worked with other very well-known community projects that do DNS to make sure that their version of ODO is compatible with ours.

And so it's important for us to not go alone or not go too deep in one direction without actually first having a touchpoint with other partners on the Internet.

And then secondly, as you said, open source, open standards move the Internet forward.

We'll just hit the final point too.

I think what's so nice about contributing to the open source, it's not just we contribute to the open source.

It's that when we contribute to the open source, we have a general sense that this code is going to work because we run things at network scale.

And that sort of ties into that final principle of we can do things at network scale.

So it's both behaving responsibly at network scale, but also because we can test things at network scale, we can move faster.

I think that's really, really important that we go from theory to practice at a scale that's very different than many people have access to.

Yeah, that's right.

And we can do a number of positive experiments at the same time. If we're looking for ways to try new technology that could make the Internet faster.

So if you think of the difference between TLS 1.2 and 1.3 for performance is there's one fewer round trip.

So latency for loading website is going to be faster. And so if we're trying new technologies, we can share it with a subset of our free customers really early and they'll get these advanced features before they're even live.

And we'll get to see how much better the Internet can be made for them. So we can do a lot of these kind of AB tests with this fantastic group of customers that use Cloudflare services and benefit from all the things that we give them.

And they also get to be participants in these wonderful experiments that help give us the understanding of how the Internet actually works.

And so Cloudflare having this whole scope of types of customers from free customers all the way up to the largest enterprises in the world, it gives us such a great opportunity because free customers can get these advanced features.

And then sometimes our really high-end customers might want to be first movers for different things like protocols.

So one of the things that we helped put together and we'll be talking about this week is a compromised credential checking service.

So it's a service that knows out there which passwords have been breached from which services and can check and tell you, hey, your password and username combination have been compromised out there somewhere.

So some attacker may actually try to log in as you.

And so this is a service that we helped build and we worked with other teams within the company to build it into Cloudflare's WAF product.

And this is something that is being trialed out by some of the really large e -commerce retailers out there.

And because they're so huge there and they're willing to trial something new with us, we can learn something new about the world and make it known to the world what's novel about this and how best to build this type of security mechanism going forward.

Which is awesome. You're on the last minute too, Nick.

I want to plug a couple final things for all of our viewers. First is keep an eye on the blog this week.

Research team is doing a whole blog takeover this week. We're going to be dropping a lot of really amazing content for you to read.

The second is that as of 30 minutes ago, our visiting researcher program as well as our summer 2022 internships are now live.

Go check those out. We're very excited to constantly be bringing in new talent and new ideas.

It's the idea that innovation comes from everywhere.

And the third thing is that research.Cloudflare.com is now live as well.

Go check that out to go get really in-depth papers and articles on what we've been working on.

Nick, it's been such a pleasure talking with you today.

I'm really excited about what research is working on and what Cloudflare is doing in the world.

With that, thank you.