🔬 SSL Recommender

Hi everyone and welcome to Cloudflare TV. My name is Wesley. I'm the product manager for the research team here at Cloudflare and we are so excited to have you joining us today as we talk about day two of research mini week here at Cloudflare. We're very excited today to be joined by Valenta and Suleman Ahmad, two of our research engineers here on the research team at Cloudflare. And today we're going to be talking about encryption and how we keep the web secure and what we're doing today to really actually help spread encryption technology to so many more people. Guys, thank you so much for joining me today. Oh, thanks for, thanks for having us, Wesley. Cool. So I feel like our viewers already know who I am, but I'll say it again. My name is Wesley. I'm the PM for the research team here at Cloudflare. Broadly, I cover a really wide range of subjects. So post-quantum cryptography, next generation Aranet standards, next generation encryption technology, as well as the distributed web. And let's go around the horn here and have everybody introduce themselves. Suleman, why don't we start with you? Sure. Thanks, Wesley. So hey everyone, I'm Suleman from the research team. I work as a research engineer, mostly working at the interception of security, privacy, and intermediated projects. Last summer, I worked with the research team as an intern and now I joined full-time and happy to be back to work alongside these incredible people and delighted to be here at Cloudflare TV. Awesome. Luke. Hi, I'm Luke. I'm a systems engineer on the research team and I also helped to develop this product. I was, I guess, supervising Suleman as an intern working on this project last summer, actually. And so happy to, happy to see this, this product keep, keep moving along. Awesome. So let's actually dive into what we're talking about today, which is the SSL recommender. So Luke, why don't you give us a sense of what the SSL recommender actually is and what problem does it solve for us? Yeah. So, so today we're, we're talking about the SSL TLS recommender. It's a new, it's actually a toggle on the Cloudflare dashboard that customers can enable for us to give them smart recommendations for which encryption mode their zone is optimally configured on. So this with this service, we hope to allow customers to improve the security of their zones without having with minimal effort on their part. So we have, we have an accompanying blog posts that you can see the link in the video description to, to go read the blog posts. We encourage you to read along with this segment and ask any questions in the, in the chat and we'll respond at the end if there's time. So the, the recommender has, I guess, as of August of last year, we, we launched it in beta after having built it as in a summer internship. And and then as of now, it's, it was a beta for only self-serve customers. So we hadn't rolled it out to enterprise customers yet. And we have over 500,000 customers signed up so far to get these recommendations. And as of today, it's now available for all customers, enterprise included. That's so awesome. I mean, you know, over half a million websites upgrading their security to be more secure, encrypt user data, and just create a better security environment on the Internet. I mean, Suleman, talk to me about why encryption is important. You know, what does it do for us? What's the end user benefit? Why does this matter to our zones and our customers? Yeah, sure. Thanks for asking, Wesley. So like almost every one of us uses a device connected to the Internet on a daily basis. Whether it's a laptops or mobile phones or any, even any of the smart gadgets, like even a fitness watch, they all manage some aspect of our personal information that flows to the wider web. Many times the data is very personal and confidential. It could be a banking transaction made to online banking or entering that password for logging into one of our social media platforms. All of them require the central information to be transmitted and communicated to some server over the Internet. Therefore to secure our online activities and protect our information, encryption over the network is not only useful, but it's necessary and it's required. We have seen the conflict examples of cases where lack of encryption has caused sensitive data leakage. Recently, I was reading an article regarding the tool that attackers used to stoop on people, network traffic or using public wifi networks at airports or coffee shops. All of the victims had the network traffic in plain text, which allowed the attacker to easily extract their private data. We have seen reports of governments listening on encrypted net communication channels, getting our last day surveillance, like ISP is getting our content-based filtering, censorship was encrypted data. Like all of these cases are generally one thing in common, which is the lack of encryption over the network traffic. So encryption has just become a vital part to keep us and our information safe and private. Like we at Cloudflare have offered free encryption, support for any website that uses our services for our goal to help make a better Internet. Like seven years ago, when we gave free HTTPS support or in other words launched the universal SSL certificates, we saw that the number of websites that used HTTPS doubled in number overnight. It was just very exciting. And today at the same motivation and quest to make the Internet better, we are launching SSL DS recommender for our customers, which makes me very happy. Nice. I mean, yeah, this just goes back to something that Nick and I were talking about yesterday, right? Which is the ability to for the research team here at Cloudflare and all of our teams to work at network scale, right? You know, you just mentioned it again, you know, the fact that we were so involved in launching universal SSL for everyone, helping launch TLS 1.3 for everyone. This is, you know, really a next evolution of those projects. You know, let's just get this down to brass tacks a little bit, you know, Luke talked to me about how do we actually secure these connections? You know, what's the magic that's going on inside the box? Yeah. So, so Cloudflare secures these, these connections are able to double the number of sites that using encryption on the Internet overnight because it acts as a reverse proxy. So Cloudflare basically sits in front of origin servers. And so visitors to cut to customer sites, they connect directly to Cloudflare first. And so Cloudflare is able to, to provide TLS for that connection between visitors, such as like browsers to Cloudflare. And then on the backend to actually retrieve the content that they get served to the customers, Cloudflare creates another connection, which we're calling the backend connection to the origin server. So, so there are two connections involved, the front end connection from visitors to Cloudflare and then the backend connection from Cloudflare to the origin. And I guess with yeah, Cloudflare easily encrypts the front end, but in order to encrypt the backend, we have to make sure that the origin server actually supports encryption. And we use the Cloudflare's encryption modes, which are a setting in the Cloudflare dashboard for customers to indicate to Cloudflare, which, what level of support their origin has for encryption. So there are a bunch of different modes available there. Totally. And let's just get, let's drill down something really important here, because, you know, we talked about what a reverse proxy is, right? This idea that we have this front end connection, this backend connection, why is it so important though, to secure the backend connection? You know, if I'm an end user and I'm secured to Cloudflare, does that made my connections secure all the way? Yeah. So, so the largest attack surface area is, is in that front end connection. So like, like Solomon mentioned earlier, an attacker in a wifi network can, can sniff packets on the network and see what, what sites users are connecting to if it's unencrypted. So encrypting the front end connection protects users from that type of attack. However, the backend connection is equally as important. And this, this is the traffic from Cloudflare to the origin server, which would be wherever that's hosted. So if this, if this traffic remains unencrypted, it's just over plain text HTTP, for example then it's it could be vulnerable to surveillance or an attacker that's sitting in the same data center as a victim origin server could potentially hijack traffic that's intended for the origin server and serve malicious traffic back to the user. So it's really important that we're able to also encrypt the backend connection to, to prevent all all sorts of attacks like this. Awesome. And Luke, I know you have a diagram you want to show us. And why don't you get that up? Solomon, why don't you talk to us about all the various modes we have for securing the backend connection, what those look like? Yes, sure. Indeed, an important question. So like customers like have the option to select what encryption mode they want for the backend connection between Cloudflare and their origin servers, as Luke explained, and that encryption mode setting applies to all domains and subdomains under that zone. So thank you Luke for bringing the diagram out. The first three encryption modes, specifically the flexible full and full strict are available to all of our customers. While the strict origin pull mode is only available for our enterprise customers. So going a bit deeper into the details, like flexible mode allows clients to connect to Cloudflare through HTTP, HTTPS, but request to the origin server are always over HTTP, which means that they're not encrypted. This is the most common option for origins that do not get support TLS. However, we encourage customers to always like we, like we encourage customers to upgrade their origins to support TLS whenever possible and only use flexible as a last resort. Then we have the full encryption mode that is shown by the light yellow lines on the diagram, which makes sure that when clients connect to Cloudflare through HTTPS, then the backend connection to the origin server is also on HTTPS. But Cloudflare doesn't attempt to validate the certificate, which means that this option is useful for origins that have a self -signed or otherwise an invalid certificate, but it also leaves open the possibility for an active attacker to impersonate as an origin with a fake certificate. So although for a password, we'll see the traffic, the backend traffic will be encrypted, but for an on-path adversity that can intercept the traffic, the option will not be able to keep your connection safe. This directs me to the full strict mode, which is shown on with the green color on the diagram. This mode allows us to fully secure the connection by also validating the origin certificate. This allows it helps your connections to be safe from both passive and active adversaries. And we highly recommend the full strict or even high encryption mode over the other weaker options. Just a side note at all these modes, for all this mode, like if a client sends a request on HTTP to Cloudflare, then Cloudflare always forward an HTTP request to the origin for the uncast content requests. This brings me to the last strict origin poll, the one shown with blue. That makes sure the connection to the origin servers are always encrypted, always encrypted, no matter what. And this mode is available for our enterprise customers. Thanks. And that makes a lot of sense. I mean, I'm just looking at this diagram here and I have to be honest with you, even as someone who understands it, there's a lot of room for permutations here and for people to build the type of infrastructure they want to build, right. But that can also lead to a lot of different pain points. So Luke, what are some issues that, you know, before we built this product that we were encountering that our customers are running into? Yeah. So, um, so often, uh, I guess one, yeah, one of the main pain points we see is customer origin sites evolving over time. Um, so, so for example, when I set up my site originally, I just had an HTTP only origin and I put it behind Cloudflare, using flexible mode just so I could have the, see that secure lock icon in the browser and, um, and secure my site. Um, however, then later I, um, I guess my origin evolved. I added a certificate, a valid certificate on my origin site. And, um, with that valid certificate, I, um, I can now upgrade my encryption mode to a stronger setting, such as full or full strict, so that the backend connection is, is also secure. Um, so for many customers, um, they might forget to update the origin setting or update the encryption mode for their, um, their site after setting the origin setting. Um, and I guess another, another possible risk that customers might be hesitant to change from a mode that just works for their site is that configuring a setting that is an encryption mode, that it's, that's too strong for what their origin supports could actually cause origin, um, availability issues. For example, if I have an HTTP only origin, but I choose full mode or full strict mode, then, um, HTTPS requests from clients or a Cloudflare would try to, um, connect to my origin over HTTPS, but the content isn't available over HTTPS. So the content would, um, not be served. Um, so that's, that's, uh, another issue that can, um, potentially cause customers to stay with whatever setting they originally configured it with, um, to avoid, uh, avoid site downtime. Um, and we, yeah, so we, we see this problem that a lot of customers stay on flexible mode because that's, uh, that just works, but, but what we really want to do is push customers to use these more secure settings, um, and, and get away from flexible mode. And, and one other thing I'd like to point out is that while all of these, um, modes forward HTTP traffic from visitors as HTTP requests to the origin, there's another, uh, configuration option that Cloudflare provides called always use HTTPS that actually causes these HTTPS connections or these HTTP connections to be redirected to an HTTPS version of the site. So you can complete using full strict mode in combination with always use HTTPS. You can eliminate all of the, um, HTTP to origin connections and just have this, um, secure HTTPS connection to your, to your origin. So full strict is obviously where we want as many of our zones or even strict SSL pull only where we want people to end up. So, so why don't you talk to us about what did we actually build to help enable that? You know, because we sat down with a bunch of data and we looked at our existing customer set and we're like, Oh wow, we have a lot of people here that might be able to upgrade. So what do we do? Yeah, absolutely. So we have had the SSL sales recommender like working in beta since an year, and it has helped provide customized recommendations for our customers for upgrading their zones to the best and most secure encryption mode for their sites that their zone is compatible with. The services scans customer origin servers and determine if they can move to a high encryption mode by sending out a recommendation. Um, that recommendation is sent to customers as an email and also visible on the encryption mode options on the dashboard. Um, as Luke just mentioned about the different pain points, like this service actually allows you to see if your zone can be upgraded and allows this messy configuration options easy for you and updates you. If you can easily be, uh, actually upgrade your zone to a higher SSL mode. So awesome. Yeah. Luke, I am so excited because unlike a lot of things that we've built as the research team, which are super theoretical, they only exist in bits and bytes, they run on the back end and you can't actually play with them. We can actually play with this. So I'd love to see a demo. Absolutely. All right. Always fun to do a live demo, but this one should hopefully work, work fine. So here's a, I've just logged into the Cloudflare dashboard. This is my personal, um, my personal zone. Um, so, so go ahead and, and click that. Um, I'm going to walk you through how to enable the recommender for, for your zone. So feel free to follow along if you want to log into your zone as well. Um, so it, the, the setting is under the SSL TLS tab, so we can go ahead and navigate there. Um, and now you see this, um, uh, this, this panel, it says my current encryption mode is flexible for my site. Um, so it only encrypts traffic between the browser and the origin. And here is the, um, SSL TLS recommender that you should find. And if you log in, uh, to the dashboard as well, let's go ahead and flip that to on. Now the, um, the, the, this will enable our, our service in the, in the background to, to start scanning your zone. We don't scan zones unless they're opted in to this service, uh, to avoid, uh, extra origin or extra traffic to their origin. But if you've opted into the service, then within, um, as little as 15 minutes, but it could, it can possibly be a few days. Um, you'll, if we have a recommendation for you, then you'll receive an email. Um, so this is, this is an email that I got. Um, so it says, uh, your, your zone is currently on flexible mode, but would benefit from the additional security provided by full mode. Um, so we're very conservative in giving out recommendations. Um, so we, we try to err on the side of not giving a recommendation if we're not sure about a certain setting on his own. Um, but the fact that I got this email means that I should be confident in going ahead and upgrading my zone to full mode. And the recommender doesn't, it's just a recommender. It doesn't actually change your mode for you. So I have to go back to the dashboard and now I can select full mode for my site. So now I'm in, I'm, uh, encrypting traffic from Cloudflare to my origin server. Um, and this also, this also, um, the recommender said full mode, not full strict because I don't actually have a valid, uh, Cloudflare trusted certificate on my origin for, for my zone, um, as currently configured. Um, so, so go ahead and give this a try. And, um, yeah, love to love to see what, um, what experiences you have with the recommender. Sure. So what I love about this too, is that we actually have the original team that helped build this. So Suleman, why don't you give us the technical details as one of the lead people who are actually in the code building this thing from the ground up? How's this thing work? Yes. And thank you Luke for the great demo. So recommender uses your current zone settings as the baseline for expected site functionality and perform the series of checks to determine if an upgrade is possible. So if your zone is non -functional, as Luke mentioned on the current encryption mode that you've configured, the service will not be able to perform as checks. So therefore recommender is not intended to resolve issues with website or domain functionality, but rather maximize your zone security when possible. So coming back to the point, how the checks actually work, uh, that's first tackle the easier part. When determining if a site can be upgraded from full to a higher mode, like full strict, the service carries out a simple few tests of validity of the DLS certificate at the origin or even multiple certificates. If there are multiple origin servers for your zone, this is enough to determine if the origin has a valid or invalid certificate, which can be used to see if an upgrade from full encryption mode is possible. Coming back to the more complex part are the checks for determining if a zone can be upgraded from flexible encryption mode to a higher mode. Remember that for flexible mode, the backend connection between Cloudflare and the origin servers are always in HTTP, which is unencrypted. Therefore the service needs to be very sure that the origin can fully support HTTPS. So in a sense, it carries out checks similar to upgrading an HTTP site to support HTTPS. The service works by crawling a customer's zone and fetches a set of a set of internal links by making item port and get requests. So the server state doesn't changes. Then based on the content of the link, it runs a similarity check algorithm to compare the content over both HTTP and HTTPS version of the site. All the scanning requests are generated from the service and are distributed from our edge network. Awesome. What were the challenges, you know, that you talked about the easy part, let's talk about the hard stuff. Yes. Good question. So to be honest, the problem is non-trivial and therefore the solution gets a bit complex. Suppose we don't do this content similarity checks and don't fetch the content and just probe the origin server for DLS support. If a customer moves to a higher encryption mode, their clients or visitors can actually end up seeing a different version of the website over HTTP as compared to the version of website over HTTPS. This can be very troublesome and I wouldn't really want that to happen at all from our own side. So the service needs to be very smart about whether a new recommendation should be generated or not. So Luke. Yes. So, so just to follow up on that. So here in the, in the blog post, we just have some screenshots of, of one site that they actually has, if you access the same URL, but change the scheme from HTTP to HTTPS, you get different site content. So in this case, a user looking at the site could probably tell the difference, say, okay, I know this is the valid version of the content. I need to be on HTTPS, but with our scans, we're, we're very conservative. So if we saw this this behavior on an origin zone, we would default to just not giving an upgrade recommendation in this case, because we want to, we want to allow a site or a content on, on sites to be I guess, we want that decision to fall to the domain owner in that case. So there's a link here in the blog post to this research paper that actually talks about these content differences in a lot more detail. And the, the lead author on this paper, Tala Peracha, actually did an internship with Cloudflare last summer and helped us to implement all of this in the recommender. So, so big thanks for to Tala for, for helping make this, this possible. And we encourage you to take a look at the research paper for all of the more, the more in-depth analysis of how how and why site content can be different over HTTP versus HTTPS. That's so interesting, guys. I mean, since we have a little bit of time, I'd love for you to dig deeper into how the crawler actually works, because I think there's something really interesting there. Yeah, sure. So the service first cross, the customer sites to like internal links and references like for very large sites where it is impractical to scan every link recommender only scripts of crawls, only a subset of links. Whereas for sites where the crawl turns up an instant number of links, we augment our results with a sample of links generated from recent visitor requests to that site. Then using all those links that have been crawled, we download the content of every link over both HTTP and HTTPS by making get requests. And in parallel, the server runs the content similarity algorithm to determine if the content over HTTP and HTTPS is similar by generating a similarity score. Only if the content is similar, the checks are passed and the high encryption mode is determined to be suitable for the site. And since a zone can have multiple subdomains, the recommender make sure that it scans each subdomain under the zone so that even if one subdomain has, can not, it's not compatible with the high SSL mode, we do not recommend the zone to be moved on that mode since that can cause a breakage. So I would highly suggest making sure your, where your site is working on the current permission mode before enabling recommender as it allows you to receive the best possible recommendation. And also side note, like if you have blocked all bots from your origin or have robot or text rules file set specifically recommender user agent, then it would respect that and not scan block bots. That's awesome. That's awesome. Luke, why don't you talk a little bit about what the future of this is? Because I think there's a really, really interesting, this we're not just done yet. And I know it's cliche to say in the tech industry that we're just getting started, but I think there's a really interesting opportunity here long term. Yeah. So, um, so we, we know this is, uh, setting an encryption modes properly has been a pain point for customers for a really long time. Um, and, and so we want to, we want to tackle this and make this as easy as, as easy for customers, as easy as possible for customers to, um, to have the best security benefits on their origin. Um, so some things we're hoping to do in the future are, um, maybe making this setting automatic. So customers can just allow Cloudflare to choose the best way to, to, um, reach their origin. Um, there, there are a lot of challenges with this. Um, but it's, it's something that we're, we're certainly looking into, um, on the research team. Um, and, uh, we'd also like to give customers more fine grain recommendations right now. We only give a recommendation for the entire zone of say, set your entire zone to this, um, TLS mode, or this encryption mode, um, for the whole, um, and then that covers all subdomains, et cetera, but really for some customers, they might have one problematic subdomain that is still on HTTPS. And they'd really like to upgrade the entire zone to a more, uh, to a stronger, um, encryption mode, but, um, that one subdomain is preventing them from doing that. Um, so we'd like to give customers the insight to, so they can identify these, these problematic subdomains and either fix them up and add TLS support, or, um, they can add, um, use, use other Cloudflare features like page rules to just set a lower SSL mode or lower encryption mode only for that subdomain, but then upgrade the rest of the zone to a stronger mode. Um, so there, there are a few things like this that we've been, um, we've been thinking about and want to want to, um, implement. Um, so I guess as, as customers listening to this, if, if you have your own pain points to, to share and, um, and want us to work on a certain, certain aspect of this problem, uh, definitely please reach out. Um, we have, uh, our email address for the research team at the bottom of the blog posts. It's ask-research at Cloudflare.com. So, um, yeah, we're, we hope to hear from you, um, about things, things that, uh, I guess in this vein of research that you'd like us to, uh, to solve for you in the future. Cause we, we want to avoid pain points for our customers, uh, whenever we can. Awesome guys. Thank you so much for being here today. I'm gonna take the last two minutes just to plug a couple of things. First is this was an intern project. This is something that Suma and Taha worked on when they were interns here at Cloudflare. And these are the types of things you get to build. And you're an intern at Cloudflare, not just a research team intern, not just a product intern, not just engineering intern, but all interns at Cloudflare are really invested with helping change and build core parts of the business here. So if you want to work on really interesting things that, you know, touch half a million users, make the Internet a better place, make it way more stable, go check out the summer internships. They're up on the careers page. The second is that if you want to learn more about this, I know Luke's been saying the blog, go check out the blog. Also go check out research.Cloudflare.com and read the paper that Taha put up. There's some really interesting work going on in the encryption space. And we know this is just talking about SSL TLS recommender, but we also announced today that we're doing a lot of work on ECH and really pushing TLS 1.3 ECH toward a full implementation of the standard. Go read that blog post, go see the work that Chris Wood and Christopher Patton have been doing. It's really interesting. And it's so in alignment with what we're trying to do here, because the core part of trying to make the Internet more secure is help build a better Internet for everybody. The Internet wasn't built with encryption in mind. And so we're having to go back through now and add it back in. This is just one part of a number of parts that we're working on here at Cloudflare between pursuing FIPS certification, pursuing post-quantum cryptography, pursuing ECH, the privacy story, the encryption story is real and alive. And if you're interested in working on it, go check out the careers page, not just for internships, visiting researchers, but also full-time roles. We're hiring like mad right now. And with that, thank you for being on Cloudflare TV guys. Catch you later.