Beyond IP addresses: Cryptographic primitives for honest bots

And today, we're going to talk in terms of like how we've been starting to sync, how we're moving beyond IP addresses, and specifically, we'll dive into like some cryptographic primitive that we're thinking for on-air spots. Before starting, I want to ask the room, who's operating the bots, a bot in the room? Okay, there's a couple of hands. Who's operating the Cloudflare Workers, who's running some like call command, something like that? There's even more hands. And like, it's already starting to get interesting, because what's a bot is actually a bit fuzzy. And so like, what we would like is to have like some signal that people would be like willingly provide that would help us to like do some job and help them to like better identify themselves. So I'm Thibault. I've been working on web bottles for a year and a half as one of like the main co-editors of the draft. And we're going to dive without further ado. So one of the main thing that we're trying to sync is like, how do we help on-air bot identify themselves? And the perception of like what is a known as bot, what is like friendly, what is unwanted is really up to origin and Cloudflare tries to like convey the intent that the origin has given us into what traffic they want to allow, what traffic they want to block, what traffic they want to rate limit, or whatever policy they want to take. And so like the bot that go here, we have like some time, like really traffic that like you don't want. You have some AI agents, which may be traffic that you want, maybe traffic that you don't want, maybe traffic that you do something else with. And you have like some friendly bots. And what you really want to help here is focusing on like the friendly bots. How can we help to help them make sure that they can identify themselves in method that works for them and that works for us. So historically, a lot of the detection mechanism and like that's what you have a dashboard, you rely on IP addresses, or you rely on user agent. And the IP address, maybe because like you're seeing a lot of requests coming from a specific IP address. And you say, okay, like this request, it seems like this a bit too many requests coming from like this IP address. I'm looking at like an IT info table, I'm looking at like Google, for instance, documentation, and I can infer that this IP address comes from Google. Maybe it comes from like a data center from like GCP, maybe it's associated with open AI. Or similarly, we have like the user agent, when we have the user agent, which is set to like GPT bot, open AI publishes and advertise that like this is a user agent that they use. And therefore, you can like reasonably think like, this is what open AI will use when making requests to you. Of course, depending on the product, each of the company may segment this type of like identifiers. Like you may have like different egress ranges for like Google search, or different type of crawlers that Google uses, similar with your user agent. Google has a whole documentation in terms of like which user agent that they would use. And to be fair, it's not specific to Google. Cloudflare does the same. We have a couple of like user agent that we use, should be workers, should be like remote browsers, or should even be like traffic that we pass back to the origin. Thing is, this type of signal like quite brittle, and we can do better. So for instance, like this is a list of IP address, which is served on a more or less arbitrary HTTP page. These IP need to rotate, because I'm on you sell them because they are not available. There's a lot of like management involved. Similarly, with user agent, they are spoofable. I mean, you know that like Google agent is like Google bot. And because you know that, you can also set that same shield and that same header as part of your request. There are a couple of like other reasons why this doesn't work. But really what we want is like something which is like is like spoof, spoof, protected against spoofing. But we'd also like it's kind of like easy to manage, to rotate, to extend, and works in an area where we have just like diverse set of participants and parties. So what we looked into is how can we introduce and leverage some of the knowledge we have on digital signatures? How can we do that for bots? And this way, when you have some bot which is unwanted, well, they won't sign anything, so you won't deal with that. However, when you have like people that like are willing to help you, similar to what Google does with like sharing IP addresses, setting a specific user agent for OpenAI, etc., they're able to have the mean and the method to do so in a safe, secure way, and in a way you can understand them. And in this talk, we're going to dive exactly how we constructed an initial version of WebBottles with HTTP message signatures, what are the properties we were looking for, and how we can achieve that. So as I mentioned, the way WebBottles work is it relies on HTTP message signature. HTTP message signature is an RFC and like it just more or less looks like that. So it's a set and a series of HTTP headers which are formatted in a specific way, which we're going to detail. It's a digital signature which is carried through the HTTP header. It's using public key cryptography. This is what provides the property that we want, which is like security and authority, and it comes along with every request. So just by looking at the request, I have all the information I need that the other party, which is a client, the crawler, the bot, wanted to communicate with me. If we look a bit, it's actually like separated in three main components. We have the signature agent, which is a discovery hint which is provided by the client, similar to have the user agent attempted to convey what type of product was making the request when it was Googlebot. You had a hint that it was Google. The signature agent conveys something from the client. In this case, it conveys a URL, like example.com, where the set of keys from the clients are hosted. The second set is already standardized in RFC 9421 for HTTP message signature and it's a signature input. It is like the set of parameters that we want the signatures to cover. We have a couple of The authority means we want to sign the domain so that, let's imagine I'm a bot, I'm making a request to example.com. I can sign example.com to guarantee that this request is for example .com. This will prevent when I'm going to evil.com, evil.com reusing my signatures and impersonating me by replaying it. We have some timeline, we have some print, which is cryptographic details on that side, and we have a tag, web bottles, such that when I use web bottles and if I have another application down the line, I already have scored my signature to the specific protocol. There are some parameters which are optional to, again, protect a bit against free play, to have some better signatures, and finally we have the signature which we will validate at the edge. Cloudflare does it automatically, but the nice property of these HTTP message signatures is that it's also transitive. Cloudflare passes the signature down to the origin, and so if you have additional set of policies that you want to apply, you can do so. If I can go on the next slides. The main question is, why do we have these specific mechanism? Because it's great, we have three headers and etc, but are there the right one? Are they the one that will allow bot that want to identify themselves do so in an unspoofable manner, in a secure manner, and in a way that they control it? The property we're after were request authentication. We want to ensure that when the request comes in, it's actually coming from the party which we think made the request, and this is why this request is signed. It proves that the sender holds the key. If we have and we see the same key ID, we know that it's actually the same key that made the request, and therefore the same parties that we have validated before. The second property we wanted is we wanted to have some sort of discovery. It's very inconvenient when you receive requests and a lot of requests from a specific IP address that you have to somehow infer that this IP comes from Google, this IP comes from like OpenAI, this IP comes from like a data centers, and it's like much more convenient if you have a way to like somehow, some ways to discover the material that you need in order to like make the claim and like have all the validation. In web bottles can be conveyed by like the signature agent, can be carried out of band, might still be on Google documentation or on Cloudflare documentation to provide. It's just like smooth line and streamlined throughout the protocol. One thing which is important through that is anyone can set, similar to the user agent, anyone can set the user agent and like host keys on the Internet. So discovery is not trust, but it helps to build it as you carry on history with it. And finally, we want to have like some sort of binding, so the key can be hosted at any domain. If I fetch the keys of Google, I could re-host the keys of Google on my domain. Same with like Cloudflare workers, we could have like the keys of Cloudflare Workers hosted at one address and these key re-hosted elsewhere. And so what's important is that, similar to what we said, discovery is not trust. It's not because I see certain keys on certain domain that like the keys are actually directly associated with the domain. That's what domain binding is doing us. When we have, for instance, the keys which are hosted on the well-known URL, such as like well-known HTTP message signature directory, we know that like the parties that is in control of the domain has taken some special action in order to host the key at that place. And this way, we're able to provide some domain binding. Okay, let's talk a bit in terms of like diagrams and interaction between the different parties. So we have the client. So in our case, the clients are our friendly bot. And then we have the origin. Could be like the origin, be Cloudflare. The advantage here is like it works throughout intermediaries. So you have a client, makes one request, two requests, three requests. We can see it's like a client that like actively wants to be like identified. It like sets a signature agent. Maybe it has like a custom user agent that it sets, etc. It makes a lot of requests and like everything goes through. And at some point, because you have some strength on your infrastructure, because you start looking at your logs, a lot of events, you start noticing that like there's an increase of traffic on your site. And so what you do, you can look at the signature agent. You fetch the keys, which the signature indicates. And now that you have the key, you can like validate the request signature with the keys that you had. The signature proves possession of the key because people, in order to make a signature, need to have the keys in the first place and therefore can apply the local policy. The next time that they will be able to receive a request, the signature, a signature agent, they already have the key. The key is already known and therefore they can still validate the request. The same key ID means like there's some continuity and therefore if I receive more requests down the line from the same actor, I can start to build some history based on that actor. And again, I can apply some policy. Can be like some rate limiting, can be like allowing, can be blocking, can be make whatever policy I want based on the knowledge that I have of the actor. One more thing about the well-known URL. The keys can be served from like any arbitrary URL. If they are served from a well-known, we gain a bit more information that the domain owner, which is in control of that domain, has placed and make some like special things in order to host those keys at this address. For instance, if the keys are on clarflare.com slash well-known slash http message signatures, I know that clarflare and the owner of clarflare.com has made some special work in order to make that happen. So it like provides some more guarantees and I can tie the keys to the domain which served them. Great. So WebBottles has been kind of like a year and a half effort already and we had some learnings from deployments. It has been deployed by bots of various sizes. We have like small bots and we have like bigger bots. There are a lot of open source implementation, TypeScript, Rust, Go, a lot of languages. We're probably not aware of like all the implementation that exists. If there are some which are more like is the trick language or like you particularly love because of the design, feel free to contact us. In terms of like what really worked, at least in observation that we had with WebBottles, is like it's signing that happened with the HTTP layers. We also considered, if you look back at the post we made a year and a half ago about WebBottles, we also considered doing that at TLS because we have a good mechanism. We have like mutual TLS and like we're allowed to do that. The thing is it required a bit more modification and it was like a bit harder for people to deploy should it be on the bot side because again if you use a clarflare worker you don't control the TLS stack for instance. It was also harder for people to implement on the server side should be on clarflare side but at like deployment of various sizes and therefore it seemed to have been like rather good. It's extensible because of like the request semantic that we have with like HTTP headers meaning that like we provide a minimum set of headers that you should sign in order to be compliant with WebBottles. If you want to like customize it and extend it because you have particular needs for your application, should be like a client ID, whatever headers, you can do that and we've seen all the protocol extend WebBottles in order to do this. And finally one of the main change if you're an implementer of WebBottles in the room, one of the main thing that like we've seen people struggle with is we like we've changed the signature agent from a signature string to like a signature dictionary which allows for like multiple signatures to be carried along with one request. Imagine you have multiple requests, each intermediary wants to sign them. This is one thing to know that's been like the spec is evolving, that's like really something I want to highlight. It's evolving, it's changing and this is because we just on our own kind of do everything right on the first time and that's why we too can discuss with people in the open at the IETF in order to like make the protocol evolve so it's sort of and it's strong and works for everyone. In terms of code, I tend to like TypeScript because like they are easy to deploy on clarflare platform. So you have like html signature in our library which is WebBottles. You can use it like already like directly in workers, you can use it in chrome and it's not too many lines of code. You can like reduce that, it's just in term of like expressivity. I do feel this was like the easiest. We have the signature agent so here's like micro load of test, we have the request on which we attach the signature agent, then we create like the fields that we want to sign. So as I said we want to like have some timestamp so like the time at which the signature was created, the time it expires and once we have all the fields we use like the signatures to get like the new headers we want which is the signature, the signature agent and the signature input. And frankly that's it. You can just like write it out today on your own. One of the things that like it really helped here is I think for Origin it allows them to have like more confidence in like the messages that they get, IP addresses, user agents like they worked but like they were not like ideal signal through that. And really the ability for Origin to be able like to choose to have like confidence signal that they have worked and reduce the number of incidents that they had. Being able to like allow parole on your website so you have like better discoveries through search engines, like AI engines and etc is something which is important otherwise you don't get discovered. Similarly for like bot and agents it allows them to like prove the mechanism without being tampered with. This is something that they control on them like themselves. It's not something which has like a governance or whatsoever and it's compatible with existing method. I don't have to move away and change my IP addresses, change my user agent. That's something that I can add on top to provide more confidence to the Origin that I'm contacting. And that's it. It has been being in the open at the IETF. This is where and why the protocol is still evolving. We have so you have like all the mailing lists, the draft and etc at these links. We also have GitHub with like the implementation that I mentioned and that's it.