Behind the scenes: Challenges of running AI models at scale

Hi everyone. Welcome to this session, which got renamed very last minute, but we're going to talk about Cloudflare as a new lab. And if you knew, we shipped a few models in the last few weeks called Clef and Cleric, and this session is going to tell you about why. So I'm Michelle. I am a group PM here on the AI platform. I run our products like Workers AI and AI Gateway, AI Search and Web Search as well. And my team is behind Clef and Cleric. So as I was preparing for this talk, I wanted to figure out the journey that we have together today. And so there's a few reactions that I really wanted you guys to have. And it's not clicking, so I'm just going to do this. The first reaction is that, wait, Cloudflare as a CDN company of 16 years actually is running GPUs on our network. And we're actually running models hosted on those GPUs and we're selling tokens for those models. And then lastly, wait, we trained our own models. We're selling tokens for them, running those on GPUs on our network. And I think the question on people's mind is, this is a little bit random for Cloudflare, so why we're doing this, right? And I think this part of this talk will go through the journey of why Cloudflare is actually really interested in this and why it makes a lot of sense for us and what we're trying to do. And the second goal I had for this session was to keep you all entertained. I have been in your shoes many times at many talks. I've stood up here and given similar talks as well, in which I got bored, I went on my phone, I zoned out a little bit. And so I thought to myself, what is the best way to keep a group of nerds entertained in a room? And so we're going to talk about Dune. And I know it works because this is me as a Fremen in 2021 for Halloween. So I'm going to keep you all engaged here and show you how the workers AI cinematic universe actually parallels the Dune cinematic universe. So first order of business, we're going to talk a little bit about why Cloudflare has GPUs. And you guys, I don't know if this is a surprise to you all, but we've been running GPUs on our network since 2023, which is when we launched workers AI. If you know Cloudflare, you know we run a global infrastructure. And so this is one of the superpowers that we have, is that we actually own and operate data centers all around the world. And it's a very distributed infrastructure pattern. And it's different than your typical hyperscalers, which compute is like centralized and US East one and US West one. As a CDN, we want it to be able to serve content from all over the world close to the user as quickly as possible. And so we have a distributed network of data centers all around the world that we own and operate. So in 2023, we actually ended up adding GPUs to those data centers as well. And so when you're running inference, we thought that you would want to run inference as close to your user as possible. The reason we did that is because we had a hypothesis here, which is that the models are going to get smaller and they're going to get more efficient and faster and more optimized, which means that network latency or where your models are running becomes more and more important because you want your network latency to be as short as possible. So your users can get your answers back as fast as possible. So we launched workers AI and we had a catalog of over 50 plus models in our service. We had specific models like M2M for translation, and we had a ResNet for object detection and Distilbert for classification. And those models are still around today. If you want to check it out, there are small specialized models that are able to do specific tasks. And I want you to think of these models as Fremen, right? And so they are very specific to their domain. They're highly adapted to the planet that they live on. And they're really, really skilled and able to do specific things on their planet that no others could, right? It'll all make sense in the end. I think what we actually saw was that the opposite happened. Models ended up getting larger and larger throughout the years. And so throughout time, I think it actually makes sense if you look back at it retrospectively. But model started as like 7 billion parameter models all the way today, where KiwiK3 is about 3 trillion parameters of models. And if you've heard of something called scaling law, this makes a lot of sense to you. On our quest to AGI, scaling law says that if you train with more data and you create larger models, you'll create generalizable, more intelligent models in the end. So I think it makes a lot of sense that these models are getting larger and larger because people want to train a model that is capable of achieving AGI. So giant models ended up actually going mainstream here. And this looks really different than the infrastructure pattern that I just talked about, where we wanted small models at the edge, right? But what could we do? An LLM could summarize and translate and classify models all in one. So we ended up throwing everything at the LLM. But we adapted Workers AI as an inference platform. We ended up adding these large models to our platform as well. So in March 2026, we added KiwiK2.5, which is our first trillion parameter model. And then we follow up with GLM and DeepSeq as well. So all of these really giant LLMs are also available on Workers AI for everyone to use. And we actually got pretty good at it as well. So if you look at these throughput metrics, we were able to achieve some pretty good standards for per -request metrics. We did a lot of learning and experimenting. We've done pre-filled decode disaggregation, KVCache quantization, experimented with speculative decoding heads as well. And so these models are actually available on our platform for everyone to use and experiment with. And with these giant models, the edge stopped mattering as much, right? When you're waiting 10 seconds for an answer from an LLM, the network milliseconds of latency gets dwarfed. And it doesn't really matter where your GPUs are located because you're waiting for an answer to be generated from the model anyways. So these large models are like sandworms. And as I was building this, I realized the sandworm kind of looks like the Claude logo. So I think this is canon. But your sandworm can go really far. It can go really fast and really deep as well. But some of these things were overkill, right? You don't really need a sandworm to cross a sidewalk. So this is kind of what we did, though, right? We sent everything to an LLM in the end so that they are able to answer really lame questions like, is it urgent? But also do like actual calls and coding and things like that. So everything was sent to the LLM. But really, that was a bit overkill. We really didn't need to do that, right? There are certain steps here that could be answered really quickly that didn't necessarily need an LLM. And the LLM is probably really important just for the end where you want to take action or write a reply to something. But the other tasks could actually probably be done with other models. So I think this is what takes us to today. In September 2026, a company called TypeSafe released a model called Jev, if you guys have ever heard of it. But it introduced a new concept of models to the world called System 1 Models. And the idea is that it split the discourse a little bit around System 1 Models and System 2 Models. And this follows Daniel Kahneman's book called Thinking Fast and Slow, where there's a lot of things that you want to think about really quickly, right? And so the System 1 Thinking is really fast, intuitive versus System 2 Thinking, which is very slow and deliberate and has a lot of reasoning. And this parallels to the models universe really well because you have System 1 Models in which you want to do things really quickly. But then you might have to use System 2 Models like LLMs as well to be able to reason and be deliberate about what you're trying to do. So System 2 Models actually write like LLMs, write and answer one token at a time. And so it takes a lot of time. It's really expensive. It's very computational. And System 1 Models are a little bit different here. It does one single forward pass. There's no output tokens generated. And it actually can give you a response back in milliseconds. So this is an example here with Jev where you have a typed state and you do a forward pass on it and you get a set of probabilities back that it's in a certain category or not. But the idea here is that System 1 Models are fundamentally and architecturally different than System 2 Models. And it makes sense to use those models when it actually matters. And so we're back, right? A small, specialized model. We've kind of come full circle. And our original hypothesis with launching workers AI in 2023 really holds up where we're like, we're back to small, specialized models and not just throwing everything at a giant LLM. So this sounds a little bit familiar, but this sounds really familiar. But one thing that happened to us was that Jev is a closed source model, which means that we're unable to host it on workers AI. And so what we decided to do was we trained our own model. And when we trained... Oh, sorry. One sec. Sorry. With the System 1 Models, the network becomes important again because the latency here matters a lot. And so it makes a lot of sense for System 1 Models to live on the Edge GPUs. So Jev is closed source. Can we build our own? Yes, we did. So Jev launched... Oh, sorry. Jev launched on September 15th. And three days later, we had a working demo of how we could stimulate Jev. We took a week to deliberate on how, if we wanted to go into the Neolab business and start training models, because it takes a lot to do that. But we decided to do it. So after a week of deliberation, on a Friday, we decided we're going to launch something here. Over a weekend, we ended up training Cleft. And then just a few days later, on Thursday, October 1st, we shipped Cleft, which is Cloudflare's first open source model. It was really our answer to Jev here. And we didn't just ship one model, we actually shipped three. So it has a few different flavors. Cleft is the eponymous model here. It is the most high quality, high precision. We have Cleft Flash, which actually ended up being cheaper than Jev. And we have Cleft Omni as well. And this is us pushing the barrier a little bit on innovation, where it's able to take in audio, image, video, and text as well, which Jev is not able to do. So I think we got pretty good at it. Cleft is also fully Jev API compatible. And so if you were using Jev in your workflows today, you can swap that out really easily and just try Cleft. We're also using Cleft in production today. And so if you look at Cloudflare, we have developer docs, we have our MDash platform, we have DLP and our threat intel teams that are using Cleft in production. One thing that I think is really cool is the DLP team has been able to use Cleft and wire it up, where it is able to detect if your email attachment or your uploaded image is a government ID or something like that, so that it can actually prevent PII from leaking. And the cool thing here, I think, is that we've moved classifications into the model layer. So any team is now able to play around with Cleft and do classifications on their own, versus before you probably had to assemble a machine learning engineering team and gather a bunch of data and train your own model specific for PII detection, right? Now you can use Cleft and it works out of the box and it's really democratized classifications for any team and anyone to be able to use it. Okay. So people are zoning out. I'm going to talk about it again. Cleft is kind of like your Bene Gesserit, right? You give it a bunch of paths in front of you and it can help you choose which is your best path to take in order to get to your goal that you're trying to do. And Cleft was kind of a smash hit. I think it got a lot of good reception. We had a lot of buzz around it. So we decided to go back to the drawing board and think a little bit more about what are LLMs also not good at? Well, they're really expensive. They reason, they think a lot, they take a lot of time, but also they're prone to hallucination, right? Because they're of the autoregressive nature of it. And so we decided to think about what are the other problems that get thrown at LLMs and what are they not good at and what else can we do in this space? So we trained a new model. It's called Cleric. It is again, a system one model and we're calling it an extraction model. And so what Cleric does is it allows you to interact with unstructured inputs and get typed values out. What I like to say is when we're going and interacting with the world every day, we're not interacting with perfectly typed JSON objects. We're talking to messy, unstructured data and inputs, right? And so what Cleric can actually do is it can extract and compute values from your data. It can estimate values and it can forecast values as well. So we're going to walk through a little bit of these use cases. With Cleric, you enter your state and you pass it through the model, you get values and outputs out. You can extract an answer. So let's say you got an email about your invoices and whatnot. You can actually take out, you can ask for the invoice ID and it can strip it out from the email. You can see that these are actually typed. So there are strings or there are dates and it actually returns a character span as well. So it can actually reference back to your inputs so that you can tell where it took the data from and it computes values. So it can go a step further where it's not just reading the data that is there, but is able to compute it. So if you ask for a total and then you want to add the tax and tip, but you're asking for the total amount paid field, it can actually calculate that in one pass or add dates together or anything like that. It also answers with a confidence score. And so one thing I'm particularly proud of is the Cleric model is really good with their calibration, which means that the answers that we're really sure about, we're very confident about and will return a good confidence score. And the ones that we're not sure about where we'll be honest and state that we're not very confident about that. And that way you can take a second look if you want to do a manual review or pass it through an LLM or do any type of more slow, deliberate thinking and reasoning about it. Oh, it's there. The Cleric model also takes image inputs. So if you've done expense reports with receipts, you know how hard this is to get outputs out of your images, but it can actually take vision. So it understands the world as we see it as well. The other thing that Cleric can do is actually it can estimate values. So if you give it some data, but you don't tell it the value exactly, it'll actually be able to look at the data surrounding your inputs and output you the estimate. And these estimates, we come with a median and then the 80% interval. If the interval is tighter, we're usually more confident about it. But if it's really wide, it's probably worth taking a second look. But it's pretty powerful that you can do both extraction and computation and estimation all from one model. And you can do forecasting. So if you have some time series data, you can be able to predict the next step from the Cleric model as well. So you can see the green line is what Cleric predicted the trend would be. And it actually picked up the trend, you know, the daily trend pretty well. So it can also do all of that out of the box. So Cleric here is like a mentat, right? It thinks through your inputs. It can look through history, pull things out really quickly and calculate things. So all of these now are available on workers AI, and you want to put all the systems together, you want to think fast, then slow. Let me walk you through an example of why this matters. So we're interacting with the world, you get an email, very unstructured data, it's not a form, right? It's just in plain text, you get calls, you get Slack messages like this every day, you take your unstructured input, you pass it through Cleric, it extracts the values from you, it can estimate it, it can extract it, it can compute it as well, and it returns the confidence scores for you. Then maybe you have a policy, let's say you want to repair the order if it's under 60% of the cost, Clef can help you actually make a decision here about the policy and decide what you want to do. And then lastly, use an LLM to apply that, right? You take action on it, you write a reply, or you use a tool call, or you code something up. But the LLM is actually just doing the step that has output. So putting this all together, you can see that the cascading pipeline of models is actually just under three seconds, versus if you used an LLM for every step, it's about 15 seconds. So Cleric and Clef here are doing a lot of heavy lifting, because they take just under a second for generation. It ends up actually being 5.5 times faster to use these cascaded pipelines of models versus using an LLM outright. And it's about 75% cheaper for you to use the cascaded pipeline versus if you just use LLMs for this whole thing. So putting this all together here, you really want to do your extraction, your decisions, and then your actions all together. And these models have become purpose-built for these specific things, rather than throwing it all at an LLM. So the big question on everyone's mind is probably, why is Cloudflare doing this? Why is Cloudflare, the CDN company, training their own models and doing this? Well, Cloudflare really wants to be the agent cloud. We want to be the platform for you to deploy your agents on. And we've had primitives that we've been working on for decades. We have workers, we have durable objects, we have newer things like sandboxes, an agent SDK in Flu. The idea is that all your agents should be running on Cloudflare. And with an agent, you need a brain. You need something powering that brain, right? You want your agents to be running with a model that is also on Cloudflare. And so we became a Neolab. We took our edge infrastructure, which is a perfect fit for System 1 models. We took System 2 models, which we got really good at serving. And now we have all of these available on Cloudflare. So the reason we became a Neolab was that we thought the System 1 shift in the tides really follows our original hypothesis of these edge GPUs. And then we also got really good at System 2 models. So we want to offer that all on our platform. So all of these models now are available on Workers.ai. You can use your System 1 models like Cleric and Fluff. You can use your System 2 LLMs like Kimmy and GLM and DeepSeek. And you can also use your specialist models as well, right? Things like image generation or transcription or audio, things like that. All of these models are now running on Cloudflare. The one thing that I think about when I interview people and they ask me why wouldn't I just go to a Neolab is why is Cloudflare uniquely positioned in this space? And I think there's some things that a Neolab can't copy. Cloudflare has been around for over 16 years. We have 16 years of network and infrastructure and data center experience on how we're running this network. We have edge GPUs, which are special and unique to Cloudflare's infrastructure pattern that can't be replicated easily. And then we have a full developer platform that people can use to deploy their agents onto, right? And these are things that Neolabs can't stand up in the next year very quickly. So I think this is one of the unfair advantages that we have at Cloudflare and why it makes a lot of sense for people to be building agents and models running on Cloudflare. Okay, we're going to run into all that with Dune. So in this Workers.ai cinematic universe, you are the Lisa Naugai. You have this end goal in mind and you have Bene Gesserits and Mentats and Fremen here who can help you, right? And I want you to be able to make the right decision. You want to choose where it makes sense to use a specialist or when it makes sense to use an extraction model or a decision model or a large LLM. And all of these are available to you and you have to decide what the right use case is for these right models. So this is my talk. I think I have one minute left, maybe one question, or otherwise you can find me after. Thank you so much for joining me today. I hope you learned why Cloudflare has been doing this and why it's so interesting for us.