Turning data questions into data infra with Cloudflare Basin

Hi, everyone. My name is Micah. I'm a principal engineer here at Cloudflare. And today I'm going to be talking about Cloudflare Basin, our new suite of data infrastructure products, and how agents are changing the practice of data engineering. So for the purpose of this talk, let's say that instead of being tech lead for Cloudflare Basin, I'm actually CEO of Basin Mountain, a totally non-IP infringing online storefront for mountaineering gear named after the beautiful Basin Mountain near Bishop, California. And like all good e-commerce CEOs, I start my morning by looking at my dashboards and seeing what my sales volume looks like. And unfortunately, this morning, I looked at my dashboard and I saw that my sales are down this week. And I want to understand why. So I'm going to talk through how this sort of works for most companies today and for the past decade or so of data engineering. And then we're going to talk about how this can look with agents and Basin. But in that kind of traditional story, I probably ping my data engineering team. I show them this chart. I ask, why are our sales down? And they're going to go probably run some other queries, look at what other data we're collecting. Maybe look at traffic and see if our traffic is down. And that's driving our order volume down. In this case, our traffic looks pretty normal. And then they're going to try to figure out, OK, do we have the data that actually answers this question? If not, they're going to have to go kind of look at what actually drives this outcome that we're seeing. So maybe we have like a checkout flow that looks something like this. User kind of goes through these pages and eventually adds the order. We can see the pages are normal. The orders are lower than we expect. But what's going on in the middle, we don't really have any visibility into that because the user or the engineer who instrumented this application at some point in the past just didn't add those events. This is a common story. When you build an application, you have to figure out what you think is going to be interesting in a year. And you often get that wrong. And you discover that when you need the data that wasn't being collected. And so in the old world, OK, maybe the data engineer files a ticket with the application engineering team. They go get that on their sprint board. A couple weeks later, they get to it. They add the code that instruments that event. Then it goes back to the data engineering team. They build a pipeline to ingest those events. Finally, it can land as a table where they can query it and start to understand what's going on. This all works. I mean, this is what we've been doing for a long time in the data engineering world. I've seen this cycle play out hundreds and hundreds of times. And it works, but it's just so slow. And by the time we get the answers, it's probably irrelevant. We're following this pretty straightforward process, looking at the data we already have available to us. Does it answer the question we need to answer? If not, going and collecting the data, building pipelines to ingest it, and finally being able to query it. And it's just such a, there's so many people involved. There's so many handoffs. And ultimately, this is going to take weeks at any kind of organization. We've been thinking a lot about this problem in the context of Cloudflare Basin, which is our suite of data infrastructure products, and how agents are really changing this experience. So I want to show you what this might look like today with Basin. So if we could switch over to the laptop. So I'm going to be doing a live demo here with conference Wi-Fi and AI. So wish me luck. Let's see what's going on. OK, there we go. So in the Basin Cloud story here, I start my day in my Cloud desktop. And I ask it, what's my order volume over the past month? And Cloud knows how to query Basin. Oh, jeez. And so instead of going to a dashboard where this is all kind of pre-configured with questions I might want to ask, I'm just going to ask my question directly of Cloud. And it's going to go figure out what data is available and how to query it. And in just a moment here, it's going to produce more or less that chart we saw before. We see that same issue here. Our orders are lower than we expect for this past week. But instead of pinging a data engineer to go pull up the data, I can just ask Cloud, so why are orders low this week? And it's going to actually follow a very similar process to what our human data engineer followed. It's going to look at whatever other data we have and see if that could explain the outcome in this first query. If we kind of look at what it's doing, it's actually doing exactly that same thing. It's querying the traffic data and trying to see, OK, does that explain the drop-off in our order volume? And it's going to think for a bit, run some more queries, explore the data a little bit, and try to get a sense of what it's seeing. And then in a moment, it's going to show up some graphs probably. And oh, OK, so it has an explanation. Yeah, so same thing we found earlier. It sees that our traffic is steady. Conversions were stable, and then the conversions dropped. It's done some more analysis of it. But ultimately, it can't really figure it out because we're just fundamentally missing the data. Well, the next step there is we can just go collect the data. Obviously, this website is deployed onto Cloudflare, so we can instrument it with Basin. And I'm just going to have Cloud do that. So, oh, jeez, OK, thanks. OK, so I'm just going to ask it to instrument the checkout flow. And this is going to take a little bit of time. So if we pop back to the presentation, we'll let that churn. And we'll see the outcome at the end. And see what it finds from this. We just went through a lot of stuff. I want to talk a little bit more about what we saw there. What is Basin? So I said at the beginning, Basin is our suite of data infrastructure products. So this gets you end-to-end from data that your, in this case, website is producing, devices, servers, into a very scalable data lake built on top of R2. There's three products today as part of Basin. The first is called Basin Catalog. This is a managed iceberg catalog built on top of R2, which is our object storage product. We're familiar with S3. R2 is like S3 on Cloudflare. Unlike S3, we don't charge egress fees. There's no cost to get your data out. And this turns out to be really valuable for storing your analytical data. It means that you can actually bring any cloud vendor, even self -hosted query engines, anywhere you want to expose your data, you can do that on top of R2 and Basin Catalog incredibly affordably. We make that even easier by supporting open standards like Apache Iceberg. For those who aren't familiar, this is what's called an open table format. It's basically a way to turn a bunch of files in object storage into a set of tables that you can query. It's basically a data lake. And because it's open standard supported by basically everything in the industry now, you can bring whatever tools you want to your data stored in Basin Catalog. So things like Spark, Snowflake, Databricks, self-hosted tools like DuckDB or PyIceberg, they can all talk seamlessly to Basin Catalog. The other important thing about a catalog, it's not good enough just to store your data. Over time, as you write more and more data, the data gets disorganized. It ends up in small files. And so over time, you need to be able to basically rewrite that data in more efficient forms for querying, a process called compaction. And Basin Catalog does this all for you automatically. You just have to turn it on. You don't have to think about management of the data over time. The second piece is, how do you get data into your catalog? Because it's Iceberg, you can actually use any Iceberg-supporting tool. But we have a tool on platform that makes this super easy called Basin Pipelines. So we saw earlier in the demo the site sending traffic events and order events into our catalog. Those were going through a worker binding into a pipeline. You can also send events via HTTP. We also have a LogPush integration. If you use Cloudflare's CDN security products, you can straightforwardly get the data from LogPush into a pipeline. Once it's in the pipeline, you have an option to transform it by writing a SQL query like this one here. So you can restructure the data into whatever form is most efficient to actually query it. And then finally, once the data is in the catalog, I mentioned you can use Snowflake or Spark or whatever you want to query it. But we also have a product on platform to query those tables. It's called Basin SQL. It's a distributed query engine that runs across the entire Cloudflare edge. That means it's incredibly scalable. It can scale out to petabytes of data. If the query itself is very paralyzable, we can scan petabytes of data in literal seconds. It's pretty amazing what Cloudflare's edge infrastructure allows us to build in this area. It supports kind of all the things you would expect a SQL query engine to support. And it's fully serverless. There's no clusters to manage. You don't size anything. You just issue a query and get the results back. So why do we think Basin is really effective for agents? We had the opportunity to actually start building this in the agentic era. We started this process about two years ago. So unlike existing vendors that have a lot of legacy built for an earlier era of mostly human data engineering and data infrastructure, we really started with how do we want this to be used by agents. And what we've noticed and what we maybe saw just in that little demo there is agents have this process of trying to understand and affect their environment, which is kind of this loop here. First, they try something. And then they try to see if it was correct or not. And then if it is, great, we'll hand it back to the human. If not, we'll redo this loop as many times as we need to get something that seems to work. And so really good agentic infrastructure basically makes this whole process fast, cheap, and non-destructive, like low risk for your existing infrastructure. And so that's what we've really tried to support with Basin. There's a few different properties of it that really align well with this way that agents want to work. One is like all Cloudflare developer platform products, it's completely serverless. So there's no infrastructure that agents have to create and manage and own over the long term. All they have to do is create it, try it out. They can tear it down if it doesn't work, it doesn't do what they want. You don't have to sign a contract over long term to get a certain amount of compute capacity. You can really let the agents just go and try stuff out. The second kind of property this platform has is that it's really amenable to agents interacting with it. Everything can be created, investigated, destroyed via an API and a CLI. That means agents are fully in their power to kind of go wild on the platform, and as we saw, like instrument an application, create the resources they need, tear them down if they don't work. And you never really need a human to step in and like sign a contract or go through a dashboard to do something that the agent needs to do to set up this infrastructure. And the final thing, agents really love to like fan out and like try a bunch of stuff. And if you have like a fixed cluster capacity or a certain amount of compute that you can use, you really are bottlenecking how fast the agents can solve your problems. So we designed Basin to be basically infinitely horizontally scalable. As I mentioned, it runs across all of Cloudflare's tens of thousands of edge machines. So it really unlocks the agents to just kind of like swarm the problem and try to understand when you're issuing a query, like what does the data look like, what tables are going to answer this question, what interesting things are there in the data that I need to understand. The second kind of Cloudflare superpower we get is that Cloudflare is basically a giant cache. That was our first major product and still how a lot of our infrastructure is really oriented. And so here we get to take advantage of that by basically caching all the data that the agent is looking at really close to where the queries are running. And that means that when agents want to do this loop where they try something, maybe like move the query over a little bit in a slightly different time range or restructure the query a little bit, they're not having to read all of that data from cold storage. They're just reading it from the cache. And that makes that process really, really fast and cheap. So all of this went GA three weeks ago at birthday week. It's all available now. It's priced, as I mentioned, in a fully serverless way. So you pay only for what you use. There's no long-term commits or costs. And it's all available today. This is normally where I'd say if you want to get started, here's the docs. You should go take a look. But we're in 2026. So honestly, just tell Claude, instrument your application with Basin. And it's going to do a pretty good job. So I encourage all of you to try it. It's incredibly easy to get started. And if we can pop back to the computer, which unfortunately has gone to sleep here, let's see how it did. Cool, so yeah, Claude went through. It added the instrumentation in the order flow. And then it went and created the, oh, geez, OK. OK, yeah, so it added the instrumentation. It created all of these resources, the Basin pipeline, the catalog. And if I just tell it to deploy, it's going to deploy this onto Cloudflare. And we'll start collecting that data. So that's Basin. I think we have time for a couple of questions. Anyone? Yeah, I think someone's coming around the mic. Thank you. I have a question on verification. So generally, largest companies, for example, if you ask what is revenue, three departments have three different answers. So do you have anything like how will the agent sort of know actually this is the number? Yeah, I mean, that is, I would say, the fundamentally hard problem. It's really challenging data in particular because you can get an answer. And it can look correct. But what really matters is, is it correct? And that can be quite hard for the agents to understand. I think what we've seen be really successful is when you sort of have the end-to-end being visible to the agent, when it can actually look at the code that is collecting the data, and it can look at all the data pipelines that are transforming the data, it actually has the ability to understand. When I see this particular field at the end of this whole process, what does this represent? But I would say this is still a fundamental challenge that everyone in this space is kind of trying to figure out. And yeah, I think still an area where we need humans to really understand, like, what does our data represent? And what do these mean when we're competing over them? Well, my auth expired since I tried this earlier today. But cool. Any other questions? I have two questions. It's going to feel a little more like a feature request than a question. And the first one is, do we have regions for this yet or not? Yeah, so all of this is regional in the sense that it runs across all of Culture's Edge. But I think specifically what you're asking is about jurisdiction support. So being able to limit where compute and storage lives to a particular EU jurisdiction or US jurisdiction. Not available yet, but coming in Q4. Cool. And well, the second one is, is there any plan to do something like we have workers and workers for platforms? So I'm kind of picturing for our specific case, are we going to have some sort of way to keep our platform users isolated instances, like basins for platforms kind of sort? Yeah, it's a really interesting question. It's not something we thought a lot about, but I definitely see the value there. Can we switch back to the slides? I think we have a couple of slides they want us to show. Yeah, I will be at the speaker's lounge afterwards if you have any other questions. And then please fill out the survey. It will make our events team very happy. Hi, so I would like to know, because the demo didn't have outputs, right? So my question is that if we would like to do that process again, do we need to set up any configuration in our website in order to use Basin? Or the Basin pipeline will basically set the whole process for us? Yeah, so the way you get data into Basin is either via an HTTP API. So you can integrate that with anything. Or if you're using workers or workers binding, which is the easiest way to kind of get data in, I would say, if you're on Cloudflare. So yeah, you'll need to instrument your application basically to send those events into Basin. Alternatively, with LogPush, you can get a lot of data. So if you're using workers, we have workers logs that can flow in, or HTTP logs. So you can get a lot of useful stuff just hooking up that integration. I mean, LogPush is actually available to everyone now. It used to be enterprise-only. So that's pretty cool. Cool. I'll pick up. For the interaction between Cloudflare access and Basin, Peter and I just called the role-based access control. But I need to walk into our environment to see performance reviews that no one else should. Not today, but it's coming soon. Cool. Thank you so much. Thanks. Thank you so much.