Artifacts: Git-compatible versioned storage, from zero to beta in 3 weeks

on Cloudflare TV. OK, I see my timer has started going. So we're going to jump into this. As you may have just heard me said, I just finished my slides, rewriting some of the tail end. So we're going to go through this experience together for the first time. So hang on. So quick poll, does anyone already know what Artifacts is or is using Artifacts? Oh, hell yeah. OK, so all my users are here. This is great. Cool. So today, I want to talk about Artifacts, obviously, and how we basically went from no product to being in closed beta in three weeks. And it was a trip. So to set the scene a little bit, before I was working on Artifacts, I was on the dashboard agent team working agently. We were working on changing a bunch of architecture for that, rewriting it, getting ready for Agents Week. I'm sure many of you have seen us yapping around our Agents Week, it's kind of like very high pressure, high rush. Everyone shifts everything all at once. So my mind was very in the zone on, I need to really make agently a great experience. And then I was sitting on my patio on a Friday afternoon, and I get a phone call from the one and only Dane Connect. He says, how soon can you stop working on agently and hand it off to the rest of the team? And that was a alarming thing to hear. He then told me that we had a large customer that had come to us with a need for, basically, Git and code storage built on the edge. So just a quick outline of what I want to try to get through here is, why Artifacts? Why is it on Git? And why now? Kind of a deep dive. I would love to go longer. We only have 18 minutes left at this point. Giddy, which is our internal engine that we have implemented Git in, we recently just completed a refactor from Zig and Wasm back to JS. A mistake I made early in the project, and just some takeaways and lessons from having built this whole thing out on Cloudflare and basically AI writing all of the code. So why Artifacts? Why Git? And why now? As I mentioned, a large multi-tenant customer came to us and had a strong need for version storage on the edge. Cloudflare had talked about this over the years, but the time, the problem was never right. But given some of the struggles from other Git providers and the nature of needing small ephemeral, not ephemeral, but small repos that agents can operate over quickly was a growing need, and it was the right time. We chose Git because Git is already in the training data. I feel like this is starting to be less of a problem as agents become more capable, but we started this basically when Opus 4.5 was the main model out there. It's ubiquitous. It works everywhere. Everyone, every developer already has Git clients on their computers, so you can interact with it there. You have Just Bash, it gets in there, so it kind of just works anywhere you need it. And something kind of interesting over this is we found out that it's like better, it's good for things more than just code. It's good for just version storage in general. And yeah, agents are writing more code than ever. So we had two deadlines. We had three weeks to get to agents week for closed beta, and then this customer was launching their own products closed beta in four weeks. And one of the constraints that this customer came to us with was that they needed to be able to create 1,000 repos per second for a single tenant in a single namespace. And I am happy to say we did meet that benchmark. You can blast away repos at a given namespace and hit 1,000 per second. So the original team was Matt Carey. He was really focused on the Git engine. He spent his winter break deep in AI psychosis playing with Zig and Git. So we kind of had some work up front. That's how we landed on Zig and Wasm. Thomas Ancorn, he's on our observability team, and just all around Cloudflare Balnor. He helped get a lot of process and teams aligned on getting this out the door so quickly. And then I was working on the control plane, DX, getting the right architecture to scale the control plane, et cetera. And then a whole bunch more people have contributed over the months. So a quick architecture deep dive. And to start with some language here, this is what one of your remotes look like if you build an artifact repo. You have your account ID, the rest of the URL, slash Git, slash namespace, repo name. And a namespace is pretty much like a GitHub org or user. So for anyone, I mean, a lot of you are already using artifacts. So I assume that you know how Durable Objects works. But for anyone, we're going to have a quick refresher. Durable Object is a tiny server with a SQLite database. It can handle about 1,000 requests per second, per instance. It comes with 10 gigabytes of SQLite storage. It's single-threaded. And it has 128 megabytes of memory shared across the isolate. So how do we track keeping track of every repo? Durable Objects. Knowing which repos belong to which customer? Durable Objects. Access tokens? Durable Objects. The Git data itself? Believe it or not, also Durable Objects. So we kind of had four design principles at the start of this. And they were, basically, it would be unacceptable to corrupt a user's Git data. We should never, ever be in a position where we can use, lose, or corrupt a user's data. Stream everything. Never buffer, especially because we are constrained by edge compute, 128 megabytes, 1,000 requests per second, single-threaded. Do as little CPU work in a Durable Object as possible. And we can't put any one Durable Object in the hot path. So this is what the control plane architecture looks like. We have our core API worker. We have a kind of per user, or per account, I should say, we have this thing called the Account Namespace Index. This keeps track of, basically, all the namespaces that that account has, or that has the number of repos in that namespace and the jurisdiction. Then we have the Registry Shard. The Registry Shards, basically, keep track of all the repos in a given namespace. And this is sharded to help absorb that load of needing 1,000 creates per second. Because we can't put any one DO in the hot path, we can't write to the Account Namespace Index in the create path. That would bottleneck everything. We would be constrained to much lower RPS. So every time one of the Registry Shards gets something written to it, marks itself as dirty, alarms, batches new writes, writes that back to the Account Namespace Index. And then at creation time, we also create a token manager. So every repo has a DO that has all the tokens that are minted, like the read or rewrite tokens. They live in a DO. And then the Git server itself is a durable object. And then we have KV just for some nice indexing, for reverse indexing on the token ID. And currently, we're using R2 for forking, but that's soon changing. If you peep the worker resource code, you'll see that a clone API is coming, but nobody heard me say that. So what happens when you create a repo? We hash the repo name, and we basically generate a number zero through four. That's how it picks the shard, right? We write the repo row in the control plane. That gets marked as dirty. It spins up this turd. Basically what I just described. I'm not gonna waste time on the rest of this slide. So in any given namespace, this comes out to a theoretical throughput of 4,000 crates per second. We chose four just because it gave us wiggle room and buffer to hit that constraint of 1,000. So that's currently what we're sitting at today. And obviously, the cost of that, sharding those, the registry shards, is eventual consistency. So you'll see if you create a repo and then try to query a list of all your repos, there's gonna be a delay there. But that's just, it's a trade-off we have to make. The data plane, because we, again, can't put any one durable object in a hot path, it never touches the control plane. So your Git client comes in, hits a worker, that talks to the token manager, validates that, make sure you have the correct access, make sure if you're trying to do a push operation, you have a read-only token that it's gonna fail, things of that nature. And then it goes off to the Git server and streams through. So, what's inside a Git server? Let's see here, we're gonna go through this. Essentially, the stream connects through the worker into the durable object, and we basically chunk it into two megabyte rows in the SQLite database. So inside that 10 gigabyte database in any given Git server, you'll find every file, every delta, every pack file, compressed with ZSDD, all the things in there. So when you push, which streams through, we check each file, and the main compression algorithm that Git uses is ZSDD, and that compresses in there. I should note that today, one of the limitations we have is that artifact repos are constrained to like 10 gigs at max. We are going to hopefully support Git LFS in the future. We have ideas for sharding the Git server itself, or possibly using lazy pointers or refs to R2 and putting worker cache in front of that. Git has the nice property that almost all of the data is content addressable, so very useful for caching. So hopefully we'll be able to support larger repos here in the future. Again, clone basically the opposite, but going in the opposite direction. And we actually have some clever optimization tricks about using deltas and chaining those together rather than storing all the objects together and taking up more memory, which I would love to dive deeper in. We owe you a blog post on the deep dive of our Git engine, and the team hopes to open source it in the near future. So yeah, as I mentioned earlier, one of the core principles is don't lose customer data ever. Don't corrupt Git data. And the way we do that is as the data's streaming in, like you do a new commit and push, as that's streaming in and it's chunking into two megabytes into the rows, we write them to like a quarantine table. And once the full push is in and it lives in the quarantine tables, we basically use another atomic transaction to copy all that data from the quarantine into the main tables and clear it out. So that's how we guarantee that we're never gonna lose your data on a push. If something fails, we clear the quarantine and your prod data still lives in the main tables. So our Git engine, as I mentioned, Matt built it over the holidays during Opus 4.5 Psychosis. It gave us a really nice headstart on, you know, already having something that can compile to Wasm and run in workers. However, as you'll come to see here, our manager, Steve, we rewrote a internal thing from Zigta JavaScript and it got faster and more reliable. The obvious conclusion is that language doesn't matter. Rewriting does. You should be writing everything all the time. And I think maybe at the beginning of the year, I would have been skeptical at a take like that, but now I am fully convinced that that is true. Like, especially if you're in an organization where you have like a decent token budget, like throw inference at improving things and rewriting things and yeah. So where we started. The Git engine basically was a hundred kilobytes of Wasm. So it was very, very small, helped with our cold start times and it basically communicated to JavaScript through callbacks in Wasm. Basically we had the main problems that we run into with Zig over time was that mem copy and data serialization across that boundary really killed us with CPU and memory utilization. So it worked for a couple of months and then as we scaled up and as we added more and more customers into the closed beta, went open beta as more multi -tenant customers started onboarding their own customers to this, we were pushing like probably at the time getting close to 10 million repos. When we started this rewrite, I think we're probably close if not past 20 million repos today. We were getting hit with reliability issues with memory issues. So it was either like write it in Rust and get it into the runtime of WorkerD or try to solve it with JavaScript. And one of those things is much easier than the other. So we did it. Michael Hart, another principal engineer at Cloudflare basically worked, used auto research, used agents and got the full thing rewritten. We basically tested it against our entire test suite, conformance test suites. Nice thing about Git is that there is a lot of tests out there for like compliance or conformance. So that makes doing something like this much easier for correctness. You can kind of just let an agent churn until it's correct. So for a benchmark, WorkerD, kind of a maybe over 10 year old repo now, quite large for the benchmark. At peak memory, we dropped from 359 megs down to 40. A push of an empty artifact repo went from 54 seconds to 27 seconds and the clone time from 17 seconds to eight seconds. And basically the memory kind of stays flat as the repo grows over time. And this is basically like a function of memory building up in the Wasm land from having to do the mem copies and doing the serialization back and forth. So some of you, if you follow me on Twitter, may have seen me tweet about a thorn in my side about a mistake I made. I considered two big technical mistakes I made in my career, one on artifacts, one when I was a very junior dev and I chose Mongo over NoSQL because I didn't understand the difference and lived with the consequence of that for a long time. So the mistake I made was that I forgot to lowercase the repo names when we created it, which led to a situation where you could have repo A and repo A and there are different places to different durable objects. And we never had the time to really go and fix this. Unfortunately, our current customers had the right access patterns that they weren't causing this problem to exasperate. You can kind of see why this is hard as I demonstrated here, like that difference in case sensitivity goes to different places. I knew how I wanted to fix it. I knew the right way to do this. I knew the right architecture. I knew the right design. I knew how to migrate the databases. I never had the time to do by hand. We were working, we had a big push internally to get jurisdiction support rolled out and we weren't having customers run into this problem at this point. Like this, I was like, you know, checking this every day when I logged into work and agents weren't able to do it. Like I probably once a week tried to throw an agent at solving this problem and gave it very descriptive ways of how I wanted it solved. And none of them could get it done along like chunking up the PRs to make sure that we didn't have like version skew. And then Fable 5.1 came out and basically one-shot it. And that's how the problem got solved. So basically the fix was lowercasing the name when you came in. We had to own each of the registry shards add a unique index, making sure that it doesn't clash. And the trade-off that sucks here is that in some of the read pass, we have to basically now fan out and for repos created before a certain date, we check basically the spelling that comes in and the lowercase version of it. So that's kind of how we got around it. There's like a lot more machinery there under the covers which made this hard to do, but that's kind of where we landed. So takeaways and lessons. Small agile teams are the future. That's bullshit. Small AI native teams are the future. Artifacts is built by a team of three in as many weeks. And to my knowledge, we were the first product at Cloudflare that was written entirely by agents. We have not contributed any code by hand. Wherever you can, cut process, friction and slow feedback cycles. Take humans out of the loop wherever possible and build tooling and automation that lets you trust what they're doing, build trust in what they're doing, verify what they're doing, let them verify their own work. And honestly, be more ambitious. The models are getting good enough. If you would have told me a year ago, we started shipping a project like this and then three, four months in, we decided to rewrite a Git engine from Zig to JavaScript, I would have told you you were crazy, but that is the world we live in now. Hot take, maybe not a hot take, but writing code by hand is dead and we should all kind of adjust. And I've talked about this a lot on Twitter, on our podcast, The Next Token that Cloudflare sponsors, like it's a hard thing to grapple with and go through. And I think we actually don't do a very good job at talking about like the human experience of that, but the practicality is that this is the reality and the future we're going towards. And yeah, I think we have to rethink things like code review, to rethink things like CICD. This is where currently kind of like where I think my team is feeling bottlenecks is like we're still very much reading every line of code that the agent produces, but especially with Opus 5.5, we're starting to trust that code more and starting to have to knit it or correct it less and less. And now the bottleneck is very much becoming us reviewing our own code and we get into a situation where it's like, oh, I'll have my agent review my coworkers PR and then it's just like agents going back and forth. CICD, like if agents are verifying their work locally, is it really worth spending all the compute to be rerunning CICD all over the place? I don't know. And also deployments. We're experimenting with building our own basically health mediated deployment rollout where it's orchestrated by an agent, like an agent, like a normal LM and something like Clef give it all our logs, observability traces. And if it detects at any cohorts of like rollout from free PAYGO enterprise, up the tiers, it has the autonomy to roll back. Yeah, and overall, I think Cloudflare, Cloudflare is a great place to have, this was the first huge project I built on Cloudflare. I've been at the company about a year and I've built a lot of stuff on durable objects over the year, but nothing at the scale of creating a thousand Git repos a second. And the primitives have gotten pretty good and are very good for agents, whether that's preview deployments, having queues, having durable workflows, like there's something there to solve most problems. And yeah, artifacts was built on the shoulders of giants. Like there is nothing in artifacts that is built in the runtime. Like any one of you could throw your agent at it and replicate the exact design I had here. And yeah, be more ambitious. Thank you. I don't know if we have time for questions, but. I do not think artifacts is being used inside the core version of Cloudflare OS directly. Don't quote me on that, but I don't think so. I don't, I actually don't know. It would not take a layer of integration. Like you could just wire up a binding and it would be right there accessible in the code the agent's writing, but I don't know if they're using artifacts in the core product. Yeah. That's a, how much time do we have everybody? I talk about this a lot in our podcast called The Next Token, Sunil Pai, Reece Sullivan and I. I think the overall takeaway is that there's going to be more software. And at least today it still needs, you know, someone that has the experience of building software, having taken down prod, having made mistakes, having shipped bugs. Like I have the existential dread of like, how do we onboard new devs into this world? And I don't know what the answer there is, but I know to stay competitive and to increase the quality of our products, because we should at the same time that we're using more inference, we should be using that inference to ensure higher bars of quality, like higher thresholds of, you know, monitoring and tracing and performance and, you know, content layout shift in UIs. But I wish I had the answer. Sorry. Sorry.