It’s time to build, not buy

Right, folks, I think we are good to start. I have been scheduled over lunch, so there may be some people walking in late who are a little bit fuller and have had some lunch. I won't pretend I'm envious of them. But hello, everyone. My name is Ryan Skidmore. I work here at Cloudflare, and I work on developer productivity and AI, which is a very broad remit, but it basically means I work on all of the stuff internally to help make our engineers as productive as possible and also safe as possible when using things like AI. And, yeah, that's pretty much all I do every day, work with our engineers and, you know, help them be as good as possible at their jobs. And this talk is about building software instead of buying it. And that has gotten a lot more realistic over the last few years. So we have some audience interaction now, which is always fun. So can I get a quick show of hands? Who here or, you know, your company, your employer, who pays for software that your team actively works around? Quite a lot of people. Okay. So now put your hand up if you've ever thought that you could build a better version of that software over a weekend. Yeah, still a lot of people. And that used to be something that was a little bit more unrealistic. Maybe a pipe dream, some would say. But actually, you know, now the idea of sort of rebuilding a piece of software on a weekend is actually pretty feasible, and you can, like, you can do it most of the time. So we're going to talk about what happens when, you know, you have that capability and what that means for the sort of the workplace and the enterprise and also personal stuff as well. So if you have written software for more than one week, someone has probably told you that you shouldn't reinvent the wheel. And that might have been, you know, in sort of a design review or it might have been, you know, in a sort of one-to -one or a conversation or something else. And the main, you know, advice behind it, which was really solid advice, was that, you know, the wheel already exists. So, you know, if someone else maintains it, your time is better spent, you know, on pretty much anything else. Your time is better spent on building the things that only you and your team can do. And it was really good advice because the math was often sort of pretty lopsided and skewed. So, you know, building meant months of sort of engineering work, and you often didn't get any value out of it until those months of engineering work had been spent. You have to have a team to run it. And you're on call for it as well. And all three of those things kind of suck. And the benefits of buying it were you could buy it and you would have it live within like a week or three months, depending on your vendor. And someone else is on call for it, which is also great. No one gets woken up at 3 a.m. or something silly. And buying also had its own costs, which, you know, they're starting to see a lot of price inflation across the SaaS industry. This was, I wouldn't say a random stat, a carefully found stat that, you know, SaaS prices have increased 16.4% over the year. And a lot of SaaS licenses end up going unused. So, like, 65% of people have a license for something and they just never use it because license management is hard and, you know, painful. And 79% of ITVs have been hit with IT price rises at renewal. That all sounds like it kind of sucks, to be honest. So, now we have some really great capabilities. In this time frame, we're talking from March 2023 to April 2025. The price of input tokens for LLMs has gone down by about 75 times. Now, this isn't like an apples to apples comparison. But this was using a benchmark called MMLU. And both of the models here scored about the same. And as you can see in that two-year period, the price to essentially achieve that has gone down by about 75 times. So, you know, you get the same outcome, but 75 times cheaper. And the more interesting thing is that agents are now finishing longer tasks as well. So, this is essentially, you know, what your money gets you. So, there's something called meter, M-E-T-R, which measures the length of a task in human expert time that a model can finish on its own 50% of the time. Which is a very confusingly worded benchmark, but works pretty well for this. So, GPT-4 managed to do this in about four minutes. And Claude Opus and some other models, you know, they've managed to really get high. And now, you know, you've got Claude Opus 4.6, which can work on a problem for sort of 12 hours straight. And the benchmark actually hit its limit with models like Opus 5 and Opus 5.5, where it actually can't measure it anymore. Which is kind of crazy to see how in such a short period of time, we've, you know, basically exponentially grown the tasks that AI is, you know, able to handle. But we've talked about writing it. Writing it is kind of the easy part. You spend a weekend writing it. All looks good. But what we haven't really talked about is the owning it part. Because when you build something yourself, that means you also have to own it end to end. And that means running it, fixing it, and replacing it. And getting woken up at 3 a.m. when it subsequently breaks, because your entire company is now dependent on it. So, the answer is almost always yes. You can build it. But you should start to think about should we build it? Because one of the trends that we've started to see is we've now entered like a world of essentially personal software, where everyone can build a piece of software that, you know, does exactly what they need. And that's great. That is amazing. The problem is when you have a company of a thousand people each building personal software to do exactly what they need, that makes it a little bit more difficult. Because you have essentially a thousand versions of slightly different but kind of the same software. So one example we're gonna go through is something called AI code review. Now, some of you may know this example, because we've blogged about it on our public blog before. But this is our internal AI code reviewer that we use on merge requests. And I was the one that built it, so I am perfect to talk about it, because I cannot stop yapping about it. So code review is really great for catching bugs and sharing knowledge. But it's also one of the most reliable ways to slow down an engineering team. If you've ever, you know, created a merge request, you know that sometimes you may be waiting 8, 12 hours to get a review, and that can really slow you down. No one really, you know, wants to be slowed down like that. You know, your merge request will sit in a queue, and then someone context switches to read it. They might leave a few nitpicks or small bugs. You go fix them, and then rinse and repeat, and things take days to be merged. And this is something we felt internally as well. Before we had this, you know, the sort of median wait time for a first review was often measured in hours. But now we found that it's actually measured in, you know, minutes, and we really love that. So we didn't start by building something. We were very early into our sort of AI journey, I guess you could say. So we tried some off the shelf AI review tools. And some of them were really good. Some of them were not so good. But basically none of them were flexible enough for what we wanted. We sat in vendor calls, and we talked about, you know, all of our requirements and things like that. And, you know, we had sort of salespeople saying, yes, we can 100% do this, but that doesn't come with any guarantees. And we just found that that level of sort of, you know, lack of control was going to be really problematic. So we just sort of hacked something together to start with. We generated a Git diff. We had a really quite dumb prompt, and we just told it, like, go find some problems in this code, and then let me know. And honestly, it wasn't that effective, but it was better than nothing. And since then, we've really overengineered our AI reviewer based on the open code harness, and we've been using it for pretty much everything for the last six to nine months. And now it's a bit more than a Git diff and a dumb prompt. It's something that is, you know, really a strong tool in our arsenal. So we built the first version in, like, an afternoon. I kind of looked a bit through how Claude Code was doing their review tool, figured out that they were doing some, like, MCP magic and stuff like that, and I built our own version on GitLab using open code. And we've since had� I think this number is actually a little bit higher now. It's requests to our internal AI reviewer, and that's not just from me and my team, but it's from people all over the company. And that's one of the great things about, you know, building your own internal software is if someone in the company sees a bug with something or sees a problem or thinks the behavior should be different, they can just make the change. And with AI, that's incredibly easy. And what we've seen is there have been hundreds of people that have chipped in to fix bugs and add extra functionality and really sort of elevate the internal experience to be faster, more secure, et cetera. So we've published some stats previously in our blog post about, you know, what this is sort of� the impact it's had on our engineering. And I think probably for the last month, every� or last nine months, sorry, every merge request has had to be approved by the AI reviewer. And you can imagine how angry engineers get when they have to fight with an LLM. It turns out some of them actually get really quite angry. But we've only had to block about 16,000 merges. And that's over the past four months. Which isn't that much in the Cloudflare scale. And what we found is that when we do raise these issues, most of the time people do act on them. And if we run into issues, if things start failing, we're able to fail over quickly. And if everything catches on fire and an AI provider goes down, there are, you know, break glass mechanisms to make sure that things carry on working. Because we still want to be able to ship code. So since March, we have run just over a million reviews through our internal AI reviewer. We don't know how that stacks up against the public ones. But I think we're pretty proud of that number. That is a pretty big milestone for us. And I think that's really highlighted that, you know, people trust the tool. It's working. It's doing its job. And, you know, we've had success by building our own. And that's across about 350,000 merge requests and just under 7,000 repositories. And we actually published a blog post yesterday about some cost savings we made to the AI reviewer. And now it costs us just 67 cents per review. If you're interested in checking out that blog post, it's on the Cloudflare blog. We managed to reduce our AI inference bill by 68% by just being absolutely ruthless with optimization. So go check that out as well. Because that's a super interesting case study. So one of the things that we ended up iterating on was that in the very first versions, we flagged everything we could because we thought it was useful. And then we listened to a lot of feedback from people and found that actually we were raising a lot of issues which were kind of like nonissues. You know, nits, small things. Things which we probably shouldn't be doing. So we went back to the drawing board a bit and we decided, okay, let's only flag things that people actually care about and that people are actually going to action. And when we did that, we found that actually people action those things more because they're more important and they're more valuable and they're a good use of people's time. So the thing that we can't do with the AI reviewer, and this is where, again, we had to take a step back and figure out, you know, is this the right tool for the job? Have we built the right tool for the job? Or do we need to extend the tool? The first one is that because of the size of some of our repos, we can only really look at the Git diff. We can't really look too closely at, like, the design behind it. Sometimes there is no design. Sometimes it's a simple change. But other times, you know, there may be a whole design dog behind it, but we just can't, you know, pull that in at runtime with everything else. It's difficult to check cost across system impact. So because we may have potentially up to, like, 100 services that are in, you know, a path that this changes in, we certainly could check that, but it would probably take about three hours and the person making the MR would probably be really quite annoyed that we're delaying their MR by three hours because of it. And there's things like time-dependent races where you have to run sort of race checkers and things like that. It's really hard to spot those in a static diff. Sometimes you just need to run the race checker and things like that. And very large changes. Some of our MRs are really, really big, like millions of lines, and it's very difficult to figure out a way to put that into the context of a model that only has a million token context. So you have to do some crazy stuff to get that to work. But we found it doesn't replace human review. No matter how hard we try, humans will always have really good input. So where this took the conversation is it really gave us confidence that in the right circumstances it is a fantastic idea to build your own software, because you can build it to do exactly what you want, how you want to do it, and when you have a request come in or when you want to support something new, you're not then dependent on a third party, a vendor, et cetera, to go build your thing. You can just build it yourself. But you should also ask, you know, should we build it? Just because you can build it doesn't necessarily mean it's the right idea. And you should try and prototype it. That's one of the big things that we found for our software is, you know, it's difficult to commit lots of people and lots of time to a project that's a very big maybe. So, you know, spend an afternoon prototyping it. See what you can slop together. And then show it to people and say, hey, this is what I built. Is this something we want to invest further on? And then later, expect your tools to be replaced. So one of the big, I guess, anti-slop kind of things that I've heard is that, you know, you're just building tools that are full of tech debt and it's going to make everyone's life a hell. That's probably true. And the way that you, you know, shift your thinking about that is, actually, if it's really easy for me to build a tool like this, I can build the replacement in six months with better models in probably half the time with 100% better features. So, actually, I don't need to hang on to the idea that this software is going to be around forever. It might only be around for six months. But because we're solving the problem now, we're actually probably saving a lot of human capital and human time in solving the problem now. So when building makes sense, it almost always makes sense to, you know, build it if it's sort of critical and there's no product on the market that really does what you want. That's like a, you know, that's like a golden egg. If it's critical, but there's something on the market that, like, really, really perfectly fits what you want to do, buy it. But make sure that there is an exit from the vendor. The last thing you want is to be vendor locked. And if there's an exit from the vendor, you can then move to your own tool when your requirements have changed, your business has grown, and you need to do different things. If something is, you know, not really that critical, sometimes it just makes sense to buy it. And, actually, the conversation there is more, does it cost us more to buy it or to just spend the tokens on building it? And, unfortunately, tokens are getting really cheap. So that's kind of changing pretty rapidly. And the final one, build something small. If it's not critical, then you know what? Just try and build something and see where you end up. So some checks before you build. Try and actually, like, figure out what's wrong with the tool you have. So if you're using a lot of legacy SaaS products that I don't think I'm allowed to name, you know, try and figure out what's wrong with them and then figure out if that's a good reason to replace them. Build something small enough that you can just throw away if it doesn't work. Don't spend five weeks on it. Spend an afternoon. And make sure you're also considering, like, the security, the, you know, the replacing, the maintenance, the on call burden, all of that kind of stuff that comes with it, too. And reuse what exists. So I reused a lot of stuff that existed for us. So open code, worker's KV, AI gateway, all of that stuff really existed so we could just reuse it. And make sure you measure it as well. Make sure you have telemetry, measurements, everything you can possibly think of. Because that will come in really useful when you're quantifying, okay, here's the impact of the thing that I've built. And then finally, keep a way out. Make sure you have an escape hatch to, you know, leave if things go really badly. So building got cheap, but owning it is still kind of a problem for now. But that is changing. So think about the why before you build it. And that is it. Thank you very much, everyone, for coming to my talk. This was my first talk, so I hope I did a great job. There's a� thank you. There's a link to our original blog post there on our AI code reviewer, and then there's a recent post that we made about, you know, about the changes we made. Go give it a read. It's super interesting, if I do say so myself. And then I will also be in the speaker connection lane after this, if you want to come talk to me about basically anything. And don't forget to give some feedback. And we have one minute left. So if anyone can get a question in one minute, you're more than welcome to. Right here at the front. Okay. So you've got an AI code review bot, and it is essentially the same model that the developer is probably using on their laptop. So wouldn't you consider any finding that happens in the PR to be already wasting the developer's time? Like, is there a reason why� I mean, yes, it should run there too. But shouldn't all of that same skills, prompts and context just be part of the developer's precommit? Ideally, yes. And if AI was deterministic, that would be fantastic. But that's one of the problems is AI is almost, you know, never deterministic. So we have to take kind of an adversarial approach where you have to say, you know, not just review the code, but review it from, like, five different angles. And that's when we build in a lot of extra tooling and prompts and, you know, extra stuff to essentially make sure that we're reviewing it from, like, all of the different kind of angles that we can find. And we've actually gotten a local version of that same AI reviewer that people can run locally. But again, because of the nondeterminism, it makes it a little bit difficult to track everything all the time. So with regards to the skills and things that you set up for your developers in relation kind of to what he had just mentioned, how heavy is that footprint for your teams right now? When you say adherence to engineering standards and things like that, how much of that is on the machines and being leveraged as part of the build out versus then being checked? And is that something you guys track internally where it's like, yeah, we have a library, a set of skills, a set of things that everyone should be leveraging in these development tools to keep our standards in place? Yeah, 100%. So the internal standards one is one of our big focuses at the moment is how do we help AI models generate code that is naturally adherent to those standards instead of doing it after the fact in review. So we have things like linters, for example. We have a bunch of RFCs, which include basically all of the different standards that they should adhere to. That's, you know, past agents' skills. They don't necessarily always get invoked when they should do because, you know, sometimes it's difficult for agents to figure out. But we really try to bring that back to, like, the developer machine so that it's, you know, when the code gets pushed up, it's as ready to go as possible. But what we've also found as well with recent models is they'll figure out that they need to wait for the CI to complete once they create a merge request. They'll see the comments and then they'll just go fix them and push them back up and, you know, loop until everything's good to go. So what I've found is, you know, especially with models like Opus 5.5, for example, I've not really needed to guide it that much to get it into a good sort of production state. It can just figure it out most of the time. Sweet. I think we are� I think we are done, then, folks. Thank you very much.