Stop Wasting Money on AI APIs： Use Cloudflare Queues

Cloudflare queues recently hit the workers free plan and I want to show you my absolute favorite way to use them by cutting your AI token bills. Now look, I know that sounds kind of clickbaity but both OpenAI and Entropic give you a 50% discount if you use their batch APIs. If a user isn't actively waiting on an AI response, don't call the model online. You're paying full price if you do. Drop the messages on a queue, let the queue ship it off to Entropic and OpenAI and get a 50% discount. It's cheaper, it's resilient, and let's dive right in. Here we have a little demo app. It's a feedback classifier. We can choose between bug, feature request, praise, and billing. Let's start with a bug. We have a web app and the user says the home page didn't load on Windows. Send it to the web app. Beautiful. Okay, a sales call. The guy was really nice. Great. Another one. And see how fast this is. We're not waiting on a response. And finally, a support email. You need a haircut. Okay, let's go to D1 and here are our entries. The home page didn't load, the guy's really nice, and they're in progress. And on the left you see a cron job and you'll see it's processing all three. It's in progress. Because batch APIs are asynchronous, we use a Cloudflare cron trigger to pull for completion. Once the batch is ready, the cron job pulls the data and puts it right back into our D1 database. So while this is churning out, let me show you some of the code. It all starts, of course, at Wrangler. And here we have our cron trigger. This means it runs every minute, which is good for a demo. In real life, you'd probably have it run a little slower, maybe every hour or so. Let's go to the implementation of the cron job. It's on scheduled. This gets invoked every minute. We do some logging, we get all the batch IDs that are in progress, and then we pull Anthropic. And of course, this code will be available on GitHub. So that's the cron job. Now, how do we actually get things on the queue? It's with the producer. And the producer is our web app, feedbackqueue.send. Here you can see we have this API call. This is our API where we have handle feedback submission. We look at the feedback, create an ID, and then we insert it into D1. Then we put it on the queue, and then we just return the JSON. That's how it was so fast when I added the feedback. We talked to our producer, we talked about the cron job, now we need the consumer. This is the queue function on our worker. And it just takes a batch. And a batch is a list of those messages. I've set in our consumer, I've set the size to three, that's why we needed three messages for it to run, and a timer to 60 seconds, whichever one hits first. If batch size is first, then it uses batch size. If 60 seconds have passed, and there's only two items in the queue, then let's And we have a maximum of three tries. So we go to this process batch mode. And all it does is it gets jobs, we turn them into requests, and then we just send those requests to the Anthropic API. If there's an error, then we retry. If there's an error more than three times, it goes to the deadline queue. If it's successful, we mark the job as submitted. And then very important, we acknowledge the message here. Let's see what we got here. It is, yay, it worked. The batch is completed. We pulled after a couple of minutes, it was completed. So the web app, the homepage didn't load on Windows, that's a bug, high priority. The sales call, the guy was really nice, praise, but low priority. And the support email that I need a haircut, I mean, I know, I know, is low. So this is cool because we're only paying half price for this. Because batch APIs are asynchronous, we use the Cloudflare cron trigger to pull for completion. Once the batch is ready, the cron job pulls the data and drops it in our D1 database. And if I go to our D1 database, you'll see it has upsetted and we have the categories, the priorities, and then a summary, what the bug actually is. But this isn't just about money. It's also about reliability. LLM APIs are flaky. We get rate limited, we get overloads, transient errors. This is exactly what queues are used for. That's why we're using max retries. Let me show you again here, max retries, three. If we hit the max retries, then it goes to a dead letter queue, which is probably one of the coolest names for a paradigm. That means even if we retry it three times, if Entropic errors out, our consumer will just back off and retry the whole batch. If Entropic is down for a longer time, it will go to the dead letter queue and you actually never lose a message. Now that's pretty crucial. You just never lose a message because it will either be processed or on the dead letter queue. To summarize, if you're building with AI and the work doesn't need to happen in real time, use queues. Save some money with the batch API. Cloudflare queues are available on the free plan now. You get 10,000 operations per day. If you're on the workers paid plan, you get a million operations per month. After that, it's 40 cents per million operations. I hope this was useful for you and that you can save some money on tokens. I use queues all the time. They're one of my favorite and underrated products. I hope with this video, you're excited to try them too. The repo is in the description. I hope you build something awesome. I'll see you later.