I Built a Voice AI Agent That Runs on Cloudflare Workers

I just built a customer support voice agent. You can talk to it, and it can also call tools in real-time. And it's all built with Cloudflare. Here's how it works. Hi there, welcome to Acme Support. I can help you track an order start, a return, or check if a product is in stock. What can I do for you? Can you look up order 9831? I found your order. It was delivered, and it included two pairs of... Yeah, can you just initiate a refund for it? I can certainly help with that. Could you just let me know the reason you'd like to return the order? They just don't fit me. I started that return for you. You should receive... Thank you. Also, can you check if you have any notebooks available? Yes, we have classic lined notebooks from Paper Vault in stock for $26.79. Would you like me to help you with anything else? What was my first message again? Your first message was asking if I could look up order 9831. Okay, thank you. You're very welcome. Feel free to reach out if you need anything else. You must have a great day. And you saw how it was able to persist the context. It was able to invoke tool calls to third-party services. And it also handled interruptions very gracefully. All of this is possible with the new Cloudflare's Voice SDK. The Cloudflare slash Voice SDK comes with a lot of nice little mixins and abstractions for you to use in your app. We have With Voice. It's a full voice agent with speech-to-text as well as text -to-speech. Then we have With Voice Input. It's a speech-to-text only option. So you get transcription without a response. We also have React hooks that go along with this very nicely. And there's also a framework agnostic client called Voice Client that you can use in any JavaScript framework. Let's dig into some code. Very important With Voice. Workers AI Flux STT. It's the deep layer model that comes with the Workers AI platform. So we don't need any API keys to use them. And we have Workers AI TTS. And the types for the turn server. You're also using AI SDK to invoke tool calls, which I'll explain in a bit. Then we are also importing Create Workers AI, the Workers AI provider for AI SDK. And then we also need Zord for creating these tools. So let me show you how the app actually looks like. We have a class VoiceAgent, which is extending the BaseVoiceAgent class. We have a transcriber and a TTS method. And we have an OnCallStart hook that will run before the call starts. So you can use this to say, send a greeting to a user. You can even use it for analytics. So anything that you want to do right before the call starts. And then we have the OnTurn method, where a lot of these agentic loops run. So all of the tool calls that we have written will run within the OnTurn method. So I'm actually using the Gemma model from Workers AI for the LLM conversation. I also have a system prompt. So yeah, this is like a nice little system prompt. I'm also adding a little bit of normalization to the prompt. So in case if the order numbers or orders come in as strings, I'm just prompting the agent to convert them to numbers. So it's normalized and the fake API that I've written will automatically pick it up. And yes, I've also added DoNotReturnMarkdown because sometimes Gemma kept returning markdown and I didn't want my voice agent to respond back markdown characters with voice. And yes, we have some text and some steps. So okay, let me go to the Ecom tools that I've written. So I've written three tools, LookUpOrder, RequestReturn and CheckProductAvailability. So one quick tip is always give references to your tool descriptions. So it's easy for the models to pick up the right model because sometimes they hallucinate and keep missing on picking the right tool. So for example, I have given the order ID example format so that it picks up correctly when I see or the user says, can you look up order 1, 2, 3, 4 and it will automatically pick this to me. Similarly, I have a RequestReturn and then there's also CheckProductAvailability. These are quick small endpoints that I've written for creating a fake API slash CRM of sorts. And then we are creating a base voice agent class and using it, I mean, extending it on the voice agent primary class. And these are the tools that I am passing to the workers AISDK, sorry, the Vercel AISDK is generate text. And I also have a stop when the step count is five. This is a native Vercel AISDK feature. And these are just some logs that I have. So whenever I, the tools we are getting called, I just added these for debugging and that's it. This is everything that you need to create a voice agent. Let's look at the UI side of things. So I have the small little react page. I am importing the UseVoiceAgent hook. So I'm using a few little things. It's exporting. One is the status one. So then the voice agent is IG or listening or thinking or speaking. The status flag will give you that value. So let's actually look at what's inside the status. So it's IG, listening, thinking and speaking. So yeah, that's right. And yes, then we have transcript. This will contain an array of all the messages the user and the assistant are conversing. And then we have audio level, which is for the mic volume level. If you have some sort of waveform that you want to show or animate, you can use the audio level. And the last thing is muted is a separate flag to mute the mic and not send any voice to the agent. The start call method and the end call method. If you want to start and end, yeah, like the name says, then we have a toggle mute button to just toggle the is muted flag. And this is just a flag to know when the agent is active or not. It's right here. So if I click on it, it just toggles the start call and the end call method. The transcript.mac actually has a list of all the messages the users are trying to converse and it's reactive. So you should be able to update the UI in real time. And there we go. We learned how to create an app with voice agent capabilities and it works in real time. We also learned how Cloudflare slash voice SDK works. So let me know in the comments if you are building something with it and if you have any questions, drop them in the comments.