Build a conversational form with Deepgram ｜ WorkerShop

See what happens. Hello, everybody. Welcome to the first worker shop. Genia, thank you so much for joining me on this. This little experiment that we're doing, we are not, this is not a webinar. This is a worker shop. We are going to build together today. We're going to, I've been working on a thing and I needed some help from Genia. Genia is coming in from DeepGram. Anybody out there in the chat using text to speech or, or the other way around speech to text and has not used DeepGram, it's time, it's time for you to use that. Right? Yeah, actually, like if you're interested in it, it can be just plug and play, or you can just follow along with us in this workshop, worker shop, and see how you can build these things and how you can make your app listen to you. Absolutely. Yeah. And, and it's called worker shop because it's a Cloudflare worker, right? We build these web apps and I've built, I've gone and I've built a web app. And I was working through it and I was kind of realizing that maybe the way that we do things is changing a little bit. I've been, I've been thinking about this project. I don't know how many people out there have this, like this idea of what a personal readme is. I feel like we're, we're in this time, Jinya, where like, we don't really know people, right? Like we, you and I have spent some time on video calls, but like, I don't know, I don't know how you like to work really. And so like, maybe I'm going to send you an email. You might not like email. Maybe that's not how we're supposed to do it. That, that happens all the time. Yeah. Because it's not just in one country. Like sometimes we even need to collaborate with different people across time zone, across the globe. And I want to be as much accommodating as possible to my collaborators, but sometimes I just don't know their working styles. That's where if they have a personal readme would be super helpful for me. Yeah. Even the other way around that if my collaborator knows my working style, that will be easier for me to collaborate. Yeah, absolutely. And I think that like, it's nice for, I've seen managers use them too, where they're like, this is the style of how I want you to interact with me or the reverse of like, I would like to know how you would like feedback or things like that. So it's a concept that exists out there. And I started thinking about it. I was like, Oh, that would kind of make a fun, that'd make a fun little app where we could kind of store stuff out there. Because I think, I think what's going to happen here pretty soon is that it's not going to just be people talking to us, right? It's going to be agents coming in and like how agents interact with us as well. We're entering a new world where it's not just humans in this space that are going to be reaching out and talking to people. So I was, I was kind of trying to think about that, about like, what would that look like? And I started noodling on it and I built a thing. I built a little thing out there. Pretty cool thing. Yeah. Yeah. And then I brought, I brought Genia in to make it even cooler. And I started with this concept of like, we're, we're in a time where we are generating so much code and oftentimes we go and we build the whole spec of things. And I tried this. I wanted to try this approach with this app where I was like, Hey, go out and research what people have been saying about personal read means. And then I want you to build me an application that helps the stuff come in. So it went out and built, I built this application. You know, it's, it's experimental, but I think that this is pretty cool. I like, I like where it's, where it's gone. And, uh, I am so happy that I was able to reach out to Genia to help me even more with it. As, as we go, I'm going to share my screen here, share my screen and share this guy over here. And I'm going to kick off a brand new profile here. And this is using the, the Cloudflare agents SDK under the cover. So I'm going to say, we're just going to do this one live just in case I had already made that, uh, before. So I'm going to click this open. We are going to be looking here at this, uh, uh, read me here. We are, are, uh, I, so I said, I said, just go and do it. And it made it a pretty complex form. Right? So, so I like what it did. I like the questions that it did. It's asking like, what are your preferred channels? You know, there's these check boxes I can go and I can click. And it's like, there's some compute, how do you communicate? What's your role? And it decided, you know, obviously I need to know what your team is and that sort of stuff. And I started thinking, I don't want to really think about this and fill it out. Right. This is forms like this are daunting. And I feel like this is kind of in a place where it was starting to go away. So, so now I'm on stage two of my thought was like, man, it would be kind of cool if I could just tell it what is happening. So I, I took a swing and I used an LLM in the background to say like, uh, my name. My name is Craig. And then I apply this text update to it. And I say, I am a developer educator and I do this text update. And I was hoping that what would happen was I could kick off a little thing in the background, run some LLM text on it. Oh yeah. And it would update my thing. Right. So I'm updating that from text. And then I started thinking a little bit more about it. I started thinking like, well, wait a second. So, so it's running. So one of the things that I was thinking about is I might want to type a lot of stuff and like not look at the form even, and just let it go. And it doesn't really matter. Eventually that's going to happen. And I kicked off, I started doing this with the agent. The agent framework has this, this concept called workflows. And I know that sometimes the LLM is going to get choked up. We've been playing with these LLMs. We know that like, sometimes you're going to run into some problems where like you, you send some text, it runs a function. And maybe that thing hits an API that gives back a 429. So you need to like back off and like, you need to retry if you want things to happen there. So conceptually, what I thought was like, oh, I should run those through a workflow. And I wanted to, you know, I I'm testing to make sure that I can kind of like check and uncheck stuff. So like, I like to use Slack and email. And I wanted to like do this where it automatically would go and check the things. And then if I changed my mind, I could undo it. And so, so this is kicking off one of those workflows. And if I look here, you can see that my workflow is running and I'm doing this extract a profile patch. Now I can queue a whole bunch of those things all at once. So I could kind of go and it would just kind of go in and edit, edit things as well. So I was doing that and I started, so this should go. It takes a little bit of time, right? So, and that's on my side. I just want to make this clear before I hand this to Jin-Yeb. I'm playing with trying to use a smaller model on this. So I'm using one of my favorite models, actually, in Workers.ai. I'm using this GLM model. So Workers.ai, we host models that you can go and use. And one of the models that we host there are these DeepGram models, which is how Jin-Yeb and I know each other. So there's several DeepGram models that we host. And I was like, oh, wouldn't this be cool if I could actually just talk to this form? And I kind of, as I started thinking about that, I think that might be what the future feels like. And so I called up Jin-Yeb to say, hey, Jin-Yeb, what should I do here? And so I'm so glad to have Jin-Yeb here to explain a little bit about what is going on and what is DeepGram. Let's back up a little bit here, because I'm coming in with a little bit of knowledge. But let's bring this to the audience here. What is DeepGram? So DeepGram is a voice AI company. We do everything with voice. So what does that mean? So as a human, when you and me are interacting, what we do is kind of I listen to what you are asking, like what is DeepGram, the question. In my head, I'm kind of processing it. And then I'm replying back to you. So when I'm hearing your question, that's actually your speech transcribed in my head. My head is generating a response. And then I'm basically replying back to you. That's text-to-speech. We think that our head actually generated the text. Computer Science Analogy. The first one, when I'm listening, that's speech-to-text. When you're thinking, that actually LLMs. And when I'm speaking back, that's text-to -speech. So when I say DeepGram does everything with voice, that is basically encompassing these two parts, speech-to -text and text-to-speech. So if you want to know what someone is speaking to you, you can use DeepGram to live transcript, to batch process. You can also use it to just pass it a text, like response from your LLM and speak back to you. That's kind of a rough overview of what DeepGram can do. And what I do is basically work with my client so that I can learn their actual business problem or use case they're trying to solve and then create a voice AI-based solution for them. Awesome. And I think that there's a lot of benefit to these agents that we're starting to have, right? I think that we're probably going to be talking to them. You're seeing this all the time. I'm sure that your customers coming in here is like, agents need to speak. Yeah, right. And yeah, I think that can be actually quite helpful. So just a few days ago on a weekend, I needed emergency medical assistance. And instead of going to the urgent care and waiting for hours, I actually chatted with the one medical bot. Now, and it was actually able to solve my issues within like half an hour, which is pretty impressive with respect to the wait time. But what if instead of chatting, I can just talk with it? Right. Oh, when you said chat, I actually thought you were speaking with, but you meant like text -based chatting. No, I was just texting. Yes, I was texting. Right. Basically, it's a back and forth chat, how we do say in Slack communication. I was considered that instead of my doctor, I was slacking a specialist who is working on behalf of my doctor that, what is your symptom? Okay. I'm refilling your prescription again. That's it. Yeah, right, right, right, right. So anyways, if you are like me and want to do less and like to make your program work for you, maybe you also want it to just talk to it instead of inputting the text by hand and in your keyboard. And that is exactly where DeepGram can help me. And basically, yeah, if you have any questions, please feel free to stop me at any moment. I think what I just accidentally did there was an interrupt. And I feel like that's something DeepGram offers as well. Yes. Yeah. Actually, this is one of the very crucial things. When you're speaking, especially when you are speaking with a bot, no one wanted to be interrupted by a bot. Human, we may tolerate it, but a bot interrupted me in between a sentence? No, I don't like it at all. So I'm not sure many of the users might not like it at all. So you want to know when the user actually complete his or her sentence. That's like end of term. End of term, what does it mean? That you have a high confidence that the user is finished. And so now you can start your thought process so that you are not working with half-baked ideas, number one. And number two, you are not deteriorating your user experience. And also you can actually reply when the user is actually completed his or her thoughts. From where it is coming, it is actually coming from DeepGram's Flux, which is our latest speech to text model. And it's the end of term event. So that basically signals the system that it is time to process. So it is not going to wait too long where the interaction feels super slow. Or it is not going to respond too early and interrupt the user and make the user experience unpleasant. Right. And so when I showed Jeanette, she's like, you should pick up Flux because of this end of term thing. And it's really powerful. Thank you for pointing me to this model. It's available on Workers AI. So this link here, you can come take a look at this. If you are watching on YouTube, we can chat in there. We have come to the YouTube, Cloudflare Developers YouTube, and we'll chat there. And I would love for you to see that. So I pasted the wrong thing. That didn't work. I pasted the repo. Let me paste that better. Let me copy and paste better. That was a mistake. So I want to show you this model here. That's available to you. So you can connect a WebSocket to it. You know, Cloudflare is excellent for WebSockets. And we're able to connect to this DeepGram model. And what we were just doing, I was waiting for Jeanette's end of term when she was done talking so that I could talk and I knew when it was the right time to talk because we want to find what that right beat is. And oh my gosh, this model is so cool at that. Playing with this model, it's like really awesome to do that. So Jeanette suggested, well, you already have the part of like, you know, this thing that we're working in here. You already have this part with this text. It's like, why don't you just send, do a transcription when that happens, and then do it into the flow that you already have working here. And so I tried that. So let's do a quick demo. I got to look at this form a little bit here. I'm going to show you the reverse of this. So like, right, I said that I like Slack and email, and it checked my Slack and email. Check this out. So there's this start speech. So let me talk about what's going to happen when I do this because I can't talk while I'm running the thing. So I'm going to click start speech. And what's going to happen is it's going to connect to Worker's AI. And Worker's AI is going to return back a WebSocket, that DeepGram flux model. And with that WebSocket, each time it's going to, I'm going to, from the client side, the client side here, we're going to send up a text to my agent, or audio to my agent. And my audio is, my agent is going to send that across that WebSocket. Okay. So we've got, we've got the browser microphone is going to send to my agents, my Cloudflare agent. And then that agent is going to connect to Worker's AI, and it's going to hold a connection available to that. And I'm just going to keep pushing audio through, and it's going to come back, back down. And I'm going to wait for the end of turn. And when end of turn happens, Genia, there's like a detection of like, is this what you mean? And I haven't tweaked this at all. This is out of the box, but we could tweak it. We'll talk, we'll talk about that a little bit after this, after, after, let's do this really quick. So I'm going to start my speech here. Okay. So I actually like video calls. I prefer regular check-ins. You know, I want to get better at technical writing. I like project planning. I don't like async docs first, actually. And now I've, I've stopped my speech, but it's gone through it, but you could see the end of turn. And remember, it's my code that's slow. It's not this. You see how the quick, the input was coming from and it, how much it nailed the right end to turn. When I was done saying what I was thinking about, it was doing it. So you could kind of imagine somebody looking at a form and speaking about this. I need to speed my side up. It was clear that deep ground was like going very fast on this. I need to clear my, my backend up, but the processing's happening. It's all happening with a workflow, right? So I queued all that stuff up. So in the case that you do have something that's running slow, you can use these workflows to go, or maybe I'm hitting some other sort of API that might tell me that I've overloaded it. I know that it eventually it will happen. That's the nice thing about this, like durable execution. So agents have a nice, they use the workflows inside now. So there's a, there's a nice workflow inside of the agents where I just say, create this workflow. It was really, really cool. It was first time I actually used that here. And I think that this style is something that I'm going to play more with now that you've introduced me this to this, this end to turn, I feel like it's a really powerful way to get what was happening. You could imagine me doing a tool call, right? Like you could imagine me saying like, ah, I'd like the lights in here to be green. And then, you know, they go green or something like that. I think that it's a, it's a new input field that, that a new style of input and the fact that I don't have to do it on the browser. So this'll work from my phone. If I was filling out my phone, I just press that and it would, you know, it's I'm using the browser, but I'm not like, no, it's not heavy computation. Cause I'm doing all that on the back. Right. So it's a really neat, really powerful. And go ahead, no, no, that was my end of turn. I end of turned and you can, you could jump in there. Is there anything you want to add? So actually I wanted to say that if you want to configure it for your use case, like as you expressed for most of the use case, the default threshold, which is 0.7 works pretty well. But if you want faster response, you can actually set few parameters, like set eager end of turn threshold. So your LLM can start processing before even the user fully finished the sentence. If your app requires that. Or if your user speaks with long pauses for their style, say you are at some care provider and there may be users who will be pausing longer in between words, sentences. You can also play with the end of turn timeout to avoid cutting turns prematurely. So that's cool. It's like, however you want it to behave. End of turn threshold controls how confident your model must be before saying that, okay, my user has completed his sentence. I can speak now. And also if you want it to adjust with your user base or your personal speaking style, you can adjust it by setting up the timeout so that it's going to be waiting longer. Yeah. And I, I mean, love that it did exactly what it needed to do out of the box. And I think that that's really neat to have those knobs to be like, I'm going to run this thing and oh, it doesn't, you know, oh, it, it, it jumped in too quick. Right? Like, cause I could ramble on as I do, as you know, Genia, I ramble on and maybe I don't want the whole thing to go. So I want to make it more sensitive. So you might, you might not want it to jump in there. Right. But, but yeah, to, to your point, it's a new tool. And like, I want people to think about that. Cause I, I hadn't really processed that before I actually touched it and played with it. Right. And I think that's true of like all the AI stuff that's out now, you actually have to like use it to see how the thing might work. And this is a great way to do that. Right. Like build a, build an app and they'd be like, huh, can I put voice in this? Cause I think that's coming. I, I feel like it's coming. I think like we had a little run, you know, we're all excited. I was just thinking, say we are actually chatting with our code agents, like by chatting, I mean texting. It can actually process pretty good text input converted to a good code. The hallucination is pretty low nowadays, at least for my experience. However, instead of chatting, I'm just giving it voice comments that how you started it. Go research how people are building personal readmes and then build me an app for that. Yeah. However, just telling it that, okay, I want this. And maybe you're doing some other productive work in some other places where you're building another cool demo where talking with one agent to build your personal readme. Yeah, absolutely. And I did do this and you're right. When I, when I was in, I was, I built this one with Codex because I want, you know, like I said, like I wanted to see how, how does, what, what will the app, what does the app look like if I don't even give it specs? Right. So I was basically like, we'll build this on Cloudflare and I want to do it with agents and I want to go and yeah, I could imagine it's like, here's the app. Oh, actually I want you to do this. But if I spoke that, I feel like it would probably be a little bit more real to how instead of me sitting there and thinking about it and it's wild that we're getting there, right? Like it's wild. I just did a, I did a thing just yesterday. I pair program building a thing and we were switching off. We were just using, there's a, there's a local, a local tool where it will do speech to text. And we were doing exactly what you were saying because we didn't want to share a keyboard. So we were just kind of leaning forward and talking and we built an entire app with just our voices. And like, it's at a time now where it's like the speech to text probably does more, right? Does more with the typos than we do. The LLMs are excellent at typos, but they're probably also great at like, if I were to ramble on about something and like, just like it'll find the right context in there, which is just, that's pretty wild. We should actually should, there might be something fun to even look at here. And here's somebody talking. I really like the idea of multiple people speaking because it's much easier than just grabbing at the keyboard from your colleague and say, okay, I have to type now rather than we're just talking in the common mic. It can pick up the sound pretty easily. I'm going to bring this, we have a, somebody, somebody responded here in chat. I'm going to, I'm going to try to show it here and I'm going to read it out loud for those people. They think it's cool. They're using a healthcare app and they worry about max connections and supported languages. Supported languages, I think is pretty awesome. And we can talk, Jean, you want to talk about that? And depending on those might be able to use Flux. So, so talk about the supported languages. What, what are the supportive languages? So Deepgram can actually support Flux currently has English model, but it is, we are going to extend it for multilingual also. But if you'd like to extend it to multilanguage right now, I would suggest Nova 3. It had different language as well as it has a code switch language. What does that mean? I'm a Bengali speaker. I can speak Bengali and English. And in my day to day, I often actually talk in Banglish if that makes sense. So can you give me an example of some Banglish? Oh yeah. I'm not feeling pretty good this morning. Which basically translate to, I'm not feeling pretty good this morning. I am having a light headache. Okay. And that happens sometimes with my partner. And sometimes it is funnily enough happens when I am in a deep brainstorming session with my colleagues. What the word I often say is, which means, okay. It's like a kind of acknowledging word in Bengali. It's like, okay, I'm just used to with that so much that I'm just, I keep saying that, but. They're like, are you sneezing? Acha? Achu? No, it's a dumb joke. Sorry. Okay. Sorry. Dumb joke. That was a dumb joke. Sorry. I apologize. But no, I can't get it out of my head. Each time I'm going to say it, I will be thinking about that. So it does do that, right? Yeah. Yes. So that's cool. That's cool. Like it does the, it does the, it does, I could be speaking English and it detects that I maybe said, acha. It's like, so it knows I said, okay. Yes. Yes. If you said, but for that, you have to set the language to multi. So if you anticipate code switching language, you should be using multi. But if you're actually anticipating that it will be just a Bengali conversation, I would use the language tag to Bengali because that's more accurate because then if you think that how you're investing your model power, it is not invested in finding, okay, what is this word? English word or Bengali word? Right. It just knows a Bengali word and it's going to transcribe. Gotcha. Gotcha. But it, but there are other, you do have other, there are other models, right? That do language. Yes. So there are, Bengali was just my language. So I keep traveling about that, but there are Arabic, there are Croatian, Dutch, French, German, different accents of English. Hindi, Italian, Japanese. There is a whole list of languages. And if someone is actually interested, let me paste the language link here. So you can just take that if your language is there. Awesome. William, there was also a question about max connections. I don't believe that that's a problem here. William, can you talk about how many connections you're thinking about there? Just in case, I don't want to say, you never want to say never. I don't know what workload you're working with there, but if there's a ton, there might be a better way to think about it. But as far as I'm aware, I have not heard about the limitations of how many active connections you can have. Now, I don't know if you want to have it all inside the same agent, right? So in my case, each one of these personal read -me's, right? And in fact, this is kind of interesting. I'll show this a little bit here, kind of a little, the power of the agents framework here is the agents automatically sync. So if I do this left half and I pull this over here and I do, yeah, there we go. Do this on the right half. Ooh, do this on the right half. This is kind of like me looking at the edited version of it, but the way that the agents work is they have state and this is connected to my agent. So if I were to ever update this, this would just kind of update automatically. Let's see if I can get it to go. Oh, let's do the, let's do the time zone. So it says time zone not set over here. And this is really neat, right? The time zone options here are kind of bonkers, right? So like this is where, this is where I'm showing off the conceptually why English might be better than forms as I can be like, I can do this. I'll do the start speech. I'm going to kick it off again and say, I'm in the Pacific standard time zone actually. So grabbed it, it queued it. It should automatically, we should see this time zone here change and we should see it change here automatically, right? So that the agents automatically are updating when you need something like that. You need something that's real time like that. And of course, again, the code's not on the back. We saw the transcriptions already there. So like if I were to update directly from that, but you see it, it automatically updated both of these because the agent itself has WebSockets, right? So, so, so I think what's powerful is you'll hit a maximum 100 speakers. Of course, not in the same room. Yeah, of course. Yeah, yeah. So, so totally fine. Hundreds of great, a great, you're fine there. I was making sure we're not talking about millions because if so, maybe we need to figure out like a team to get you connected there. And then, you know, Cloudflare also has a great web RTC. So if you're thinking about doing chat channels and things like that, and we are very much able to plug things together. You should be seeing more and more of that coming from us. I don't want to talk too much about roadmap here, but you should be able to speak to these agents, right? Automatically and, and use models like, like these powerful DeepGram models, right? You should be able to choose your model and you should be able to speak to it. And then one of the things that we talked about, what we've been talking about here, what we're doing here is speech to text, speech to text, but we could go the other way, right? Right, right, Gina, there's like a, there's the whole other world that we talked about a little bit here. And you can imagine that these agents here are coming, things are coming in and then they're coming out as well. So, so, so they're talking back. One of the things that Gina suggested was that it's like, I've updated your form. Like each time I went, like to say it or some sort of like notification of that. I could say, let's think about this the other way, right? So here's the, I'm a, here's the profile here. What if I came in here and I don't want to read this. I don't want to know what this is. I just want to click a button that says, just summarize it. Should I send him an email? And you could say, you should not send an email or he actually likes emails. Yes, he does like emails, right? But if I uncheck emails now, I'll uncheck that here and I'll, I need to do a save on it. It'll automatically, you know, will happen. So if I was editing by my voice, now we'll not send an email. So again, what I was talking about was like, I think the future is gonna, we're gonna want to know what this is at all times. Like, because it's not just people talking to each other. I love this for people. And I've seen this work really well on teams about this readme, which is where my brain went for it. But I think that also, again, agents are going to talk to us. And I would imagine one of these checkboxes is like, I would prefer to talk to your agent, right? We're getting to that point where it's like, I would like, let's just communicate over however we, I don't know. I don't know what that world is, but it is coming. And Gina, you're in this world like way more than I am. Do you see that? Yeah, obviously. So what like a real next step for me is that, okay, my app listens to me, it understands me. Now I want it to response back to me. So instead of going back to my terminal and check that if the output is ready, what if when the agent is done generating the output in this case, like saving the profile, it will be just giving me a voice answer. Your profile is saved. And then when I'm sending an email to Craig, it is responding. Craig has not said email has his preferred communication method. Are you sure you want to send it? It's like, I will prefer that. So instead of me going through the whole thing, say I'm just typing the email instead of sending the email, I can copy paste it to Slack as he likes Slacks. That's awesome. And I think maybe even I want to send this message to you're speaking to something. I don't know what it is. You're talking to an agent that you've built that has access to your Slack and access to your email. And you say, I want to tell Craig that I'm going to be five minutes late. And it goes, okay. And it looks and sees that. Oh, Craig likes email. He doesn't like Slack. He's never on Slack. And it chooses to write an email that says, Hey, Craig, this is Jinya. I'm going to be five minutes late. Just from you being able to communicate and then the agent having access to these tools that we're talking about. And this is. If you allow access, that's totally doable. That's one of the caveat that you need to allow it the access. But if you allow it the access, that's doable. Yeah. And that's a situation where you might want your end of turn, right? If I'm talking to my agent, you might want that to last a little bit longer because you might want to say, tell Craig, I'm going to be five minutes late to the lunch tomorrow. But it's already gone and kicked off and tried to go and start. But like that tuning of that is something that like will be powerful and something that you can control where I feel like I don't, I know that when we talk about voice, we talk about voice agents. We talk about my, I like to tease my Alexa who's gotten a lot better, but she, I couldn't interrupt her. She would misunderstand me and start get going before things would happen. But I think that like. Cloud Jin is not there. Yeah, exactly. Exactly. And I think that now those experiences that we had, everybody and builders out there who are watching this, we felt what that is. And now you have the ability to tune that, right? You have the ability to create the experience that you wish that you could have done here in the web, right? Like it's, this is like through a webpage of we're able to go and we're doing a bunch of stuff on the backend as well. Right? So like whatever sort of connectivity and things like that, but it's a new input channel. And output channel. And I think that that's really powerful to get, to wrap your brain around. This code is available. And I wanted to like, just be like really clear about this, that there is a, there is a repo that's available. And I started this from code generation, right? Agent code generation, which I know not everybody has leaned into the codes really good. I'm very happy with how the code works. It's a little bit structured differently than I, than I would normally do it, but it's still good. I went and I tried to make sure that everything was looking good. And in fact, I'm going to click into this code here so that I can show you. And then I told it that I wanted to use the Cloudflare agents SDK, and I used the skill and then I told it, I wanted to draw what is happening here. So let's, let's take a look really quick at this architecture. If I can figure out how to do this. So it made, it generated this, this mermaid diagram for my readme, because this is something that you can do, right? So in readmes now, and I'll, I'll share this here so that it's not so big on the screen. Let me put this here so we can kind of talk through what's happening. So I have, we have React running on the browser, right? And Jinya helped me with, with figuring out exactly how we use this like media recorder. So there's a media recorder API that we wanted to do that we get, and we get chunks and I send those chunks, chunks of, of data into my personal readme, which is it's a, it's that durable object. So each user has a durable object, right? And so if you haven't seen our durable objects before, you can kind of think of them as an instance, like an, an instance of an object that lives and it lives close to wherever it was created. And it has states. So it has a little database inside of it. It also, and each one of them have that and they live forever. So it sounds a little too good to be true because sometimes this is exactly what you want for an agent. And that's why the agents framework wraps around this and add some additional things to it. So one of the things that is really nice about this is it will, it's serverless, right? So we don't want you to pay for stuff that you're, when you're not paying for it, right? We only want it, want you to pay for when it's actually doing work. And so one of the things that happens typically with web sockets, awesome technology, but you need to have it connected to a server at all times. So one of the nice things about durable objects is we have this hibernation API. So the durable object will fall asleep, but the connection will appear to stay alive to your client, right? So, and then your client will, will say, Hey, I'm sending this thing and I'm ready for it to kick off. But I still have a connection to my agent. Automatically. And I don't, the code for that is not, it's not, uh, I'm basically saying use this agent that has this name. And it gives me that. Yeah, exactly. So Lillian said that it's exactly what I most use in my app. So, so durable objects are great when you see, see DO there. Yeah. AR, I always lean on them after I started using the, these agents, I haven't stopped using it. And people are like, well, is that really an agent? Like, yeah, it's going to be an agent. We're, we're in a world where like, I think everything that we build is going to be an agent. And so this, this personal readme agent has all the state of what that form says. Right. And I, I can query that in different ways. I can query that with voice. I can, any sort of applications, you think of it as like, it's an object that says, this is how Craig likes to behave. This is how Craig likes to interact and what he's working on and things like that. A durable object sends to a worker's AI web socket, right? So that's, that's the deep gram model that's here. So it's, it's sending a deep gram flux. It sends those audio chunks so that the chunks come from the browser into the agent and the agent sends to this web socket, right? So I can connect to that web socket. It is so fast. This is so awesome. The, the, the deep gram flux running here and it's closed. Right. So it's, it's going to move really, really fast. And when it detects an end, it sends back a stream of data and it has in there the end of turn transcript. And when that comes back, that's when I say, okay, put this in the queue because now I need to go use Zod and to Zod's a, a schema here. I actually should be in here. I'm surprised it's actually not in here. And that's how I'm doing the patch. Right. So I'm saying like, here's the current schema filled out this way in the form. And I want you to make changes to that form. And as you saw, maybe I'm not doing that the best, maybe the, maybe that's, that's slow. This is what's happening. And I worked on, I worked on this with the coding agent, so it looks right, but I think maybe there's some, some things in there that I need to work through. And then I kick off one of these workflows, right? So we kick off the text update workflow with the text that's from the end of turn. And, and then it does these, these callbacks. And then there's this, it, we update from that. We use the workflow to update when the thing has happened, because that could take time. And again, what I was trying to show here is that you can kick off a whole bunch of things, right? With the, with the workflow and the, and the agent itself has the workflows built into it now. So it's called an agent workflow. It, it kicks off and it comes back and I can update the profile as the, as the text comes through. So it's a really nice fan out method, right? I can throw a whole bunch of things in there and it can all run in parallel. I know we were talking earlier about how many connections we can have. We can run a whole bunch of things in parallel and it will come back to that one. So it's like come a fan out and come back. Right. So that really powerful pattern that we're just starting to play with here. Cause that's pretty, that's pretty new into, into the agents framework. And yeah. Great. Sorry to interrupt, but one thing here, if you want the interim results that is also available. So what I mean by interim results is say, I love building, I'm thinking with Python, not with CPP. Now there was a pause and you are getting the interim results. And if you want to save those interim results or start working with them, you can do it right away. So if someone doesn't want to wait for a end of time and do some process, start some processing on the backend for some reason, like live transcription and stuff like that, that is doable too. Nice. Right, right, right, right. So, so those messages that are coming through, I'm specifically looking for end of turn and then I'm doing some work, but there are other messages that are coming across that web socket and it's like word. And then like how confident it is that it is that word. Yes. And then I actually, one of the things that I think we actually had something in the chat talking about was that there's like medical things. And that's probably something that's pretty dangerous, but you can control a little bit with that, right? Yeah, that's actually a great thing. So if it is a medical thing, that's a domain specific thing. And that means there will be some key terms or keywords and you can specify those keywords. So you are not going to mess up the important words for your domain. Nice. That's this key term here? Yes. This is this key term field? Okay. And you can pass multiple key terms. Now say I am a CS person and I can talk about different CS fields, but I have no idea what the medical terminology stands for and I can mispronounce them or I can write the spelling incorrectly. You don't want your bot to do that. To avoid that, you can use key terms. And the fun thing with it is that you don't have to append the full dictionary to this list of medical keywords. That can be a lot. But you can actually append only the necessary keywords for this specific conversation. Say we are now discussing... That's clever. Like contextually. Exactly. Contextually. So from the dictionary, say this patient is specifically discussing on the last prescribed antibiotic and say that was amoxicillin. So you are going to just add amoxicillin to make sure it is spelled out correctly. It's kind of recognized correctly. Can you spell amoxicillin? I'm just kidding. I'm kidding. Why? Why you have to do that? No, I was just saying that because of course, I don't even know if I can pronounce it correctly. But if I was pronouncing it, you know, I don't know, amoxicillin, because I'm reading my thing. I don't know how to say amoxicillin. Amoxicillin, I do know how to say that. But if let's say that I was reading my pills, I'm like, I need a refill of blah, blah, blah. I don't know what that is. This is going to help you detect because you know your prescription. Is that right? Yes. That's so cool. Because it's personalized to me. So again, in the agent space, this is a nice tool for, I know that Craig is on these medications. Here are the key terms he might say in this conversation. Let's preload the key terms with what's happening. That's awesome. That's really neat. So then it feels you don't get the, I'm sorry, I don't understand what you're saying, you know, not to go back on our old voice agents, but like, it's a different world. It's a different world. And it's a different... It will not keep saying that. I don't know what you were saying. We're going to get past that. We are going to get past that. That's awesome. I love it. Do you know how big that field can be? I think that's what my question would be if I were in the audience right now. So it's like, how many key terms can you pass? I think that that is a hundred key terms per request. Okay. Okay. That makes sense. That makes sense. And again, this is... I set this when I kick off the stream, right? So I could actually, to that point, you know what, you're making me think like, you know what? I think one of the powerful things about agents is they get better over time. So you could kind of figure out like if your person is interrupting you or it's talking too quickly, it's like, I know now that this person, this specific person takes longer to talk so I could tweak these with AI. I could like find the best speed for this person over time as it's going and we get less and less interrupts or angry responses, right? We could do all sorts of stuff in the back end, tracking what we want to do for these end of turns, right? I think that's cool. I hadn't even thought about that. I'm sorry to interrupt, but I just found that we can actually support up to 500 tokens in the key terms. So not 100, it's up to 500. 100, awesome. And you were not going to wait for my end of term. You're like, I am so tired. There's no end of turn coming in the future. He is such a rambler. No, actually, that's the thing. I'm a human, so you are not angry with me, I hope. I'm not angry. I am not. I am not angry. I could never be. You helped show me all this cool stuff. I can't wait to build like what's next, right? I think I do want to work with more DeepGram models where I'm like doing text to speech. Let's make text to speech next so that this one is talking back to us that I have saved to your profile and we are deliberately trying to send you an email where you don't like it and see how they're actually interrupting us. Yeah, yeah, yeah. Let's wire up an agent that gets better at knowing when the end of turn stuff happens. I really like that. I think that that's a really cool, fun idea that we can do there. We can also experiment. Like some fun experiment I like is to balance between the latency and the conversational safety. So by conversational safety, I mean after an awkward pause, like say we are in a room full of friends, but if there is awkward pause, someone start talking. But how much long pause is awkward pause? Right, right. So yeah, it's something like that, that, okay, this is just having a latency, lower latency. So we are going to start talking or is this just audience is a slow speaker and we should respect them. So play with those two values and see what is the most suited, well suited one for the use case. Like say I'm now kind of talking about the real life use cases. I just try to listen what the actual audio is, play with it and say that what is like parameter tuning, what's the best set of parameter where the margin is not like awkward interruption. Right, right. Do you think this is the future? Is voice, like are we going to get rid of, are we getting rid of the, are we getting rid of the front end altogether? I don't believe that. No, okay. I don't think it will be getting rid of everything, but I think voice will enhance it. Voice will enhance how we are interacting with it because seeing is still a very important part of how we are perceiving the world. So I think front end will be still there, but there will be say a way to communicate with that front end. So the front end is not just abstract or just a device sitting rather than the front end is changing in front of your eyes that that will be more user-friendly or more exciting at least for me. I think that there's like, there's this bit that I want, that I've always wanted, you know, I've been making websites for a long time and like you show somebody something and they're like, can we make the font a little bigger? I want them to say, can we make the font a little bigger? And it gets a little bigger. Like I want that. I would, that is what people want. I feel like we're in that space where it's like, can we change that text to say this? And they just say it specifically. And it's in context. We have the key terms, we know when to stop. And then it just goes in updates. I feel like that's a cool app. I would love to, it's like mixing of the coding agents, but not, not to us developers, but to more like you're, you're allowing more people to play in this new space that we're in. Like making it more accessible based on the needs of the user. Yeah. Are you seeing people do like long form content with this? Like, like, could I write a, could I just narrate a blog? Yeah, you can. Yeah. That you definitely can like keep. So instead of blog, how about thinking of personal diary? Like, okay. Too late at the end of the day, too lazy to write it, but you are just taking the voice memo and that voice memo is safe. So it's not lost under thousands of files in your phone rather than just safe. And now you have, if I say 10 years down the road, you want to just see, okay, what I was doing, what I was feeling is there. That's awesome. And yeah, they could also, you could use that. I mean, I was leaning to AI. You could always use that to train some stuff as well. Like make, yeah. Depends on how personal you get with it. If you're willing to share that information with your, with your agent. Right. Yeah. And yeah, it depends on how you want it. I really like that part of using agents or AI in my life. Like how much I want to share how I want it. Do I want it just to transcribe or do I want it to generate a summary of my last month? Like say I'm just transcribing it as my work that I completed this, this, then that. Then after the week, instead of me looking back the whole thing and writing the weekly summary, I asked my agent summarize this week and I can put a note there for myself for reviewing for my teammates and for the leadership that, okay, this is here. So yeah, I've been simpler. I've been driving around and using some voice stuff like in my car, right. Like safely driving around, but like also like to that, to that memo point of like, Hey, help me remember to do this thing or like, Hey, let's research this. What, what does end of turn mean? I want to like, and write basically getting like an interactive podcast with a, if your LLM is good enough and, and the, it, the, it's getting better. It's getting better as it goes. I often write code in the car with my voice. And I think that this, this, like, we're getting to that, to like what you were saying earlier, where you're like, I want to build this thing. And if you talk to it, I want to build this thing. It works really well. And it can ask questions and the questions are all punctuated, right? The speech to text feels so rich these days. I think that we have a lot of that future coming. I really think that this is such a cool input output that will enable like accessibility. Text to speech is also getting pretty good right now. You can also play with how fast or slow you want it to respond, like kind of, or what accent accent was there already, but you can also play with the speed that I want a fast response or no, when you are speaking with me, speak slowly so that you can control that. Yes, you can control that. That's neat. So that's like a, that's like a new style, right? Two of like a, it's like a style, like almost like CSS, but like, this is how I want my agent, my, my response. I like that analogy. Yeah. CSS for agents. That's cool. Speaker agents. Yeah. Yeah. That's really neat. Let's build that. Let's work with that a little bit. Yeah. Let's see. We could, you just keep talking to it and tell you tune it, you tune it with your voice to it. You have it speak something back to you and you keep on talking back to it. That'd be kind of cool. But my voice, you mean that voice cloning? Do you do that too? No, not yet. Oh, okay. Okay. But what, I mean, I'm not sure maybe if the client wants a lot, we might do it down the road, but not yet the voice cloning, but you can pick what you want. What does that mean? You want a calmer accent? You want an emphasis person? Or you want like pretty excited one? You want to respond slowly? A sports commentary versus a healthcare representative. They should speak different. Absolutely they should. Or that's weird. That's a weird trip to the doctor if they're doing sports commentary on your... Yeah. I will be. And he's coming in for the, he's got a broken leg. Here it comes. Yeah. That would be like, I don't know. That's wrong. Fun for a day, but every day, maybe not. Probably not. Probably not. And for your personal read me or not really read me like personal assistant, which is talking to you on your car trip. If it is a busy one, busy road and jam-packed traffic, you can just speak slowly because you have to navigate through too many. Yeah. Agitated drivers. Let's just say that. But when you are driving long drives and you want to learn something, you can say, okay, you don't have to worry about others in the road. It's a free road. Just you can say that, okay, now I'm less occupied in my driving. So you can speak more faster and it can adjust like that. It's so cool. It's such a neat time. And I think that we're just starting to scratch the surface of what these tools can do. I don't know if anybody's thinking in the audience about what you could build with it, but I think, and I hope that you saw from this and these talks that you're feeling that you can literally build anything. You can control both sides of what's coming in and out, right? You're controlling both sides of the style. You're controlling how long this, when you need to know the information from where they're at. Like if you need it immediately, they don't even need to finish their sentence. You can get each one of those words as it goes. If you're doing that, building something with that, which when you started saying that, I was like, Oh, that's kind of fun. Cause you could do, um, you've seen that game. I've got a 14 year old. So like there's a game where they're like, they put a bunch of emojis up and you have to say the emoji as it happens. Have you seen that? Yes. Yeah, yeah, yeah. You could make that game. Right. Have you made that game? You play that game. You knew what I was talking about. What are you talking about? But I'm old, so I haven't played with that. You can't move that fast. No, they're like, you know, I hear from the room, they're like apple, banana, uh, cat, dog. And it's like, just, you have to try to hit the thing at the right time. So you can imagine that's an app that you could build on this with the stuff. I don't, I don't know if that's exactly what you want to do, but like. That's a thing because you get the web. So it's like image recognition along with, so once the image is recognized, it's just saying the name. That's it. Yeah. Yeah. Oh, you could probably cheat. So image recognition followed by. You're suggesting cheating the game by doing the other side, doing text to speech. So it does image recognition in it. It can always win. You can impress everybody with that side of it. But, but I feel like there's, there's these, there's these tools that are now available that like. I told you I'm old. I don't play games. If I have to wait, I will just find something. You're going to reach for an API. I see. I see how you work, Gina. I get it. I get it. That's super fun. Uh, everybody, we're coming up to the end of, of this, this first worker shop. I hope that you had a good time here. We do have access to that. I'll, I'll just share it. If you want to, if you want to give it a spin, you could come here to this and we'll, we'll, we'll send out an email with a link to this as well. With all sorts of information, Gina, thank you so much for a first helping me build this app, helping me make my dream come true. Um, and for coming on and for sharing your knowledge, doing your knowledge drop. It was really fun. Thank you so much. Great for inviting me. And thank you everybody for hanging out. And if you're watching this in the future, make sure you click into those links and come play along. There's a, the footer of the app has a links to deep Graham and links to the workers, AI models, and also a links to the code. Again, that was generated and you should be doing that. I want everybody to know that like, I, not only did I say I want to use the deep Graham models, I want to use agents and it went and it built the things that I wanted to do. And I talked about what my concerns were and it helped me write those workflows. It was really fascinating, uh, watching that whole thing happen. So I would love to see more of this. Did we, we saw there, there was a comment here, building a design agency with a colleague. Let's, let's look at this really quick. We're just about ready to close last last words here. Building a design agency with a colleague of mine, where part of the process involves editing AI generated images. Oh, that's cool. And use, use it for a TTS. Yeah, exactly. I make that a different dog. If you're like, you know, like, I don't know what you're generating. I don't know why I said dog. Like there's a, a picture of a dog and you want to make, make the dog longer. Make the, make the dogs longer. Change the maltese to a husky. Yeah, there you go. And like, and it switches, it switches the dog out just by talking to it with it on the screen. I think that that's super powerful and that's a great use of the end to turn. Same thing, right? So like, like you, you say what it is that you're trying to go and make the thing go and change as it happens. So awesome, awesome idea. And I think there's so many of those. There's so many of those, like, first of all, the fact that we could generate an image is relatively new and being able to generate an image with your voice is also pretty great. And I don't know, you can also generate it and then put it in the back. Audrey, thank you for showing up. We have somebody who just showed up two minutes before this is over. Thank you, Henry, for being here. Appreciate you always showing up, even if it's at the last minute. This will be recorded. I'm glad that we have the opportunity to say that this will be recorded or this has been recorded and we will be sharing this out with links to the readme and more information about this. Yeah, exactly. Yeah, and to the point, William, just one more thing. We were talking about a dachshund. He said, make it not a make the dog hot and it makes a hot dog, right? Like, but make the dachshund. But we could put a key term in that it was a dachshund that we were talking about, and it probably wouldn't accidentally draw a hot dog. It would probably know about the dachshund. I'm hoping. I'm hoping that's what happens. That's, yeah, that's the hope. I'm not sure what the image generator will do, but I hope it will get from the context. All right. So we will be back because Gina and I are going to build lots of stuff. Voices is only only picking up speed. And in my brain, I love to build new stuff. And I'm going to reach for voice all the time. Thank you, Gina, for introducing me to that. Awesome. Thanks, everybody. We'll see you on the next workshop. Bye. All right. Let's just hang out here.