Introducing Agent Lee - a new interface to the Cloudflare stack

hello there everybody i hope you are enjoying your agents week i know that i am i am here with two of the blog authors of one of my favorite releases uh this this agents week and i am so excited to dive a little bit deeper into we're going to talk about gently and if you haven't read that yet it's time to read that blog post but maybe first watch this because we're gonna we're gonna talk about a bit but first i have with me i have a parna and kylie and i would like for both of you to introduce yourselves and tell us what you do here at cloudflare awesome uh well once again i'm a parna i am the pm for agently and for customer support at cloudflare been here four months now and really excited to be talking to you about what kylie and i have built with agently yeah so your first agents week yes your first your first innovation week yes pretty intense right yeah it's been quite a lot of fun and to see how things are done here. It's been quite an exciting start for me. Awesome. And Kylie? I am Kylie. I'm the engineering manager of the Agent Experience team, which has been working closely here with Aparna on the Agently experience. And I've been at Cloudflare for just a little bit over a year. Awesome. Awesome. Great. Let's talk a little bit first about Agently. If I'm not mistaken, Agently is named after one of the original co-founders of Cloudflare. And why do we think, why is that a good name for what this product represents? Before we even get started, we haven't even told them what the product is yet. We, as one of the three co -founders here at Cloudflare, had a really strong belief in making sure that we could not have to do repeatable work, really outsourcing these things, making sure that systems could scale very well. And that is one of the reasons that this agent here that we've introduced to the dashboard meant to kind of offload some of that. context you need to have awareness of in order to navigate the dashboard I'm just making it easier to really see what the Cloudflare platform has to offer this name is in honor to him as the founder of the company here that is awesome that is so cool let's talk a little bit about that actually so uh the the main use case here of Agently is uh the way that I think it the way that I would I would imagine this is I've got some problems oh my gosh what do I do here's this dashboard help is that is that what am i right is that the vibe well it's one of the use cases uh craig i think think about this as um with agently you can think so things that you want to find out about and you need to think through you can if you want to act you can so that means configuring things you can and then if you want to create as well you can because we have this very cool generative ui component to it as well so there's a lot of things that you can do troubleshooting is one deploying is one just searching is another one so yeah lots of things that you can do that's awesome and i know i know that we help people a lot when there are emergencies which is why i leaned there first i i end up meeting in my line of work i end up meeting a lot of people with like you saved me thank you so much you know because i went through some sort of flow but i know that seeing seeing the the demos of this and in this blog post we got some examples of you know it's like it's the dashboard is is there's a lot of things that you could do on cloudflare and navigating it can be hard and if you are in the middle of an emergency this is like such a such a life -saving thing there are all kinds of different users of the cloudflare platform as well and that's something else that agently is seeking to kind of just make simpler for all of these different types of users you don't have to be that incredibly technical developer that has wired up the configuration from the start you could be somebody that's coming in just to debug, maybe a point of access for an already deployed website. Agently is hoping to help meet you in the middle there. You don't have to know exactly which level of setting you need to get to. Agently can help you navigate and make that change yourself. That's so cool. What do you think the most common use case of this is? What are people going to use this for? I guess it's been out for a little bit now. What have people been using this for? I can speak a little bit to just what we've seen in the data so far. obviously of the internal test case, which is a lot of maybe greenfield development creation for the first time. But in practice, I think we've seen a lot of asking questions about something that's already been seen in the dashboard. So maybe trying to understand an error or something that has gone wrong in your deployment, going to Lee to ask, you know, why is this website that I thought I had deployed not actually resolving as expected? Or did I do something wrong in the configuration settings here? And Agent Lee is able to help not just answer that question for you, point you to the document. but maybe make the change on your behalf if that's something that you want to allow it to do well that's super cool what what is what what's an example of that like give me give me an example of like an end-to-end of it actually taking action i don't know if you want to highlight a specific example from our data there's been there was a really interesting one recently where someone wanted to have their cname record updated and just reading through that end -to -end conversation lee was able agently was able to effectively handle it end-to-end and the person is just profusely thanking the agent like it's an absolute delight to see conversations like this right so this was several like steps literally hand holding the person through it and uh at the end it was thank you there was a little thank you so much at the end of the uh at the arms of which as a pm is great to see right i love that that's a that should be like a a kpi right like how many how many times has agent lee been thanked which which sounds like a lot right so so are there are there unexpected things it's being asked to do have you have you had like whoa i didn't think that would be asked i mean the cname record is kind of a big deal that's definitely a big one i think it's always really interesting to see the uh you know massive production use cases where somebody might be just trying to debug that one little instance the biggest surprise at least for me so far has been the ways people are trying to maybe learn a little bit more about like the cloud floor platform on the whole or other websites asking through lee uh what are the settings that somebody has used to wire this up and lee has actually done a pretty good job of making sure that we keep information scoped to the user independently so changes that you can make are the changes that you're going to be able to make and then learning about like publicly available uh websites and things that you can use the cloudflare platform for it's still isolated out to what is publicly available so it's nice to know that it's still well protected at least for the use cases that users are trying oh that's interesting so they're trying to get like Wow, that's really fast. How did they do that? Like that, that sort of thing is what you're talking about. And, and, and Lee is able to take care of that in a tasteful, they're probably like that sort of thing. Is that the, yeah. Without, you know, unpacking a bunch of source code or, you know, trying to uncompile JavaScript code on the other end. It's still helpful for, I'm trying to think of use cases that I'm not trying to think of, but, uh, users that might go to other AI assistants versus what agent Lee is able to offer with that dashboard context. It's nice when agent Lee is able. to make a recommendation for maybe a specific service that the user hasn't enabled quite yet when they're trying to figure out how to speed up a website or make sure that it's properly authenticated potentially. Oh, that's huge too, because there are so many products and the amount of times that I've been like, oh my gosh, I didn't know that existed. That's so cool that I could turn that on. That's a button to turn on? I'm like, yeah, I love that we have something there I've been using Coding Agents a lot and being able to use this at this, at the. dashboard level it's huge i i can't i cannot wait to to push my when i push my next project into production and like just kind of ask it some questions about what am i missing it what happened is that that is that ambiguous enough of like hey i'm about ready to push this thing what what am i missing does that work that's absolutely aligned with the vision yeah this is definitely on the cards for us um particularly to see you know what changes are going to happen right now you're not able to when you're not able to see the diffs but there are several cases is like on security side you want to see you know what's going to propagate so definitely on the roadmap for us so cool so cool so one of the things that i know happens unfortunately with some of these agents that are trying to take care of it is like stuff gets out of date and it might give the wrong answer and i know that we are constantly like we are that dashboard looks looks better every single day i know that we're constantly adding tools and things like that like this agents week there's a whole bunch of stuff that came out how does it how is it ready is it going to give old information this is something that we really want to make sure in developing that ai assistant for the dashboard that has a little bit more contextual knowledge as you're navigating cloudflare it is way more aware of these specific nuances to the platform so while you might go to chat gpt and find you know something that is out of date or a document that's been uploaded in 2019 2024 it has a lot of hits because it was a really popular release but maybe that api is no longer relevant agent.ly has the context to make sure that we're always referencing the latest uh Flare API and looking at the most up-to-date reference. There's also checks in place to make sure we're not navigating to outdated dashboard links. It has all of the most up -to -date context on not just the CloudFlare features and functionality, but where you might go to make those changes. And that's something that we're working closely with a lot of other teams internally to make sure it's always up-to-date and that we're quality testing these things. As Agents Week, for example, all of these new products want to make sure that Agently has all of the context needed to execute. on those areas right and if you're thinking about something that's absolutely real -time Craig we had a scenario recently where a customer was using it to figure out why there was something wrong in that environment and agently responded with as an active incident right now going on in this so it's something that's real -time on our status page is now also being displayed as part of the conversation and agently and this was a complete surprise for us that's so cool and that's totally helpful right that's that's totally helpful of like uh what did i break oh oh no it's something something's happening oh that's so cool oh that's so rad that i i love that that's that's really neat let's let's talk a little bit about um let's get a little nerdy let's get a little nerdy let's get a little nerdy let's talk about how this thing's built what's going on what's going on under the under the covers here we've been working really closely with the agents team internally you may have heard of the agents stk so we're using the agents stk to create these individual instances of the agent for each conversation that you're having as a user that allows us to really isolate the context that might be relevant to the conversation and make sure that the agent is a little bit more performant in the way that it's thinking through the request here something else that has been a huge unlock for us is code mode so it's something that has been really interesting i think as we navigate the way agents and llms in general think through problems there's obviously a language that's going to be a little bit more familiar to the way models maybe approach thinking. And something that we've seen in real time, I think this has developed over the past six, eight months, as with anything AI, it's all very, very new. But LLMs have a better time. They're a little bit more effective in thinking through in code, in TypeScript. They've seen a lot more real world examples of actual code execution, not necessarily calling a tool in association with context. So code mode is something that has also been a big unlock for and making sure AI is able to effectively call the rock. tool as it thinks through in code in typescript in real time uh it's able to reference a lot a broader knowledge base in making the right choices and actually executing accordingly so that's a it's a big one for us the the dog food or sip champagne or i don't know if you came up with a new one during this too but like just just i love cloudflare building on cloudflare so cool what a great example of of uh of those tools and yeah the code mode unlock is is really really wild and then i'm assuming also there's the there's the new there's a new CLI as well that that I'm sure I mean I'm gonna talk to the thing about it what can I do with the CLI let's talk about so the security model a little bit right I'd imagine that I could I could do some nasty stuff if I if I if I told the agent to do it what's going on there what's what's happening that's probably the scariest thing that you open up to AI is what can it do can it delete everything on behalf of a you I know Aparna we've been thinking a lot about like different use cases for big teams maybe where somebody we want to make sure that we're mirroring the correct entitlements for that user so the intern to the project maybe isn't able to delete every single worker associated with your production instance specifically within the way that the agent is able to make choices we have a permission model that does as mentioned mirror the entitlements given to an account but you're also going to explicitly any kind of destructive action is explicitly confirmed by the user. So we make sure that agent.ly isn't just free-for-all removing things. You are going to explicitly, as a user, approve or deny that request anytime the agent thinks that that is what you're requesting. Aparna, I don't know if you want to expand on some other, I guess, pieces that we've had to think through with the permissions model. Absolutely. I think more so than the permissions model, it's also great when you asked about security and people trying to do things here. A very common pattern that you see is people trying to jailbreak any new AI assistant or AI agent that comes out. But one of the most important decisions that we took when designing this was every session has to be an isolated session. It spins up and it spins up an isolated sandbox. So we see many, many jailbreak attempts every day and we don't have to worry about it because it's just it's not going to break the system. a one individual session tied to a user right so these were some of the the decisions that we had to take in terms of making sure making sure that this is a secure product that we're building the isolation is also important for a nice mapping for that permissions model that we were already mentioning um but being able to you don't get to persist the state from a previous sandbox for example i think making sure that these conversations don't leak into each other is another important consideration in the security model i'm assuming that the sandboxes that you were just talking about there are cloudflare sandboxes yes okay awesome what other things have we dog fooded some people don't like that term dog fooded or sip your champagne i don't think we found the right thing but cloudflare on cloudflare what what else are we what else what other tools do we use what are primitives we mentioned the agents stk um which is an important partnership for us i think another big one is the workers ai partnership that we have been developing we're using workers ai to power Lee and to make choices between the models that we're actually executing on, the ways that we measure performance of the model and being able to dog food kind of these more recent inferences that have been released by workers AI. I think it's a really important one for our team. Are we running, is it Kimmy that's there? Is it running on Kimmy? As of today, yeah. I love that. Yeah, exactly right, because they're always going to get better. The models are always going to get better, but I'm really loving how fast Kimmy is moving and the reasoning and yeah, Awesome. So I'm sure we learned a ton there too, as well. One of the things I like about that Cloudflare on Cloudflare is we use it, we're like, oh, we wish it did this, and then it does that, and then everybody wins. So what does good look like? How are we measuring success here? All right. So we're looking at a number of different things. First, we're looking at user interaction with it, how often people are using it, how many people are coming back. We've got a baseline that we want to maintain, and given that this is a brand new product. Right now, our focus is on adoption, but as this moves down, of course, we're going to look at improving things down the funnel. We're looking at performance metrics, and also something that we've got running is evals. So we're scoring all of these responses across multiple parameters. Accuracy, correctness, resolution, hallucinations, which is a big part of this. So we've got baselines that we are constantly monitoring. And since you asked about... the model that this is running on, we also keep those in lockstep every time we try out a new model to see has anything regressed there. Right now, since all of this has been instrumented pretty recently, this is now going to evolve into things like, you know, number of sessions where it's actually effectively helped a user complete an action. It's also going to turn into a feedback loop to product teams within Cloudflare as well. You know, if something's hallucinated, is it problem upstream product is it so it's going to it's going to turn into like a full feedback loop we're really trying to balance the uh the quality and actual you know latency performance against overall satisfaction as well so as long as users are actually uh getting their problems resolved and we mentioned that thumbs up metric it it really is a kpi we are trying to evaluate the satisfaction as well as like our own perceived performance so two different uh angles of approaching there but both really important that's awesome i i i saw a little thing in the blog post about being proactive about thinking about like what does it feel like to be proactive with an agent like this talk talk me through that a little bit there is a very we mentioned the agentic harness and being able to interact with all these different cloudflare products having that knowledge of what a user might reach for and then also this you know association to the the cloudflare products i think there's there's a strong vision for not just making recommendations that a user might be completely unaware of, as you mentioned, there's all kinds of releases where a user might not even realize that's something the Cloudflare platform can offer. But there's also even proactive consideration of like problems that a user might hit. Aparna, I know you represent customer support and customer success here pretty strongly. There's a lot of different use cases we're evaluating for extending Lee in the future if you want to cover a couple of those. Yeah, well, I don't want to go on to details very specifically about what those are, but definitely top of mind is in terms of if we see unexpected changes for example like this is one scenario we see unexpected changes in a customer environment we'd like to proactively inform or alert the customer that there's something going on in there and of course because one of our principles over the here is to always make sure the customer approves any change so if say there's a spike unexpected spike in traffic and it's because because of some whatever it is right but if there's something we can inform the customer about we'd like to do it and then let the customer take a call in terms of what needs to get done. So that way there's a reduction in terms of the tickets that come to support as well for things that should not be support tickets. Yeah, I love that. And that really lines up with like the vision that you were talking about from the start of this, right? Like the context is there and every customer should be able to achieve what they're trying to achieve. So thank you so much for building this tool and for moving it. forward. And for, again, dogfitting, everybody's benefiting from all of this and everybody's learning more about cloud. It's what, what a wonderful tool, what a wonderful tool y'all have built. And would you like to give any shout outs here? I mean, I always, I always like to let people do that. I know there's one more person on the blog post. I definitely want to shout out the entire agent experience team, uh, for working really hard on this one, especially, you know, down to the wire on release, getting all of that real-time feedback and making sure that we're able to keep Lee responsive and performant, um, agent experience, uh, and Braden as a supporting director here, you mentioned the third author on the blog post has been instrumental in making sure that, um, we keep our vision on that, that agent to Karnas and moving towards the North star here. Pardon any shout outs from you? Well, it's the Asian experience team. Like this has really been an incredible team to work with. Well, thank you. Thank you for, for pairing up with all that. I'm sure that there's like so much thoughtfulness that went into this. And, uh, if you haven't seen the blog, go read the blog and y'all anything else right relief for anything else want to close off with try out the agents sdk and workers ai yeah use it right use it go go build blood stuff awesome awesome thanks so much for hanging out i hope you enjoy the rest of your agents week and we will see you