Event-driven applications with Cloudflare queues and Dapr

Hi, everyone. Welcome to another segment here on Cloudflare TV. I'm Gift Egwuenu, developer advocate at Cloudflare and with me here I have Marc. Marc, hi, thanks for joining. Thank you for inviting me. You're welcome. In this segment, we're going to be shedding light into Cloudflare Qs as well as Dapr. We've seen developers utilizing our developer platform that includes workers, KVR2 and even now Cloudflare Qs. Since Cloudflare Qs was released sometime last year, we've seen people using it. I'm excited to have Marc show us how you can use Dapr and Cloudflare Qs to build event-driven applications. Marc, do you want to introduce yourself and then share with us? Yeah, thanks. As Gift mentioned, I'm a developer advocate at Diagrid and we help developers to build and run distributed applications that's based on open source technology such as Dapr. So I've started there last January, so still quite fresh there. And it's my job to spread the love about Dapr, the open source project to build and run distributed applications across cloud and edge. I think it's a good time now if I share my screen and share some slides to explain Dapr because that's probably quite new to a lot of viewers here. So let me share my screen. All right, here we go. So well, these days a lot of organizations are building these distributed applications like you can see here. So it's just like a typical e -commerce system where you have a front end and there are some dedicated services for doing email or payment or shopping cart and checkout and so on and so forth. And each of these services have a very specific function. And each service in its own is like easy to maintain and easy to upgrade. And that's quite powerful. And that's why all these organizations are doing this. They're also like event driven. So nothing is really polling or nothing is related to batch work. So it's also quite quick and performant and also very scalable. But if you start doing these kind of applications really at scale, then it's quite tricky. There are quite a lot of developer challenges going on there. Because how do you do like service to service implication and how you want to do like end to end tracing of your calls in case you want to do some debugging. How do you handle like failed calls and how do you then apply some retry mechanism on top of that. So yeah, there are a lot of cross cutting concerns that apply to all of these services. And it's very useful then if you can use something to actually take care of that. So one of the things that you can use is then Dapr. And Dapr stands for the distributed application runtime. And it's a portable event driven runtime for building distributed applications across cloud and edge. And Dapr runs everywhere where you can run basically Kubernetes and all containers. So it runs across many of the cloud providers or just your own virtual or physical machines even. What I really like about Dapr is it really speeds up the microservice development because it offers some building block APIs. And this is a list of all the building blocks. So with Dapr you can do like service to service invocation. You can handle your state management. You can do publish and subscribe. You can have input output bindings. And in this demo we'll do soon we'll use the Cloudflare queues output binding. You have observability built in. You can use the actor model. You can use like different secret stores. Use configuration stores. And these days these are the two latest newest ones. You can even apply some architectural patterns such as distributed lock or long running workflows. So it's quite a broad spectrum that Dapr offers. It's really focused for developers building distributed applications. So how does Dapr actually work? Well it's built on a side core model. So if you're familiar with containers and Kubernetes you probably know about this. But if you don't Dapr runs in a separate process next to your application. And you communicate from your application with the Dapr API. And here are just some examples that use the HTTP protocol. So in case you want to invoke a method on your application you can use this invoke command. Or in case you want to retrieve some state or want to publish a message or retrieve a secret from a vault or start a workflow. So these are typical things you can do with Dapr. And you can use any language that supports HTTP or gRPC. So that's probably a lot of programming languages that you can use. But it's even easier if you use one of our client SDKs. So these are the top five client SDKs that we have. There are more but these are the most popular ones. And in the demo that I'll be doing soon I'll be using the JavaScript SDK. So Dapr is a CNCF project. It's an incubation maturity level. And it's actually the 10th largest CNCF project. Which is I think quite an achievement because there I think there are over like 200 CNCF projects. And it's also doing quite well in the community. So it's quite popular with a lot of different contributors and quite a big Discord community as well. So in case you want to know more or want some help Discord is the place to be. And there are a lot of contributing organizations for Dapr. So although it originated from Microsoft these days a lot of big organizations are actually making contributions to Dapr. And there's also like a very big number of organizations that are using Dapr. So talking about these building block APIs. So from your code you call these APIs and you can see this building block APIs as sort of interface. But for instance when you want to save some state or want to retrieve some state there needs to be an actual database or key value store behind that API. So those are called components and there are literally dozens and dozens of components. So for instance like an example for the state management want to have things for AWS, Azure, GCP. But there's also one for Cloudflare KV. So the same goes for all of the different APIs. There are really a lot of components for all of these. And in this session we're going to talk about this binding for Cloudflare queues. So just final few slides just to explain how it works. So from your application and you want to for instance store something in Cloudflare KV. So normally you would use the Cloudflare API right like a like a packaging application. But in this case we're not doing that because we are using the Dapr sidecar. So we talk to the Dapr sidecar with the Dapr API. And we can also we can use the client SDK for that in this case JavaScript. So we have these save and get operations we can use. So your app in this case doesn't know anything about Cloudflare. It only knows about a component name and that's it. But then Dapr needs to be aware of that it needs to save the state in Cloudflare KV. So how does it work? Well there's actually a yaml file that contains the configuration. So when your application is loaded and your sidecar is loaded then Dapr knows whenever I see a component name called my storage that means I'm using a specific storage type called state.Cloudflare.workers.kv and that's the way Dapr knows where to connect the storage with. So this model makes it actually very easy to switch the underlying storage. So what you actually can do is provide a different yaml file and point it to a completely different storage such as Redis. So your application code stays exactly the same only this one yaml file changes that points to Redis. So this makes it really powerful for instance to do like a local development completely locally using a local store maybe a local container using Redis or using a SQLite instance as your local machine. But as soon as you deploy your workload to the cloud then you switch to your Cloudflare .workers.kv. Interesting. All right. I'm curious in the KV side you typically need to set up for example your binding how does then how do you connect that because I see this state.Cloudflare.workers.kv type. Yes there is some additional metadata so I'm not showing it here but there's a list of metadata and some of the things in there are like an API key and they had some rules and things like that. So that's typically part of the metadata and although you can do this while doing like development and testing put the metadata in this yaml file but when you go to production you should use like a like a secret vault and then you use your secret references in here and then under the hood it'll be retrieved from the secret vault. Okay. I'll be actually showing that in the demo that we're going to do. So good question. All right. So let's see the demo. Exactly. So what I'm going to do is just to go through a readme that is in a GitHub repo. So it's a Diagrid Labs Dapr Cloudflare queues. So I've opened it up in VS Code here. So this repo consists of a producer application and that will be our Dapr application and that will post messages to a Cloudflare queue and that Cloudflare queue is pushing the messages to a worker that's subscribed to that queue. So this is what we're going to build and the application the producer application is already here. Also this consumer worker is already here and but we still need to deploy this to Cloudflare. You still need to create this queue in Cloudflare and we'll be using a Wrangler CLI for that and the application is just running locally on my machine. So I'm talking about prerequisites. In this case we're using Dapr so Dapr works with a Dapr CLI. That's already installed here. In the case it's a Node application so we need Node.js. We need to have Wrangler installed and since we're using Cloudflare queues it's part of a paid plan so you have to make sure that you're also using a paid plan and in case you haven't enabled queues yet you need to enable queues in your Cloudflare dashboard. So let me also have a look at the Cloudflare dashboard by the way. So this is the Cloudflare dashboard and here is the queue step. So I already enabled queues but if you didn't then there will be a button here from please enable queues. All right scrolling down a bit. So what we're going to do again we're going to create a Cloudflare queue. We are going to publish a consumer Cloudflare worker that reads messages from the queue and then we will run this producer Dapr application that will publish messages to the queue. So I'm already logged in with Wrangler so I don't need to do that anymore. So the first thing we're going to do is actually create the queue. So we can use Wrangler queues create and then the name of the queue in this case is Dapr messages. So maybe zoom in just a little bit more. Yeah I was going to say to close the site thing but this is perfect. All right yeah so it's created the queue named Dapr dash messages so that's there. Perfect so we can actually verify that in the dashboard and there it is. Okay so that's perfect. So now let's have a look at the Cloudflare worker. So the readme also includes steps to actually create one from scratch but we already have one in this repo so I'm not going to create from scratch. I'm just going to publish it but before that I'm going to show you how it looks like. So I'm in the consumer folder now source. This is the TypeScript file. It's a very small function. This is it. This is the method. You have to name it queue and then it can receive a batch of messages. We're just going to stringify those messages and then we're just going to output that to the console. So nothing very special going on here. I saw that you also added a few things to the runglab file. Yeah exactly that is not very important. Yeah exactly that does make it a good question. So of course I have to give this a name. It needs to refer to the TypeScript file but this is very important. So here you indicate that this worker is actually a consumer of a queue. So this is important and you have to specify the name of the queue otherwise it doesn't know where to subscribe to and you have to specify a max batch size of which the number of messages you want to take in at once. In this case I've set it to one because I want to show each individual logging message but you can definitely set this to a higher size because it's definitely more performant if you do some batching for receiving messages. But in this sample we're only going to send 10 messages so that's not really important. All right. So I'm now going to publish this Cloudflare worker. So there we go, runglab publish. It's now uploading it and again we can check the the dashboard if it's there and there we are there's the consumer. So that's perfect and the next thing I'm going to do is I'll start a log tail because we will send messages from the producer app to the queue and then the consumer will consume those messages. It's nice if you can have a look at the log that these messages are actually arriving there. So wrangler tail and it will start a log tail log there. Yeah that's good detail. Perfect. All right so now it's time to look at the producer side. So that contains of two parts. So we have the application itself. So I'm in the producer folder here. So if we look at the index file. So this is the producer app code. This is the function. We're using the Dapr client so we're using the client SDK for Dapr because it makes it a bit easier to interact with the Dapr sidecar. We're specifying a host which we are calling in the SDK and this is then the main function. So what's important is we need to refer to our binding and the binding has a certain name Cloudflare queues and that is specified in this yaml file I spoke of earlier and we'll have a look at that very soon. But this is the only thing that you need to use in your application to refer to this binding component. The binding operations that we're using is in this case publish. So this is different for each binding component but in case this is a queue it's quite natural to have like a publish operation on a queue so that that is the name of the method we'll be calling. Then we create an instance of the Dapr client and then we do a for loop and then we create a message like hello world with an incremental number and here we're using the Dapr client and then we use the binding API and then we're going to send something to the binding that then contains the binding name so Cloudflare queues. It contains the operation which is publish and it contains the message that we're sending. Okay and then we sleep a second and then we go for the next iteration. All right so the other important part here is under resources that's the binding yaml so I'm going to show you this template file. So this template file is checked into a source code because that doesn't contain all kinds of sensitive information. The ones that the yaml file my application is using is this one because that contains all kinds of keys and secrets and which I'm not showing here in this live stream. So if I scroll all the way up here in this template file so this is the name that our application is referring to Cloudflare queues. So our application is using this and then the Dapr runtime sees okay I see this component name what is the type of component name that belongs to this Cloudflare queues. In this case we're using the bindings of Cloudflare queues type of binding and then there are some different metadata involved here. So you can actually use this binding in two ways. What happens under the hood is that Dapr will actually create a Cloudflare worker for you and it will send the messages to that Cloudflare worker and that Cloudflare worker will then put the messages on the queue for you because the only thing that can interact with Cloudflare queues are Cloudflare Workers. So we are not directly sending a message from a Dapr app to the Cloudflare queue. No message is sent first to a worker and that worker is then responsible for delivering that message to the queue and then you can choose either to have Dapr creating that that worker for you or you can provision a worker yourself. In this case I'm choosing just to have Dapr create that worker for me. So what's important here is that you specify the name of the queue that you'll be publishing to. So it's Dapr messages. Then there's the name of the worker that Dapr will create for us. So I just named it Dapr message worker. Then there's a part that you need to have to create a private key to do secure communication and that's described in the in the docs. There are some links there. I won't go into detail into that but you can just follow the documentation. Then you need your Cloudflare account ID where you can get that from the dashboard and you also need to have a API token for Cloudflare and that's probably useful if I show that. So if I scroll down here so I have this your account ID that's required but there also like API tokens. So you need to create an API token that the Cloudflare worker can use. You have to give it a name and you have to give it permissions and let me zoom back out a bit. So the only permission you need for this key is account.worker.scripts. So that's the only permission that is required. All right and then the last one value is optional in case you already have a worker provisioned but in this case I have no value for this because I want Dapr to create a worker. So this is the template file. This is the actual binding file. I'm scrolling up a bit so that contains the same information but then filled in with the actual keys and API keys and so forth. All right so I think it's time now to actually run this application locally. So again I can use the Dapr CLI to run this application. So I do Dapr run. I have to give the application an ID so that's always required. In this case I'm just calling it producer. I'm then referring to the resources path that contains the YAML file. So Dapr will read this file then it will know I need to connect to Cloudflare and then the last part of the CLI command is npm run start so it actually runs my node application. So let's run this. There we go. So there is some output that is shown. So now we see that it's deploying an updated worker to my Cloudflare worker URL. So now we can see that Dapr is actually deploying a new worker and this might take a while this process especially if you do it for the first time. It might take up to one or two minutes so we probably have to wait a bit. We can actually go back here to the dashboard to the worker. It does publish it to your profile so you should see a new one. Okay yeah yeah so we see it's created it's good but then Dapr needs to do some checks as well so if it's there so it does some some sanity checks and that initially might take a bit but okay it's actually done and actually we were too slow because it already it already also published the messages which Dapr followed. But let's run it again while looking at this screen now. So we run the same command again because now it will be faster because the worker is already there so it doesn't need to do that again and then at the right hand side we will see the output again of the of the log tail. Okay let's do this again. All right so now the application is publishing and we see new records appearing on the consumer side that contains only hello world statements. So we have your neat yeah selling messages from a Dapr application to Cloudflare queues and a worker is consuming it there on the edge which I think is pretty neat. Yeah I agree and the process to get it open running is also pretty seamless as well. Yeah exactly yeah yeah there are some things you need to take care of right so make sure you have the right API keys and you need to create a secret but everything is quite well documented and it's it's I think everyone can can follow this this readme that that I'll share. And then there is more information even also in the this my my docker setup there's more information on on the Dapr side as well. So if you look on the Dapr docs so this is the actual specification of the binding that we're using so that also contains a lot of information that I'm referring to in my github repo. I also would like to add on the Cloudflare side so for Cloudflare queues your consumer the output data that is coming from your consumer you can also store that in R2. So R2 is like Cloudflare's like the NoSQL store right yeah yeah yeah yeah. It's more like I'm trying to remember the exact words I'm sorry. That's okay yeah no I haven't tried this yet but I think it's nice yeah yeah I also like it's more like a yeah like a bucket. It allows you yeah storage bucket how the word I was looking for was unstructured data so the the data that you get in your console you can then batch that up and store it in R2. So if that is also what is necessary or required of you and to do that on the consumer side you just need to create an R2 binding and create a bucket. Right yeah exactly yeah yeah oh yeah that that sounds quite quite easy yeah yeah so I have so of course the consumer now doesn't do anything it just did console logs but then when you mentioned I can just add a binding in this function and it's like store in the bucket. Yeah that's nice yeah it can be a nice extension. Yes one one question before we wrap up. This was great thank you so much for sharing how to set up Dapr and also sharing the magical keys as well. I remember you shared an example um e-commerce application in your in your slide. I'd like for us to talk briefly about some use cases of this. So if someone is looking to actually do something with Dapr and Cloudflare keys in your application what specific use cases have you seen that people are looking looking at? Right well I haven't actually seen uh anything yet that that that makes this cross over from any generic cloud platform to to the edge so I'm also very interested in seeing real like production use cases for this but but I think it's I think it's a great opportunity because usually people either leave their workload in one cloud or maybe they do like a cross cloud thing and using the major cloud providers but I haven't seen actual like examples where they go from one cloud to like a like a like an edge globally distributed system such as Cloudflare. So yeah I think this this solution has a lot of potential right because for the end users it'll be much much faster if you have published two queues and publish two workers which are close to to end users. So yeah I think performance performance wise I think it can have a lot benefits if you yeah move your workloads to closer to the user. Yeah I agree I agree as well. Um that's interesting so if anyone is looking to find out more um about Dapr or even this demo you want to see the demo um the the github repo is yeah yeah yeah so yeah so this is the github so Dapr labs and Dapr Cloudflare queues apps yeah so that's that's a good one but another good one would be uh I think just the docs is also I think good so docs.dapr.io okay that's always a good good start and yeah like I mentioned before we also have a Dapr discord so and there's a separate section there for for bindings so in case you want to know something more about the Cloudflare binding or also the the creator or the maintainer of the Cloudflare binding so who created it is also there so if you want all of the in-depth information about it you can you can ask some questions there. Great um I don't have a slide to show mine but also if you have any questions around Cloudflare queues in general we have a discord as well and there's a queues channel there where you can ask questions or just see what other people are also building um using Cloudflare queues and yeah with that we will call the segment a wrap it was really nice to have you on here Mark. Thank you for sharing. My pleasure. Great all right thank you everyone for tuning in until next time have a wonderful day.