🎂 Queues Private Beta + D1:What’s New, What’s Next

. Okay. I think we are live. Hey, everyone. We are from the Cloudflare Developer Platform team coming to you with some really exciting announcements. I think we got Queues and we got D1. Leroy, why don't you take us away on talking about some of these really* exciting topics? Oh, no, thank you all. I appreciate it. Thank you and welcome again to this episode of Cloudflare TV. As never mentioned, we're really excited to share with you all what's new with Cloudflare. And again, we have today some great speakers to speak with you about Queues and D1 and all the things new relative to Cloudflare. So first we're going to start with some introductions here. Neville, Would you like to start and introduce yourself? Yep. Hey everyone, I'm Nevika, as I said, or maybe I didn't say that, but I am the PM of D1, which is our database on the edge and also the PM for Cloudflare Pages. All right. And next we have Ashcon Partovi. Would you mind introducing yourself Ashcon? Hey, folks, my name is Ashcon Partovi. I'm the product manager for Cloudflare Queues. We're excited. We just announced it today. We'll be talking about that a little bit more and excited to be here. All right. And last but not least, we have a Glen Maddern. Introduce yourself, Glen. Hi, everybody. I'm Glen, the lead developer on D1. I'm here to talk a little bit about some of the stuff we talked about announced today. But yeah, great to be here. Awesome. All right, well, let's get started today. Today we're going to start with Queues, Cloudflare Queues. So Ashkan, tell us what Cloudflare Queues is and how does it work and what does it do and how does it fit into the Cloudflare ecosystem. Yeah, so our team is really excited to expand the developer platform portfolio of products with Queues. So if we go back a little bit to the inception of Workers, when we introduced Workers, it was very much our first compute primitive at Cloudflare, and so it enabled developers to kind of write JavaScript code, be able to customize the websites, start to build applications. We quickly realized that any serious application needs storage, and that's why we introduced Workers KV and a whole slew of new storage products that have gone all the way to D1 now, but that includes KV, R2, Durable Objects. And one of the things we started to realize in feedback we've gotten from developers is that while compute and storage are great, there is a missing piece that can tie those two together and very much that comes out to be messaging. And so you need some way in order to have messaging that goes between your compute and your storage and even just supplements just compute. And so that's why we launched Cloudflare Queues and Cloudflare Queues is very much a durable, reliable message queuing service. So it allows developers to send and receive messages where there is a guarantee you will deliver those messages. And that's really important because often to guarantee delivery of messages is it's simple to say, but it's actually quite hard in practice to ensure and guarantee. And so whether you want to batch messages and kind of do batch processing or maybe you want to send a data from one worker to another worker, Cloudflare Queues is a new way that you'll be able to kind of set that up all together. And of course we'll we'll talk about that a little bit more. But we're really excited to see what developers build with Cloudfare Queues it's in private beta. So we're announcing the private beta right today. We're going to start to onboard more developers and get feedback on our APIs and how it works. And at some point we'll eventually graduate that to an open beta where or anyone is free to try it out and get started. It's great. Can you just talk briefly a little bit more about how it's integrated into Workers and what kind of things can be built with Queues? Yeah, absolutely. Just like our other storage products where there's kind of a direct integration with the Workers runtime. So when you create a Worker, we have this concept called the binding. It's essentially a way of connecting to an external resource like R2, like KV and D1. With Queues, we're doing the same thing. Where all you have to do to set up a queue is you go into our dashboard or use our command line tool for Workers, which is Wrangler and you'll be able to just define a queue, give it a name and you'll be able to just start using it in your workers code immediately. So there's no complicated setup. You don't have to install another tool or figure out how to get it to work. It's just kind of built into the platform and you can get started with it right away. And so there's two types of ways that you can interact with a queue you can produce. There's a producer in a consumer. So a producer is something that is going to send messages to the queue. So let's say you might have a worker that's doing some type of processing, but you don't want that processing to delay the execution of a request. So maybe it's kind of an asynchronous task. Maybe you want to save logs or certain metadata to process later, and so you can just send that off to a queue and you'll get a guarantee from us that we've written your message to disk. So we've we've confirmed that we've got your message. We're going to make sure it gets delivered. And so that's on the producing side. And then separately, the second part is on the consuming side. So now you have all these messages. They're kind of lining up, waiting to get processed. And what we've enabled you to do with Cloudflare Workers is essentially define a worker that will consume those messages off of the queue. And so we have an ability to define a new event type, a queue, event type. And so any worker will be able to define this event handler and consume messages from the queue. And so this gives you a lot of flexibility on building more complex, robust applications where one application is gathering data and then the other application is doing the processing. Of course you can do this all in one application as well. We also allow you to have a producer also consume itself, which is really cool. It's all just integrated in the Workers runtime and just like our other storage products, it just kind of works out of the box. And as far as costs, are there any cost impact to using cubes or having an integrated with Workers? So with Queues, we really wanted to continue the trend of having transparent pricing. And so with Queues, it's very simply priced on the number of operations or messages that you send and receive to the queue. And so we've priced it at about at $0.40 for every million operations. That is exactly in line with what some of our competitors like Amazon's SKUs charge. The big catch is that we do not charge for egress fees with Cloudflare queues. So just like with R2 and Workers were waiving egress fees entirely. When you look at the competition and how developers are using comparative queue products, bandwidth and egress is a large amount of what your monthly bill comes out to be. And so by waiving the egress entirely, we're able to allow developers to build kind of really global applications where you don't have to worry about the region or where a message is coming from or being received. It just works because you know that you're not going to have to pay for those extra fees. We're really excited about continuing this trend. We think a lot of developers are excited when you don't have to worry about recipes. Certainly it is a huge burden off of bills and not just bills, but just kind of developer's mindset. And so we're really excited to continue with Queues. Definitely zero interest fees is something to be excited about. So we appreciate that. Where can our viewers find more information about Queues? And you mentioned it was in closed beta. Is there some way that our viewers can actually access or sign up for the closed beta? Absolutely. So we the waitlist for joining Cloudflare Queues is open today. We posted it in our blog post that went out this morning. And we also have in our documentation, we've added a section on Queues. So developers, go to cloudflare.com/queues, you can see how Queues will work, the APIs, the configuration, and at the top there'll be a place to register for the beta access. Once you get beta access, we'll send you an email, we'll give you some more information on how to get started. But it's very much you can read it today, even if you're not in the beta and see how the product works, limits, pricing, everything is in there. So you can take a look and we're definitely open to feedback. So in our developer Discord, we have a Queues beta channel. Definitely feel free to leave comments, questions or other feedback in there. You've certainly got a lot today already. The team is pretty active in there, so we're really excited to hear from you all. And yeah, you can. We'll start opening up that beta waitlist soon in letting people try out Queues. It sounds great. I appreciate it. Thank you again for sharing that information with Queues. I think one thing I might add on Queues is in as we'll talk about D1 in a moment. We also see an opportunity where we're going to start to integrate with not just Workers but other Cloudflare products in our developer platform. So you might imagine you have an R2 bucket and you want to process kind of events or lifecycle events that happen in that R2 bucket. And so you might send that to a queue and have be able to process those in a queue. In the future we might have something where you can use a queue and then. Trigger operations is equal statements to to send to a one database. There's a lot of really interesting things that we're thinking about and considering. So Workers integration is just the start. Our plans and ambitions are greater than that and we're excited to see what people want us to build. Exciting news and that's a great segway into the D1 with Nevi. Never mind talking about the new news of day one and how it works and what we can build with D1 and all of the expectations with D1. Yeah, absolutely. Well, I just wanted to say congratulations to Ashcon and team on the great announcement. We're all really, really excited about it and I know it's been a long project in the works, but yeah, so today one had more of an update rather than a big, big announcement. But I think it's really important to kind of show the journey of where we got with D1. So back in May we announced that we will be launching D1, which is our database SQL database on the Edge. D1 is built on SQLite, which a lot of developers know and love. And on Cloudflare today we have a lot of different storage options, right? We have D0, R2, KV. When I talk to users, a lot of them say, Well, we're using KV, but I want to be able to query on my data or I want to query by multiple fields, which isn't really possible today with KV or just provides a lot of complexity. And so what we really want to provide was an easy way for you to query your data and really just drop it into your project that you're building on Cloudflare already. So that is a little bit of an overview of D1. What was the other question, and what can you build with D1? You can build almost anything, right? But some caveats there. So anything that requires the database, right? So you build an e-commerce site and you want to query your product inventory, you want to have a travel reservation site, anything that kind of requires you to store your data and also piece together certain parts of that data that you want to show in your application right now. I guess the expectations around D1 and within the beta are we're looking at kind of smaller projects, right? So what kinds of things can we build with smaller sized databases and how can we kind of grow that to larger databases in the future? All the things that we're kind of receiving feedback on and kind of testing the limits of what we can actually build with D1 today. And so, yeah, and thinking about what was announced today, looking at all that we've achieved in the past couple of months, the biggest thing I wanted to harp on is Cloudflare is D1 is designed for Cloudflare Workers, right? So you can create kind of this really great developer experience with D1 by accessing D1 through Cloudflare Workers, you can interact with our regular CLI to manage and update and maintain your database. We know from kind of previous feedback and from users of Cloudflare Workers today that they know and love the Wrangler CLI. So we thought this was a really important thing to be able to prioritize. And so you can when you're part of the private beta, you can kind of test out and see the different commands that we have implemented with Wrangler, which uses Wrangler 2.0 by the way. And then we know that not everybody kind of wants to use Wrangler CLI or maybe wants a more simplified experience. And so we've also been able to release an interactive dashboard for you to not only create your database but also create tables within your data, within your database, and then kind of access things like access, things like backups, downloads. You can bootstrap your database all within Wrangler and the dashboard. What else do I want to say? I think the goal that we're trying to achieve here is kind of what the blog post title was, which is our quest to simplify databases, right? Creating a database is always this really complex, heavy kind of task for developers to do. And sometimes you need a database expert. Sometimes you will try to make a database, but you don't have all the tools to do it. And we just want to simplify it for you. We want you to be able to get your database on our network within a couple of seconds. And I think a lot of that, I guess these talking points that I'm saying reminds me a lot of pages where building a website can be difficult too. But Pages kind of aims to simplify that experience as well as kind of the target of our developer platform in general. So those are some of the things that we announced today. And of course it's still in private beta, so not all of you may have access, but we are going to be releasing our documentation for developers to kind of take a look on how this experience actually plays out. We will also be kind of clearing the queue of people who are in the private beta waitlist right now. So expect that in the next week or so. And then, Glenn, I thought you could give us a little insight on what we're working on next and this big difficult challenge of transactions. Yeah, I'd love to. I think it's really interesting that we we title this blog post in quest to simplify databases. And I think something that Cloudflare does in general is try to present the absolute most simple interface to these things. You see it with queues the way how easy it is to attach events to a queue or to respond to a queue to chain together multiple queues or multiple Workers. We really trying to give you these, these building blocks that are powerful, but also that make sense to developers and, and as, as somebody who works on building these and designing these APIs, that's a, that's a challenge. That's some that's really kind of exciting because you effectively get a blank slate to be able to design these things. And then you have to choose exactly what to expose. One of the problems that we've had is about how to expose database transactions. So anybody who's familiar with databases will expect that a database product has transactions. But because we've been so simplifying of things, you don't really separate your code into a bunch of backend workers and front end workers. They're all just workers and you just query these things, you send queries to them in batches. Maybe if you need a bunch of things to have together. But we really want to give people the power to do a bunch of work together. So for example, make some changes in the database and then go and tell a third party service like a credit card charging service that a transaction is now happened. Those things need to happen together and need to both succeed or both fail. But when we're trying to make everything so simple, we've got to figure out a way to allow users to specify, Hey, these things need to work together. And at the blog post today, we talked about one specific area where that's that's actually kind of difficult to keep the simplicity low. And we had to invent some new things. And what we've put in the blog post is actually a proposal. And this is something that's kind of exciting too, that the one we're really working with our private beta users to understand what they how they want to work with these things, because everybody has expectations about how databases should work and other databases that they've used and liked. And we really want to run this as a collaborative process. So in this blog post, we are proposing or describing something that may or may not be the exact syntax that we end up shipping, but it's it is the best example, it's the current implementation and we want to hear feedback. We're getting some good feedback through Discord. The discord is absolutely the best place to get in touch with us and collate that information. But fundamentally we have this problem to solve with D1, which is that if you work as if your users on one side of the world and they need to do some some work that needs to be coordinated, needs to be consistent, that computation has to move to wherever the primary one database is. And this giving starting to invent new kind of primitives where we say, Hey, you can actually identify a large section of your worker that's actually going to run next to the database rather than next to the user. And half way through a piece of operation, you can say, actually, I'm going to hand off from, say, I'm in London. Maybe my database at the moment is running in Seattle and hand off that computation and execute it the other side of the world. And this is an interesting kind of step in this direction where we started to develop as a bit more control of how the code is executing. But in a way that's sort of pretty new and like it's a little unexpected that maybe a piece of code that you're writing is suddenly running on the other side of the world. So how do we make it debug able and how making logging all work and that sort of thing. But in general, we sort of have this situation where D1 gives us an opportunity to bring in the power of Workers and the same way that Queues does as well that other queuing products might be very configuration driven because that's how they, that's how the ecosystem works. Whereas Workers is such a powerful kind of primitive that it's usually better to write some code, to connect to things than it is to try to design a configuration language that can specify these things. And so it's a pretty it's a pretty cool concept to be starting to introduce the idea of, well, okay, I'll run, put some stuff on a queue, and then I run a worker that puts on another queue and that goes into hour to bucket. I'm starting to connect these things and Workers becomes the glue of those things. And then to enhance work is to be like, okay, well actually this chunk of code inside my worker, it needs to run really close to the database because of the work that it's doing is well, I mean, read the blog posts, have a look in. And see what you think of the syntax. I think it's going to work really well and the demos and the people that I've tried it to. It looks like what we'll end up releasing to everybody. But we want to be cautious because it is pretty new. So. So yeah, if you take a look at it and let us know. And I think that's the point, right? Like this whole concept of D1 and what Glen just kind of described as is new territory, right. Like it's new, it's a new concept that maybe you haven't seen before. And so we really like to lean on our beta users and the people who are using the product to drive the roadmap forward. And that's something I think a value that is instilled in all of club player teams as well. So as we start rolling new folks into the beta, we would love to hear, as Ashkan said, about cues, but the same goes for D1. We love to hear your feedback. I definitely take time to interview my users and kind of understand the things that you're building and yeah, hopefully can get your wishlist items on a roadmap. That's great. No, it just continues to move forward as Cloudflare continues to enhance and simplify the developers experience. So I, think, Thank you all for sharing. I'd love to see. Yeah, like in this private beta, I guess something like entirely built on Cloudflare, right? The database aspect was kind of that missing piece in, in full stack applications. Like I said, we have other kind of storage offers as well, but our storage offerings as well. But yeah, I'm just really excited to see, yeah, I guess people building those applications that are fully built on Workers. I think just to add to that, there's also this idea that every new product that can talk to every new kind of feature inside of Cloudflare has a multiplicative effect. So at the moment I haven't done anything to make rd one talk to Queues, but that isn't very difficult. And if that adds, that adds an extra kind of connection between those two things. And suddenly you can think about a database over here that has a particular purpose, but every time a row is written to it, it emits an event onto a queue and that gets collated over here and some of the process happens. It is really we're starting to see real virtuous cycle with each of these new things, enhancing the other features. And it is great to see people be able to to ship a full stack product entirely on Cloudflare things. But it's also just great to expand the set of things that you can do under one developer ecosystem using the one kind of zero risk policies that we have. And it does kind of feel like the better we make this, this platform, the better experience we can give to everybody, regardless of what they're building. And with the addition of these two products as well. It sounds like sky's the limit is what to developers can actually build with Cloudflare. So well again I thank you all Nevi, Ashcon and Glenn for sharing your time and what's new with Cloudflare on Cloudflare TV today. Again, viewers can get more information from our blog at blog Cloudflare and also on our general website at Cloudflare. Any last moment? Sorry. Any last comments or remarks from you guys? Now just get ready for another great week ahead of announcements and happy 12th birthday to Cloudflare. So much we've. Accomplished. We're just getting started. Just getting started. 12 is like. So even after 12 years, we're just getting started, I think. I think developers are really going to be excited to see what we have cooking. Yeah. Listen. All right. So, again, we thank you all for your time. Thank you for sharing on what's new with Cloudflare. Again, check us out at Cloudflare or our blog at blog, Cloudflare. And we'll see you again soon, the next episode. We're betting on the technology for the future, not the technology for the past. So having a broad network, having global companies now running at full enterprise scale gives us great comfort. It's dead clear that no one is innovating in this space as fast as Cloudflare is. With the help of Cloudflare, we were able to add an extra layer of network security controlled by Alliance, including one of the DDoS Cloudflare users. CDM also allow us to keep costs under control and caching and improve speed. Cloudflare has been an amazing partner in the privacy front. They've been willing to be extremely transparent about the data that they are collecting and why they're using it, and they've also been willing to throw those logs away. I think one of our favorite features of Cloudflare has been the worker technology. Our origins can go down and things will continue to operate perfectly. I think having that kind of a safety net provided by Cloudflare goes a long ways. We were able to leverage Cloudflare to save about $250,000 within about a day. The cost savings across the board is measurable, it's dramatic, and it's something that actually dwarfs the yearly cost of our service With Cloudflare, it's really amazing to partner with a vendor who's not just providing a great enterprise service, but also helping to move forward the security on the Internet. One of the things we didn't expect to happen is that the majority of traffic coming into our infrastructure would get faster response times, which is incredible. Like Zendesk just got 50% faster for all of these customers around the world because we migrated to Cloudflare. We chose Cloudflare over other existing technology vendors so we could provide a single standard for our global footprint, ensuring world class capabilities and bot management and web application firewall to protect our large public facing digital presence. We ended up building our own fleet of proxy servers such that we could easily lose one and then it wouldn't have a massive effect. But it was very hard to manage because we kept adding more and more machines. As we grew with Cloudflare, we were able to just scrap all of that because Cloudflare now sits in front and does all the work for us. Cloudflare helped us to improve the customer satisfaction. It removed the friction with our customer engagement. It's very low maintenance and are very cost effective and are very easy to deploy and it improves the customer experiences big time. And Cloudflare is amazing. How short is such a relief off? It is very easy to use. It's first. Cloudflare really plays the first level of defense for us. Cloudflare has given us peace of mind. They've got our backs. Cloudflare has been fantastic. Fantastic. I would definitely recommend Cloudflare. Cloudflare is providing an incredible service to the world right now. Cloudflare has helped save lives through Project Fair Shot. We will forever be grateful for your participation in getting the vaccine to those who need it most in an elegant, efficient and ethical manner. Thank you. 80.