Originally aired on March 27 @ 10:00 AM - 10:30 AM EDT
Join Cloudflare Director of Product Marketing Dawn Parzych, VP of Product Management Aly Cabral, Director of Product Management Jon Levine, and Product Manager Alex Krivit to learn more about what launched today for Developer Week.
Read the blog post:
Visit the Developer Week Hub for every announcement and CFTV episode — check back all week for more!
Hello and welcome to What Launched today in Developer Week. I am Dawn Parzych, the director of Product Marketing for the Developer Platform, and I'm here today with three of our authors. It is our second day of Developer Week, and today we have lots of announcements related to R2, or R2 related announcements. I'm going to ask the group to introduce themselves and then we will dive in. So Aly, can you kick us off? Hey, guys. I'm Aly Cabral. I run our product team for a developer platform that includes R2, but also Workers, Edge Compute, all of that good stuff. We just announced Super Cloud, all of that under our purview here on the developer platform side. I also wanted to mention that today I'm going to be talking about the Migrator the super sleep mode. I'm very excited about all of that. Big Alex. Yeah. Thanks, Dawn. Hi, everybody out there in Cloudflare TV Land. My name is Alex Krivit, and I am the project manager for Cache at Cloudflare. And I'm going to be talking today about an open beta announcement that just went out this morning called Cash Reserve. So super excited about that and happy to dive in a little bit more with you today. Great. And then last but not least, John. Hello, everyone. I'm John Levine or JPL. I'm director of product here, working on all of our data products, so our logs and analytics across our whole product portfolio. Today, we're going to be talking about a new product we just announced today called Logs Engine, which is super cool, which will let you store and analyze your logs on Cloudflare and it is powered by R2. So one of the things we're focused on this Developer Week is not just about the products and those announcements, but also how we are building on it and how our customers are building on us. So we're featuring a lot of stories this week from customers, from partners at internally to show people it's easy to say, Hey, you can build anything on Cloudflare, but to actually show like, here's some of those examples of anything. So we announced the general availability of R2 in GA week. Six weeks ago. Such a long time ago. And since then, we've seen tremendous feedback and uptake on it. And one of the things we've been asked about is migrating data from existing cloud providers to R2. So, Ellie, you mentioned this migrate or sleeper that we've announced. Can you tell us a little bit more about that? Absolutely. So I'll talk about where we are today. So today what just went into closed beta and anybody can kind of go to the dashboard sign up for is basically a one time migration managed migration from Cloudflare. So you kind of put in your S3 bucket URL. We seamlessly migrate over those objects and upload them over to R2. Now we have some big next steps though you might imagine, right? If you have a big data set, things don't move over at the speed of light. I mean, they do move over at the speed of light, but not beyond that. Right? So it's going to take time to migrate petabytes of data over to R2. And you don't want to start or like you don't want to stop accepting new rights as soon as you start that migration over. So where do those new rights go? How do we make sure that we're able to kind of do passive migration while there is this in-between land of migrating over data, large data sets from your S3 to R2? And also there's going to be interesting next steps that we take kind of related to the work that the Cash Reserve team is going to talk about later today. But making sure that we're only pulling over migrating data opportunistically when we already are doing like look up from origin on cash, right? Like that allows us to kind of consolidate cost, minimize the additional overhead of a migration expense over to R2 and allows you to do it seamlessly over time. So there are lots of different options that we need to talk to you being users of R2 about and make sure that we're creating these fully managed processes for making it seamless to come onboard to Cloudflare. We want it to be a no brainer to bring your data over to us, and this is a really large piece of that puzzle. Yes, So much of what we talk about is the developer experience and how can we make things easier? How can we have it so that developers can just spend their time developing and not worry about minutia or configurations or any of these other things that are completely necessary to run applications but shouldn't be taking time and energy away from writing code and making sure that code is running as optimally as it can be for applications. Of course, and that's just a piece of the puzzle. We don't we want to we have this vision across our platform, across all of our developer products to, where possible, remove human decision making and just like make it work, right? Like everywhere, the system can make a better decision than a human. We should be pushing the system to make that decision, and we want to do exactly that same pattern here when we're talking about data migration or data movement. And that can be movement from where it's hosted today into our two or can be movement across regions in the globe. Like these problems are all related to each other. And we're really excited to kind of expand the art of the possible and push beyond the boundaries of what other products are doing in this space. Alex I see cash reserve fitting in that bucket where we're trying to like automatically just make sure that the information in the content is there to increase cache hits. So tell us a little bit about cash reserve and what it means to be an open beta. Yeah, thanks. and Cash Reserve definitely fits into this sort of goal that Aly is talking about here where, hey, if you should spend your time sort of building your building your product, building your application, building your website, doing things besides sort of network management. And that's really what cash reserve fits sort of squarely into that puzzle piece, 4 to 1 button configuration. So really, you don't have to spend any time configuring it. There are very few knobs to coming on board for cash reserve and you push this button and things are stored in cash for a long period of time, which helps you in a number of ways. It can help you by shielding your origin from needing to pay these egress fees that Ali talked about a little bit, moving your data from your wherever your data is stored to a provider, oftentimes you're charged a fee to accomplish this, to move data from one place to another. And so cash reserve helps to shield you from experiencing a lot of these origin egress fees. It also is built on top of R2, so it's highly integrated within the Cloudflare ecosystem and network. So at the end of the day, your cash data will be sitting much closer to the people requesting it. And so it will be much faster than having to serve it out of the origin. So again, it's moving things close to people saving you money and it's all requires one or two buttons. So it's like really trying to remove all of that configuration and getting you back to building what you want to build, building your service, building your application, building your website. So that's that's what R2 is or that's what it seems to be a cash reserve. It is. And it's built on top of R2 going into open beta today. So we've been in closed beta collecting feedback from users for a very long time, iterating on that, trying to get it in as good shape as possible, finding some of the initial sort of like skeletons in the closet and everything, getting rid of all of those and making it just like a really good experience, really resilient. And so today opened up and you don't have to wait. You can sign up today, start playing around with it and seeing how it can help you. And so from an availability perspective available to all existing CDM customers and how is this kind of priced so people know what to expect when they click that button? Yeah, so it is it's available to anybody who's on Cloudflare right now. You can push a button if you're an enterprise customer, you can work with your account team and make sure that you have all of the right permissions. It's priced in much the same way that R2 is priced so based on storage and operations, and those are documented both in the documentation and other places. And so you can just sort of use it out of the box and go and you should be able to see in the dashboard how much storage you're consuming and how many operations you're seeing. So there shouldn't be any sort of surprises at the end of the month or anything. You should be able to look directly in your dashboard to see how many operations are being used and everything else. It's also a good sort of metric to show you this is working. This is actually serving real traffic to your end users. So it's great. Great. And so we built cash reserve on top of R2, just like we've built our logs engine on top of our two. John We had I think two posts today talking about announcing and then how we built that. The logs. Yes, we did a little bit about what logs engine is. Yeah. So let's first talk about what you mean by logs Cloudflare logs, what are we talking about? So Cloudflare has so many products. Obviously folks tuning in now probably are familiar with our developer platform. So like Workers. But of course our CD and our DAS are WAAF. All these products we have a Cloudflare generate logs, so in the context of a worker it might be every time your worker runs the context of CD and every time an HTTP request arrives in our network, every time we resolve a task where we write a log event. And logs are super, super important, right? In our dashboard, we also have analytics, which are mostly meant for this kind of aggregate level, like what's give me the lay of the land. Like how many requests overall do I get? How much CPU time in total are my workers taking? But when you have a problem and you're trying to troubleshoot an issue or you have a security incident, you probably need the precise events. You might need to see every event that matches a certain criteria. So you might need to say, you know, classic example, this was it was about the end of last year. There was an infamous event known as Log4j and very, very bad exploit and a lot of our customers wanted to look at their logs and know, okay, well, how many times did someone try to exploit this? And you need to really see every single event that contain that possible string. And the only way to do that is actually by keeping all your logs. So why is that harder? Why is that problematic? So the first thing to understand is just that the amount of log data we deal with is just it's absolutely massive. Our network, just the CDN, is dealing with tens of millions of incoming HTTP requests every second. And then those requests come in is folks know if you're running workers, you know the workers running. So we're logging events. When that happens, the workers are making sub requests, the firewalls running, all this stuff is happening and it just adds up to and frankly incomprehensible to human amount of logs that we generate every single day. And even for customers, even our customers, what I would describe is a modest size. If you're running like a medium traffic website, like I always think of, the Cloudflare Blog is like your kind of classic, like medium kind of Goldilocks size website. Just the amount of log data generated by that is insane. And so what can you do with that? So good news is there's like all these products that exist out there in the world that are really good. I mean, we partner with a bunch of them. We can name like Splunk and Datadog. And then also famously, the big the big cloud platforms have tools like Athena and BigQuery, and these are actually great. And these problems, I don't to say their solve problems, but it is possible to analyze log volume at scale. A couple of problems, though. The main one is cost. And this is what we hear from customers all the time is they when they describe what they're spending on these products, to actually ingest all of their Cloudflare logs and just be able to run a simple query, just answer some basic questions. We're often talking about cost numbers that are like more than what they spend on Cloudflare. That's that's actually not uncommon because the thing about every single event that's coming in is generating sometimes kilobytes of data. So it's a lot. So it's very, very expensive. It's also just hard to use. So we have a lot of products, we have a lot of every log line contains a lot of fields. And you have to know what all those fields mean to know what you're looking for. Then when you find something, well, great. You found something. You use BigQuery or Athena. But so what? So now what do you do? Like, you have to go back to our product and make a rule. Like, how do you how do you even know what you're searching for in the first place? You may not even know where to start your search. And so the usability of this isn't great. And then finally, and this is more on Cloudflare are log products I don't know have only been for enterprise customers. So we said you had to be an enterprise customer to push your logs to one of these other sources. So with logs engine, of course we want to fix all those things. So first of all is cost. Let's just go through these one by one. So of course, the good news is that storage is not expensive for the average customer to store all of your logs for Cloudflare for even a year. It's just not that much money. It's going to depend wildly based on customers who have orders of many orders of magnitude versus amount of traffic. But even for a larger largest customers, it's quite reasonable. So then on the cost, it's a surge in one piece of it. So how do we how do you then make the search cost effective? So we have a few tricks up our sleeve here and we can talk we can talk more about it, because actually we did a really cool companion blog post about how we can efficiently find Redis, which is a really cool example of this. But I would say the key insight here is we really understand it's our data that we generated in the first place. And so we we understand it really well. And I think the key insight here is that we have not just these logs products, we have these analytics products with analytics products. What you do is they let you do this very high level sifting through, slicing and dicing, getting the label in so you know what you're fighting for. So when it's time to actually execute that logs query is much tighter scoped. You know, you're looking for you're not like running these queries over and over and over again, which is generating these potentially massive bills. And also I want to point out we're going to be building out these querying stuff. We're planning to build that in the developer platform itself, which is which is quite cost effective, quite efficient for these kind of operations. And then of course, that also ties into the easy use piece. We want this to be integrated with the product experience in the dashboard. So you should see the analytics, you can add filters, find the things you're looking for, and then say, okay, we want you to be able to push a button and say, Give me the logs back. Right? And then finally, when you get the logs back, we want to be able to take action on it. So we want to show those in the dashboard and say, Hey, this looks weird. Did you want to do something about this? And, you know, push a button, create a rule to stop that traffic. So that's kind of that's kind of the vision for where we want to go. And then, of course, who can use it. We're announcing today that we are planning not just you have a very soon planning to make this available for customers in all plan types. So not just our enterprise customers, but are all our paying customers. We plan to let you push logs into Logs Engine and have a way to query that out. That's like a super quick version of what launch is about Whirlwind tour. Everything that this product is going to do. Yes. I will say quick about very comprehensive. So I really like that. Yes. Alex, I'd love for you to take us on a journey of customer feedback on Cash Reserve. I'd like I think you had some really great stories in the blog. Can you tell us about them? Yeah, absolutely. I think that the biggest piece of feedback that we've gotten so far is that catchers are really fits nicely in this, Hey, I want to cash things long. I have a I have a big library of data and I know that I just want to serve it all from cash. I want it to be fast and I want to shield my origin from this egress. And so we've had really large customers and really small customers both use it. So customers like Delivery Hero is one of our first use cases. They run this global website where they deliver items to end users. They have a lot of assets that they want to serve out of cash. And they were saying that, hey, you know, it's we want to continue to serve it out of cash. We don't really want to spend any additional time or effort or anything working with this already than more than what we do to configure our cash today. And so with the push of a button, they're able to really shield their origin from egress. And they were talking recently about how it was able to really help them clarify both their origin logs and some other things so they can really get a clean picture of a what's happening on their origin because Cloudflare is just able to shield so much of their traffic. And so I think that sort of feedback has been definitely a common theme, is both that egress savings and then also just how easy it is to configure and use. And so the other customers like Blackbaud and enjoy Patron Technologies, they've all sort of come forward with sort of similar, similar stories. All of them are collected in the blog and it's just been sort of great to see how beneficial it's been to sort of customers across the landscape. Very, very different use cases, very different traffic profiles and everybody sort of at the end of the day kind of wants the same thing, which is just like, Hey, I want this to be cached quick and maybe money. And with the push of a button, you know, all of that is accomplishable, which is sweet. Let me ask you a question, Alex. Would you have been able to build Cash Reserve without our tube? Absolutely not. We really took advantage of a lot of the engineering work and effort that are to put in. And I think that's one of the major themes of this discussion today. And I'm sure, John, with the logs products can probably echo these same sentiments, is just that with a lot of the flexibility of the developer platform and then also just are to so many opportunities are open to you and you can do the same thing like in your service that Cash Reserve did or that logs did, which is like integrate and string together a bunch of these different Cloudflare tools. Workers durable objects are to caching logs and you can really come to market in a pretty quick amount of time with a really scalable, quick performant product. And it's just sort of like opens up a whole new world to a lot of developers out there. And I think that without a lot of the work that the R2 team and the Workers team, the developer platform team did, we wouldn't have been able to experiment as quickly and deploy things and sort of figure out like, Hey, how are we going to scale this Cash Reserve thing? How are we going to build on top of these things? And so we were able to, with pretty low effort, experiment and find the best way forward and a lot of these things. And John, I know you said you have a companion post talking about how. Yeah, you built. Yes. The Logs Engine on top of R2 as well. Same question over to you. Yeah. Well, I wish we'll have to do another segment with coal. Where you a deep dive? Because he'd be the he'd be the person he built it or the blog post and it's super interesting story, but I can speak a little bit maybe to the... Just kind of like the higher level, maybe a higher level. That blog post, which is we've been thinking about product like Logs Engine for a long time, I mean probably more than two years like really since before R2 is just a glimmer and this I hear and we and for in full transparency you know we'd actually consider like what would it look like to build a product like this using public cloud storage. And we looked at this and we looked at, okay, well, what if customers brought their buckets to us and we did something similar to this and we decided not to do that. And one of the reasons we decided not to do it is just how astronomically expensive it would be. Imagine if we were actually going in and reading the data out of the out of those buckets, not just writing the data in, but then reading it out and trying to scan it and do these operations, it wouldn't really work. And we decided that was not a good experience. And so that decision is really coming to fruition now. Now that we have R2. And the thing I want to say, give a shout out is like, I think it can be really hard for people to understand how hard it is to build scalable, reliable object storage. It is such a crazy, hard problem. And once that is like just speaking from our team's perspective, if that is solved and you can just kind of take that for granted, which you should never take for granted, that just makes everything so much easier. And then you can just let's you open the door to really what you can build on top of that. So, yeah, I mean, we can talk a little more about the specifics. There's actually a lot of interesting details of like in particular how we built the radio retrieval we can get into. But yeah, certainly without our to this doesn't, this doesn't exist. So and one question I want to ask all of you so you can start thinking about this is what's kind of next for these products, Aly, whether it's the Migrator or R2 in general, can you give us a peek as to where we're going with this? We have so many places to go with R2 specifically, top highlights, right? Are investing more in the mediator, making sure that we're supporting different kinds of use cases, largest data sets, always online, like REITs being active while you're migrating all of that. That's a big problem space that's going to take time to develop out. And we're we're excited about the journey there. Additionally, like in the object storage space right now, we play in the standard storage tier. But if you look around at the other object storage providers, there's often infrequent access tiers. We know that we need to build towards that to support these large data sets. They don't want all of it on more expensive hardware because if you're not actually reading the data frequently, go use cheap hardware, You don't need the throughput, you don't need the scale that you would for servicing hot reads. So that kind of infrequent access tiers I think is going to play pretty nicely. I think with logs and like long retention periods with logs, I think you just get even better economics over time as we have those tiers. If you're querying that data you don't want to need. But often those logs is a lot of people just you need to know that it's there and if you need it, it's there. That's the important thing, right? So exactly, exactly. We have customers that retain those logs even for compliance reasons for seven years, right? So those types of things, that data adds up. That data adds up real quick. And being cost effective in these large data sizes is something that we absolutely are looking toward building out in '23. And then additionally, third pillar that I'll just call out here, we're doing a lot, so don't take this as the comprehensive roadmap for our but making sure that we're investing in more like security controls customers bringing their own keys to encrypt data, more granular access per bucket access. And from a user perspective, all of that coming excited about where we're going. And John, where are things with Logs Engine heading. Yeah and it would just announced so. Yeah, yeah. I was going to give a shout out all the things that I just said are super important to our roadmap as well as you for storage. The granular access is a really big deal we're going to be taking advantage of too. You'll see that's going to like feed into our product. So cool to see these things like cocoa evolving on the Logs Engine side though. So just to be clear where we're at today, what can you do today? So you can if you are an R2 customer, you can go and create an hour bucket and then you. Go. Just push your logs into that bucket. And we have in beta right now two ways to get the logs. You can either query them by time range or you can look up by red. That's what we were just referring to. So it is like really quick on that. It's like a unique identifier for request to Cloudflare and we show them on our error pages and like to be quite honest, until now it was always like, Cool, I have a red ID for a failure. Like now what do I do? So just the ability to like go type in that red and get something back is a really big deal. Something we're really excited about. And obviously we plan to like integrate into the dashboard so you can find things much faster that way. So that's part of it. But so much on our roadmap. So a lot of this is going to be in the user experience side. So just as Alex said, like as our to get some of those more capabilities, we're going to make it to that want logs are going to be like a couple of clicks from the dashboard to set up. So you say, I want to use Logs Engine, I want to sort my HTTP logs or I want to store my Workers logs and I want them stored in the EU only and I want them stored for 12 months. Go. And that's it. You push a button and then just in the background you'll get you'll get billed for your usage. But. That's how that's how we're thinking about that experience, that sort of logs management experience. And then on the querying side, I just mentioned two very simple use cases, but obviously there's many, many, many more kinds of use cases. The simplest would be to have just better support for filtering. So just be able to say, okay, can you show me all the requests where like the user agent was this or the query string pattern had this particular thing going on? And then obviously to build UI support, obviously to build the analytics integrations I mentioned before, like that Rule Builder. So you click on that event and create the rule. So yeah, that's where we're going with this. Oh, and another important thing I should mention is the ability to really work across data sets at Cloudflare. So we've been pretty siloed. So the idea that like your HTTP logs are one thing and your Workers logs are another thing. But I think what we've heard from customers is they really want to find show me across all these data sets what's going on. So those are all things that we're thinking about. And Alex, I would assume some of the things that Ali is talking about for R2 would also be used by Cache Reserve. But what else is planned for? The next phases. Yeah, I think that a lot of the things that are is building and providing sort of that foundation experience for are going to be important for us in the future as well. I think that going forward we're going to continue to build out some visibility in terms of like analytics and dashboard integration so that you can see exactly what's in Cache Reserve, how many times things have been requested or read or written. What types of content types. And just getting really specific about how Cache Reserve is shielding your origin from egress and how much it's saving you, I think is going to be a really powerful future integration. And then also sort of for next steps as well, you know, removing some of the limitations that we have currently being able to write more to Cache Reserve, I think is going to be big. Just so that, you know, the world sort of opens up as we as we continue to remove those limitations. You can just sort of again, click a button and you don't really have to think about how things are eligible or whether things are eligible and they will be cached. And so I think those are the next sort of big steps forward for Cache Reserve. And I think that they will probably be on the roadmap and start to look for those in the coming quarters and really excited to continue to open this up to many, many more people. Great. Well, thank you all for taking the time to share what was announced this week and talk about where we're going. We're always looking for feedback when we're doing these products, and this is why we do open betas and private betas. Please join our developer Discord and share your feedback there. Our engineers are there, our product team is there, or you can write comments to us on Twitter. Everybody in the company, including our CEO and CTO, like to dive in and answer people's questions on Twitter. So enjoy the rest of developer week. And I would like to thank all of you for joining me today. I think,
