How Cloudflare Built this

Welcome to this week's episode of Cloudflare TV, or how Cloudflare built this on Cloudflare TV.

This is the third episode. We did two last week, an intro episode and then a sort of deep dive on some of our analytics products.

And this week, we're lucky to have as our esteemed guest, Rita Kozlov.

So, Rita, do you want to introduce yourself quickly?

Sure. Thanks so much for having me. So I'm Rita. I'm the product manager currently for the developer productivity team for our product or platform, Cloudflare Workers.

Cool. I guess I should introduce myself too.

I am Rustam. I am a director of product here at Cloudflare and I manage part of our product team focused on our performance and networking suite.

But lucky enough to work really closely with Rita and excited to walk through workers and what product being a product manager on workers actually means.

So before we go deep on workers, why don't we talk a little bit about Rita, the worker?

Yeah, so you've been at Cloudflare for a couple of years now.

What did your sort of career path look like prior to Cloudflare?

And then what have you done since joining? Yeah, so I've been at Cloudflare for four years as of like a week ago, which in tech terms, I really honestly feel like makes me pretty ancient when people ask me how long I've been here.

But yeah, so I started out as a very innocent, bright eyed student at Georgia Tech.

Luckily, did not pursue like my original idea of what I was going to do, which was do foreign affairs.

That was like something that was really exciting to me in high school.

And then I ended up taking AP Computer Science in high school.

And I remember the teacher kind of asked me like, what, what do you want to do with foreign affairs?

And I was like, I don't know, like, maybe I'll go be a lawyer.

I'm pretty good at arguing. Which turns out a pretty good PM skill as well.

And he was like, well, actually, if you want to do that, you'll stand out more if you're coming from a computer science background, because like there are so many poli sci and like international affairs students already.

So yeah, my original plan was to double major. And I don't know if you know this, but Georgia Tech is pretty notorious for people sticking around for a fifth and sixth year.

And I didn't want to do that. So yeah, very quickly, I was like, let me stick to one thing.

And that thing was CS. Cool. So so you you graduated?

What what came next? Did you resort to Cloudflare? Or what what was the next step?

So the next step is pretty interesting. So when I was at Georgia Tech, I did a couple of internships, both of them at Microsoft.

I guess we have that background in common.

And yeah, I was like, Okay, do I want to go back to Microsoft and do the big company thing?

And I thought, I'm young, I can take risks, I'm going to go to the heart of where everything is happening.

And yeah, I decided I wanted to work at a startup.

And actually, the interesting thing is, out of college, it's kind of hard to get a job at a startup, because big companies fly you up for interviews, and they're willing to take big bets on you.

And smaller companies don't have the resources to kind of waste their one precious headcount and resources on flying out interviewing you.

So I yeah, I moved out here to make the process much, much quicker of being able to apply and talk to companies.

And I ended up so I wanted to, I wanted to do software engineering, because actually, what I ended up working on at Microsoft was the SDK for Windows Phone throwback.

So I quickly realized that to work on a developer product, I needed a lot more credibility as a software engineer.

And the path I think of software engineer to PM is more common than the other way around.

So I was like, okay, I'm going to go actually write software for a living.

So I joined this company called Sixth Sense as a software engineer and started out my career there.

Cool.

And then so you spent a couple years there. And then what what prompted you to jump ship and go to Cloudflare?

So I guess the more interesting side of that question is what prompted the interest in Cloudflare?

I think what prompted the interest in Cloudflare was actually what prompted the interest in changing things up.

And that was that I very quickly realized that as much as I enjoyed like the problem solving side of software, I really needed talking to people to be a part of my day job.

So I talked to a few people at the company and they were like, you know, you'd be a really great solutions engineer, sales engineer, which I think was not a career that I'd ever heard of before.

I think it's one of the like, like, in college, you still kind of know, like, maybe a few more roles than firemen and doctor.

But there are so many roles that you just never hear of. So I was like, Oh, what's that?

So solutions engineer is a person that helps connect the commercial side and the sales team with a customer on, on the technical common ground, right?

So you're basically their technical advisor, as I like to explain it, where you're looking out for their best interest, you're like, you get some commission, but for the most part, you're not incentivized that way.

So you're actually trying to help solve their real problems.

If your product is not the thing that they need to solve their problems.

And you can be really honest with them about that.

And so what prompted my interest in Cloudflare was that I, I already knew about Cloudflare from reading the blog, and taking InfoSec in college, and learning about DDoS.

So yeah, I got to college class. It was awesome. Um, yeah, I think we I don't think we learned how to launch one.

Um, but there were a few assignments of like breaking stuff with causing stock overflows.

The point you made about, you know, going to college and studying CS, you know, one of the best CS schools in the country or the world and but not knowing about these different roles at a software company is always interesting to me, right?

Because like, CS degree, hopefully, this is getting less true over time.

But you know, it's still seen as this like requirement to get into the business, but they don't actually teach you anything about how software is actually built.

Right? I think I use like Git once in a CS class.

Right? Right. Like the world according to a CS major is you write Java outlets, right?

You build a fake photo album. And then you sort it six different ways, which is useful.

But yeah, and to Georgia Tech's credit, like there are a few classes that are designed around group projects.

So you get a little bit more of a feel of like, most code isn't developed solo in a silo.

Funny anecdote about that, too.

I don't know how true this is. But apparently, they were going to introduce a software engineering major into the program.

But the College of Computing at Georgia Tech is actually separate from the College of Engineering.

And the College of Engineering kind of like owns the word engineering, so they didn't want us to pollute their major.

Academia, just focus on the right things always, right?

So, okay, so it sounds like you left Sixth Sense, you joined Cloudflare, as a solutions engineer, it sounds like this was early 2016.

So it sounds like that job is like just consultative, sort of figuring out which products best fit a given customer's needs, and then figuring out how to sort of like, maybe fit square pegs and round holes where necessary.

Talk to me about what what the day to day was like, and what, what was what was interesting about that job?

Yeah, so I think the first interesting thing, and the thing that probably terrified me the most when I joined is the whole role is based around, this is how you present your solution.

And this is how the company interfaces with enterprise customers, right?

self serve customers, or free customers, kind of go through the flow on their own, and you're left to talk to these bigger companies.

And I didn't have any experience talking to these bigger companies.

So um, yeah, I remember just sitting in on calls in the beginning, and just shadowing and absorbing, like, okay, this is the appropriate jargon to use when I'm talking to the CTO of a company or, like, their CSO.

And actually, so one of the ways so because I came from a software engineering background, I was like, Okay, what's the quickest way for me to start providing value ASAP, because I was also actually the first junior hire of the team.

And at first, there was some, so there are a lot of things that you have to do pretty manually, especially in the beginning when you're starting out.

So I would write some Python script to automate, like a bunch of API calls, things like that.

Yeah, sitting in on customer calls, running customer calls.

And one of the really big other things at the time was helping out with this thing called edge side code.

Um, edge side code, I assume it's code. And it runs on the edge.

You could consider it, I guess the earliest, like the most basic MVP of what's today known as workers, which we'll get to.

Um, but yeah, so I think, like, early in the stages of a company, like you start to onboard customers, and then you try to land a really big fish.

And because they're a big fish, they have their own set of kind of more specific requirements, right?

So you get into this pattern of snowflaking, right, where every time you're like, Okay, we'll do this one thing special for this one customer.

And so what you end up doing is basically hard, what I code was, was hard coding, a bunch of logic into our edge, that when a request came in for a particular customer, if they needed something done based on a header, or based on a country, something that didn't come out of the box with a page rule, then we would write it in and actually not short before I arrived, our engineers that worked on the actual edge features themselves.

So the same engineer that would, like write the feature for a cash DTL would set aside time in their week to like, okay, we're going to get a few requests from customers.

So I'm going to have to go and write those myself. And eventually those were passed over to us.

So as I code is basically the framework we use to handle like exceptions from large customers who wanted the behave the way our CDN or other edge products, they wanted the way those products behave to be slightly different from what we offered.

And this is our way of sort of shaving down rough edges, or not rough edges, but just like making sure that our product fit, given customers deployment, right?

And it sounds like it was heavily customized, but there was at least some process and some framework around writing this stuff and deploying it and all that, right?

Yeah. And ideally, even at the time, we would keep track of what were the top use cases that we ended up writing workers for.

So I actually think one of our first meetings together, Rustam, was me coming to you with a list of like, these are the top things that I need to write edge side code for now.

Can you build those features for me? I don't remember this meeting, but I can imagine what I said.

And we'll save that for I actually think you said something to the extent of this is the most reasonable request.

Okay, well, I'm glad.

Okay, so and then that makes sense. And why weren't we letting customers write this edge side code themselves?

Like, why go through the effort of having our engineers do it?

Or, you know, sales engineers, why not expose this directly to customers?

So first of all, it was written in Lua. So that in and of itself, it's a great language for when you're running an edge.

And it was a really good choice of language for our purposes as a CDN and network provider.

But that's not the most common language in the world.

So first of all, there was, I don't think that even as a product, it would have been the best experience for our customers.

But furthermore, letting customers run arbitrary code is really, really scary, right?

The whole purpose of writing this edge side code was to onboard these really large customers.

And now if you can imagine that I, as another entity can write some code that can intercept another customer's requests, that becomes really, really scary.

Got it, right. So the safeguards are just not in place. And so arbitrary code execution as a service is dangerous.

Right. Be careful. It was sort of the gist of it.

Got it. I don't know if you remember CloudBleed, but if we didn't want that every day.

Yeah, or ever. I do remember CloudBleed. Maybe that's a different Cloudflare TV episode.

Yeah, write it down. Yeah. Okay, cool. So it sounds like you enjoyed being a sales engineer, a solutions engineer, and ESC was part of that.

This is not a spoiler to say, clearly you're a PM today. So there was some transition at some point.

What prompted that change from being an SE to a PM and how did you find that change?

So in parallel to me helping close deals and working with customers and writing the code, there's this other revolution kind of happening in parallel at Cloudflare where we were starting.

So someone was also thinking about this problem of, okay, this is not a scalable approach.

We can't deploy code once a week for every customer that needs this modification.

It was also just a really terrible process.

And if we made any mistakes, there would be a rollback.

Customers had no insight into this. Clearly that wasn't the right solution.

So people smarter than me started thinking about, all right, well, what then is the right solution?

And that was at the same time as we hired Kenton and he and JDC and Mayor Wankowski all went, I think, to London along with Dane and I think a few others and started thinking about, okay, how do we allow people to do that?

And so they came out with what's known today as workers. And honestly, when I first saw and started to interact with workers, actually kind of to tie things together, this yes student in me was like, wow.

The idea that I could write code and not have to go through all the pains that I had to go through previously to set up a server and it just worked was so honestly mind-blowing for me.

I was like, I really want to be a part of this. And I'd also kind of seen the customer pains very directly.

So I had a very close feeling of, I know exactly the problem this solves.

And it was helping a lot of customers onboard onto workers when the beta that came out.

So I was like, okay, I think I know what needs to happen next for this to become even better.

Got it. And then just from a sort of pragmatic or tactical perspective, what did that job change actually look like?

Was it like, I want to be the PM for workers?

Like, great, you're hired. Or what did that actually look like?

I think it looked something like that. Yeah. I mean, as I mentioned, I was already working with the biggest workers, customers, because for the SV role, generally, at least previously, the main competencies that we look for are around networking and security and CDN.

And I was basically the only person that came from a software engineering background.

So it made sense that when this product came out that helped people write code, I was on the front lines helping people get started with that.

And so, yeah, there were also structural changes going on.

And I basically reached out to Jen and I was like, I really want to work on this.

And yeah, just kind of worked out. I had to hand over my accounts.

But I was, I think SC to PM is actually a pretty common transition.

So I was giving this advice to someone recently and I was like, yeah, I just got to rip the bandaid off.

Why do you think that's a common transition? Like, why is that a well worn path?

I think you spend so much of your time helping customers solve their problems with the products.

Probably there's a hubris there that's like, you're working with the tools that are given to you and you're like, I can build these better.

I was talking to Tom Brightville, someone else who's made a similar transition.

And I was like, what, what are the like, how does it feel being on the other side?

And he was like, you know, that hubris was a little misplaced.

Yeah, all the things that seemed so simple from the other side that are like, oh, well, we can I'll just do this when I'm in charge.

And then you're like, that's a very complex problem.

I will take a seat now. So you've been at the PM thing for how long now?

For two years. So I guess the transition happened roughly exactly at my halfway point.

Wow. It feels like it doesn't feel like that long ago, I guess.

It feels like both that long ago and not that long ago.

But time doesn't feel like anything right now. So it sounds like you assumed product management responsibilities for workers after it launched, but like, very early in its lifecycle.

So it seems like a good time to transition to talking about workers themselves.

What is a worker for those the audience that don't necessarily know?

Well, the definition of that, I think, has also changed throughout the years.

So when I started on workers, workers was a way for you to run some JavaScript on the edge, basically, presumably between your, the eyeball request, and then the response of the origin hands out, right.

So you could modify headers, you could possibly modify the HTML, do other types of modification.

But we really quickly realized, and I think that's one of the really cool things about being a PM is like, you give customers your product, and they use it in ways that you never expected.

But what we quickly realized is that it actually wasn't just the small piece.

So workers now is our platform, our serverless platform that allows customers to write code in JavaScript, or now we support WebAssembly, so all these other languages too, and allows them to run now full applications directly on the Cloudflare edge that is also continuously growing.

Got it. So the story or the sort of inspiration for this being all this gnarly Lua edge side code that Cloudflare employees used to write on behalf of our large enterprise customers, and then that evolving into workers makes sense and sounds like one concrete use case.

Who are some other customers that that are types of customers that that use workers and what are they using, use the product for?

Yeah, so I think a really good example of a customer that really helps crystallize what workers, the essence of workers, what workers is, and how it is optimized.

And so one of the things that I think people have a difficult time wrapping their mind around is the edge and what does it mean to run code on the edge.

And if you think about it, generally how A-B testing works is kind of one of two ways.

So you can do A-B testing on the client, and that means that your browser receives some HTML, and then it'll run some JavaScript, and it'll change the HTML around.

And so what you as a user experience is either like you see blue, and then it changes to red, or we go, okay, let's pause the whole thing for a second until we can pre-render it and then show it to you all at once.

But then you still see this like flicker of white, which everyone really cares about performance now, it's been proven that it's a big determinant of conversion, things like that, so not ideal.

The other way to run A-B tests is the server receives a request from the client, and then it decides which experiment to run on you, and then it'll return the HTML, but that means that...

It'll return like pre -modified HTML, right? Right, right. Which means that you can't really take advantage of things like caching, it means that for every single request, you have to go back to the origin, which if we haven't figured out how to be the speed of light yet, so that means that if you're in North America, and your customer is in Australia, you're traveling, you're making that round trip every single time.

So just to summarize, if you're trying to do A-B testing, and you're not using something like workers, you're either going to get weird render side problems in the client, or things are just going to be really slow, right?

Exactly. Yeah, got it. Both sound bad. So how does Optimizely fix that bad trade-off with workers?

So the whole thing, what if I told you, you could run code so close to the client, that it felt like it was on the client, but you had all the control over it that you do on the origin, is kind of the problem that workers solve.

So a request can be answered from a Cloudflare POP. So if you and I are in San Francisco, we'll probably get routed to San Jose, and then San Jose can receive that request and go, okay, like Rita is in bucket A, and then it can take that, you can find the experiment and then serve it to me directly as I should see it.

So it gives developers the benefit of like they have full control, they can roll out changes whenever they want.

And they, yeah, you don't get like the weird flipper problem.

Got it. So that's an interesting use case. And obviously, well, it's like conceptually related to but also different from like, here's a HTML request or HTTP request, like, change minor things about it, and then send it onward.

Do folks, do we have examples of folks building like actual soup to nuts applications on workers and not using an origin at all?

Yeah, we're beginning to see some of that.

I can tell you, so anyone watching Cloudflare TV right now, Cloudflare TV, the website is like fully built on workers.

And if you go to workers .Cloudflare.com slash built with, we've kind of built out a gallery of different applications that are fully using workers.

So there's like a website that will, will tell you what movies are playing nearby.

I guess that's less relevant now, because we're not really.

But yeah, it's, it's been really crazy seeing what kinds of applications people build fully on workers.

And I think it's also not that binary. I think one of the really interesting things about workers is seeing that transition and customers.

So some people Yeah, we'll go straight from running code and an origin somewhere like not having an application at all to, I want to build something that's really performant and infinitely scalable, and I'm going to choose workers for it.

But realistically, most, most companies already have existing applications that are running somewhere, right.

And so workers gives them the opportunity to break it off chunk by chunk.

And by having workers as this proxy use case, it's like, okay, on day one, I might move my authentication to workers.

That makes perfect sense, right?

Because, again, like you shouldn't have to travel back and forth to your origin and back just so someone can tell you, you have access or you don't have access.

And then the next thing, you know, you go, okay, I have this new capability or this new API I need to build.

So this one API can go directly into workers.

And, yeah, next thing, you know, yeah, that's a pretty common pattern, right?

Like folks using workers as sort of pseudo API gateway, and then slowly, like, fewer and fewer routes go straight to the origin and are just served within the worker.

And then all of a sudden, you have a totally serverless deployment.

Exactly. So that's a cool, cool, cool journey to watch. And it's not theoretical, right?

I can think of a bunch of customers that have sort of walked that path.

Yeah, ourselves included. Yeah, 100%. Yeah, right. The dog aspect is interesting, too.

Do you want to talk about that quickly? And then we can, we can?

Yeah, sure. I think the best and brightest example of this is probably access.

So access is our VPN replacement product, to put it in like very broad terms, where basically, you and I don't really have a VPN installed anymore, right?

When I need to access my internal tools, it'll go through Cloudflare. And then it'll check whether or not I have the token.

And then it'll let me continue on to, to my destination.

And the interesting thing about access is, it was actually originally, like the authentication was implemented on workers, but many parts of it were originally running out of some of our core data centers.

And I think it's been a really big dog pruning success story.

And that's especially evident now. A couple months ago, we announced we were giving, we were giving away our teams and access product for free for a few months to through September, I think, to help out to help companies out as they're transitioning into everyone being remote, right.

And so as you can imagine, we're onboarding all these customers and the traffic access started seeing, I don't know exact numbers, but has gone up significantly without any effort from our side in terms of scaling it.

And it allows the performance to be so much better than a VPN that gets rerouted centrally.

So that's a really good story.

Yeah, no, it makes a lot of sense. So it sounds like the products evolved quite a bit from from original conception and original ideas around use case and users to what it is today.

But at the same time, like a lot of the original stuff is still there.

What were some of the early challenges and moments that will sort of drove that evolution and changes in the way we thought about how we were building this thing?

I think we've evolved a lot with the product.

And it's made us challenge a lot of the assumptions that we're able to make early on about how Cloudflare worked.

And as we mentioned earlier, it was introduced as this way to to modify the CNN originally, right?

So Cloudflare's assumption.

Yeah, just to give you an example, the very first project that that I worked on as the PM of workers that was very close to me, because I had just dealt with these issues on the customer side, customers would enable workers or start using workers for something small, where they were modifying a header on the way to the origin.

And all of a sudden, the next day, they went into their analytics. And they went, holy cow, my traffic has bubbled.

And they would email their customer success manager and go, are you going to charge me for all of this double traffic?

What is going on? And what basically happened was that a worker when it receives a request, when it makes a request to the origin, it actually initiates it as a new request.

And this is really useful in the current context, because you're not limited to a one to one model of request to the edge and then back.

You basically use the worker as like a mux or like a way to fan out, right? Exactly.

I can make a whole application just by calling a bunch of APIs together, right?

And then stitching them together into doing something that I wanted to do. But that violated sort of like fundamental, not even not even explicit decisions we've made, but like just like mental models, people would forget about how Cloudflare worked, right?

Yeah, exactly. I mean, even like, what does the cash ratio mean?

When you're modifying the response, and it took a lot of thinking around that, and we realized that, okay, like, cash ratios actually have nothing to do with the request that happens from the client to the edge, what you really care about when you're talking about cash ratios, is how often does the request end up hitting my origin server, because that's when I care about caching it, right?

I want to alleviate that load. So yeah, everyone's analytic doubled, and it took a bit of work to split it out into, okay, there's eyeball requests, and that's one thing.

And then there's origin requests from the worker, and giving everyone the same visibility that they were used to as well, right?

Because if we just narrowed it down to eyeball request, then all of a sudden, there's a lot of meaningful data.

So there's a kind of interesting thing going on here, right? Where like, a lot of these challenges, it sounds like were actually created by internal Cloudflare sort of views of how our own system worked.

How did we, it sounds like that's one set of problems, and the other set of problems is like customers were just confused, right?

And they might not have had the same mental models, but whatever mental models they had were also broken.

How did you figure out which of these to fix first, and like how to actually correct the model in users' brains?

Is that just through fixing the analytics? Like, yeah, how do you tease those two things apart?

I think it's been, so we started out with the things that were the most on fire, right?

Like analytics was very visibly broken. And when we sat down, it was, and thought about, okay, what is actually happening here?

How to decouple the problem actually made quite a bit of sense.

But I will say we've actually also evolved.

So the first iteration of workers analytics was basically, the concept was around splitting this apart.

Now, a few weeks ago, or maybe a month ago now, we launched workers analytics v2, where the product has evolved even further, where customers don't even have maybe zones that tie to a worker, right?

So originally, the assumption was I have a website, and then I need to modify stuff on the way to and from.

But now that I'm building my website on workers, I actually kind of want those analytics independently, and I want them split on the basis of what I consider to be a unit of an application.

So I honestly think talking to customers, probably is the best way to get clarity around that kind of stuff.

Like they'll just ask you, oh, can I split it this way?

And you go, oh, been splitting it wrong. Um, but I mean, do you like, so in that example, when you ask a customer, like, Oh, how would you like to see analytics?

Like, when do you listen to them? And when do you say like, Oh, like, I actually think I have a better idea?

Like, how do you walk that one?

Well, I don't think you ask them, how do you want to see analytics?

I think analytics is actually really interesting to work on, because you kind of listen to the questions that they're asking you.

And then you're asking yourself, how can I answer those questions without being physically there?

So when they're asking me, okay, but how much traffic does this specific worker get?

And I can't answer that.

And I go, okay, there's, we should have a graph or chart, or like a number dial probably that says that somewhere.

Yeah, that makes a lot of sense.

Um, what so so so I mean, it sounds like analytics being broken was like an obvious, like, Oh, like, this needs to work.

And these numbers need to make sense.

Um, what were some other like, sort of early lightbulb moments or, or, or, or epiphanies you guys had as a as a product and engineering team that sort of influenced product direction?

I think so one of the other early challenges with workers, which again, it makes a lot of sense in the context of, if you're an existing Cloudflare customer, you can can be you could conveniently write some code directly from the dashboard and integrated with the zone that you already had on Cloudflare.

And then we started seeing some more interesting use cases being built, and people playing around with being building their own little serverless applications and doing things like Slack bots.

And we started getting really excited about that and actually went to speak at a conference about that.

And I was giving a workshop afterwards, I was like, look at this awesome thing.

It's completely serverless, and it runs on the edge.

And I'm going to demonstrate it for you guys today. And I'm going to teach you all how to deploy a Slack bot using workers.

And I started the workshop, and everyone's following along, and they write their code.

And then they went to deploy it.

And there was this really weird moment of, oh, I'm going to need you all to go purchase a domain now.

So everyone's like a little grumpy because they have to get their credit card out and also like think about what they're going to name this thing that was a Slack bot in the first place.

And then we lost a few people in the workshop.

And then even the ones that purchased the domain, they actually couldn't complete the demo by the end of the workshop, because we were still waiting on Cloudflare to pick up the name server changes.

Because Cloudflare had always assumed that the way that you work with it, right, is you onboard a zone and you change your name servers and your traffic starts flowing through.

And now we want people to use it for so much more than that.

So that was the moment that the Slack bot went off in my head.

And I was like, oh, we need, like, if we really want people to just go build stuff on this, we need to make those initial steps, like, way, way easier.

So that was kind of the genesis of what became workers.dev, which is a subdomain that we provide for you that you don't have, you don't have to do anything other than pick one, and you can instantly write a function and run it.

So I can now successfully run Slack bot workshops.

The thing that strikes me about this story is, it's kind of just a bigger version of the analytics story, right?

Like, and speaks to an interesting tension, right?

So we were able to get workers to market quickly, because there was all this Cloudflare infrastructure around it, whether that was technical, or semantic, or whatever it was.

But then, the same thing that helped us get to market quickly also impeded progress, because the model that we had constructed from a data perspective, and from the information hierarchy perspective, just didn't match exactly what workers needed.

How do you go about prioritizing, fixing these things, right?

So like, the point about Cloudflare being built around the idea of domains, that's sort of like deeply baked into Cloudflare from day one.

And like, there's all sorts of interesting little product quirks that I can think of, and I'm sure our loyal users can think of that stem from that.

On the one hand, that's let us move really quickly.

On the other hand, like, it's clearly kind of suboptimal at this point.

How as a product manager, do you go about prioritizing fixing that sort of, it's not technical debt, right?

It's like, product debt. Conceptual debt, yeah. Well, I think to your point, like, at the same time, already having all this stuff built in, and having existing customers, did allow workers to get a lot of adoption early on from people that were familiar with Cloudflare.

And so, the way that I think about it is kind of from putting yourself in the customer's shoes and thinking about who are all the different customer personas that I can service.

And the Cloudflare persona that already has onboarded onto Cloudflare, turns out I can service them like, decently well, right?

They've already configured most of what they need.

Which I think one of the really interesting things about workers, and I think one of the really interesting things about Cloudflare is that it plays in so many different markets.

And so, we make, we have some assumptions that we made early on based on like, the primary markets that we play in, which are like, around security, and DDoS protection, and CBN, and all that stuff, and making it really easy to onboard onto those in one fell swoop.

But when you step back and think about like, okay, I don't already have a website on Cloudflare, I'm just a developer, and I just need to get a task done, right?

Maybe it's a Saturday, and I'm bored, and I want to put together a website, or I'm thinking about writing Slack bot, or maybe I'm an engineer on a software team, and I'm just trying to find like, the easiest way to do something.

And you step back and you go, okay, where are the biggest friction points?

And it can, in this case, the friction point was in the beginning.

In the analytics case, the friction point was kind of in the beginning for the customers who were already using us, right, where like, we started using the product, and quickly they ran into a problem.

But yeah, I think a way to think about prioritizing these types of issues is by where, like, where is there the most pain today?

Would you say, would you say we, that that's the, that's the forcing function is like, rank by user pain and solve number one first?

I think the hardest thing is seeing user pain.

Like what user pain for the workers .dev example looks like is that we just didn't see those customers.

So there's user pain that's really easy to feel because customers are writing support, and maybe you have a lot of SEs or salespeople pinging you about a problem.

But yeah, then there's user pain that you don't see until like, I ran this workshop right now.

And I think the other point here is like, we talked about dogfooding a little bit and like, Cloudflare using our own products is important.

And it's something we do very regularly.

Something I know I have to force myself to do occasionally. But like, I every time I do is like, Oh my god, why didn't I do this sooner is just using our own products, right?

Like, I'm so focused on I've got tunnel vision on products that I have sort of day to day responsibility over.

And then sometimes you have a side of how all the other pieces fit together.

What are some, what are some things you found just like using the product?

I know I have a bunch of examples like, that's how that works.

Like, we got to fix that immediately. I don't know.

Why don't you provide your example? Well, the biggest one that comes to mind is like, I logged into the dashboard, I was trying to do something.

And I knew we'd recently shipped a bunch of functionality.

I just couldn't find it in the dashboard.

I was like, how do I get to I forget what it was. It was actually something that that interacted that customers would interact with an account level, right?

And to the point, we're just talking about might have, yeah, it might have actually been workers.

And to the point we're talking about a lot of coffee is built around this model that involves domains.

And so I was looking for this account level thing.

And I was like, oh, that's cool. There's an ad on the side talking about the feature I want to use, but I can't find the button to get into the feature.

And turn out the ad was the button with a thing I thought was an ad. And I was just like, I walked into I remember walking into our next product meeting and being like, has anyone else had trouble finding that button?

Because like, I spent half an hour looking for it.

And we all just went. Yeah, I know. It's like this eruption of like, oh my god, me too.

Yeah. I will say, as an SC, I've seen, I think I've probably used our product pretty thoroughly.

I think one of the challenges as a PM, and I'm feeling that more and more is that you get siloed into your product, and you have less incentive to check out other products.

But I don't know, as far as workers go, I definitely play around with it a lot.

Especially on weekends, or if I'm building something, I feel guilty, not building it on workers.

Actually, one of the things that we're, we have an alpha out for and we're currently working on that I ran into.

So Wrangler is the workers CLI. And so in the user in the developers development journey, right, like the way that most people end up shipping code is like, you end up settling on the technology, and then you figure out like the initial way for you to write a few lines of code and see if they work.

And we provide that experience pretty well through the dashboard, where if you click on workers, you can instantly see this like editor and you can start playing around with it and has a debugger where you can quickly get feedback.

But the whole purpose of CLI, right, is to get you out of the editor, because for most developers, if you're working on something serious, that's probably not what you're using most of the time.

Okay, in a browser, but yeah.

Okay, PMs are different. Do you write in Google Docs? No, I wish Jupyter.

I wish you anyway, keep going. Well, we might end up talking about the same thing.

But yeah, I was like, Okay, I'm gonna use Wrangler to publish my thing and using VS code to develop it.

And then I was like, Okay, I have to keep opening up my browser just to get the console logs like this does not feel good at all.

So yeah, and this kind of made me understand like, oh, when users are asking for like, I really desperately need a local development environment, like, that's the pain that they're experiencing.

And so luckily, we have this really talented engineer on our team, Avery, and started out as a project called Wrangler curl, where I was like, Oh, I'm building an API, I'd like to be able to like curl it locally, right, and like emulate the response for me.

But he ended up building it into something just so much more phenomenal, which is now known as Wrangler dev, where basically, you can get the full feel of local development environment.

But actually, it connects to our edge. So that way, you get like a high fidelity experience as well.

Hopefully the best of both worlds. Yeah, took a page out of our own playbook.

If you can't put on the client, you don't want to put on the server.

Yeah, makes a lot of sense. So we talked a lot about the product. What about the team behind it?

How? What? How are sort of engineering functions put up? What does a product manager work on versus engineer?

Like, shed some light on that? So I think kind of the same way that the product has grown and evolved, our team has grown and evolved.

So the answer to that question six months ago was very different from the answer to that question.

Now. But I mean, at Cloudflare, I think PMs and engineers work really closely.

But I would even say the line is so much more blurred, because it is a developer product, that as a PM, you're like, you're actually my customer, right?

So tell like, you value your engineers opinions, really, really deal.

You're like, tell me what you want, or what would you do?

Um, and so, yeah, the way that we're organized internally within the workers team, we kind of have these three teams.

So the way that I used to describe the roadmap for workers effectively was like, allow, like build more tools for to make it easier for people to develop with workers, right?

Things like Rango dev, or make it easier to deploy stuff logs, or allow worker, allow people to build more stuff with workers.

And so we kind of ended up splitting up across those lines to have slightly more focused teams, because there was a moment where the workers team was getting a bit unruly in terms of how large it was growing, which is a really good problem to have.

But yeah, this has allowed us to focus a bit more.

So we have the developer productivity team, which is the team that I lead that focuses on how do we make every step of the way as easy as possible for you.

And then we have the new markets team that expands the number of use cases that you can build with workers.

And then there's a third team, that's the distributed data team, which kind of if you go back in history, in the history of workers, one of the things that we realized really early on in terms of use cases, like almost as soon as we launched was like, there's not a ton of stuff you can build without in state, right?

So KB was the first thing that was built to expand use cases.

And so a way to think about it conceptually is like, the distributed data team is has like spawned out of new markets.

And now we have a team that full time thinks about like, okay, what are all the different storage primitives that we need to provide in order for someone to be able to build the next, like, I don't know, Uber, Airbnb, whatever on the edge.

You mentioned, you mentioned something in passing there, you said that the team was getting large and unruly.

What does that mean? Like, what isn't like, I think, I think, you know, this isn't necessarily, or this certainly doesn't have a conflict.

But I think in general, sometimes people conflate, oh, my team got really big with the product is successful, or vice versa, right?

There's this weird, perverse relationship between the two.

And you seem to suggest that smaller teams are better. How do you how do we think about that at Cloudflare?

Um, yeah, I mean, I would say most of Cloudflare is actually relatively small teams.

And it allows, I think, first of all, like, I felt relief when we got split up, because I as a PM was able to focus my attention a lot more.

But you're a lot more focused. But also, another really important factor is like number of people you need in a room to make a decision, right?

And so yeah, on one hand, like product become successful, you need to build more features.

And like, as you feel yourself falling behind, you're like, okay, we'll do better with another engineer, okay, we need if we only had like one more person, then we'd also be able to tackle this problem.

But then pretty quickly, you start to kind of lose focus a bit yourself.

And yeah, I think like independent teams can make that can make decisions independently can also, as a result, move much more quickly.

Right, the communication overhead of adding the seventh or 10th or whatever engineer outweighs any increase in productivity.

Yeah, I mean, there's a there's a balance here, for sure, as well.

And just because you kind of put a line in the ground doesn't fix, like all the problems overnight.

I know this is something that Spotify has written about quite about quite a bit.

But basically, now that we have this line around missions, we're still working on a few shared code bases, right?

Like at its core, workers is made up of there's the dashboard, which is the UI that you see, there is Wrangler, which is the CLI, there is kind of the platform itself, like the back end and the API's that you interact with when you push code.

And then there's the runtime itself, which executes the JavaScript and different projects.

Some of them need something done across every single one of these, some of them need only a small portion, when we get escalations, like there's no such thing as a new market escalation, or a developer productivity escalation.

So it introduces an interesting new set of problems of like, how do you now load balance these things?

What is your day to day look like these days, now that the product's more mature?

Hopefully, there's fewer fire drills.

How do you spend your time? There's, I've been trying to find this quote, in Crime and Punishment.

But it's like very early on in the book, and it makes fun of the main character who's like, I'm a philosopher, I think.

And like, that's been the really nice thing is like, now I get to have time to think.

By which I mean, like a few hours in a week. How do I spend the rest of my time?

We're prototyping and working really closely with engineers is one way.

So one of the things that we're working on has been like, very hands on, like, first of all, just in general, we found prototypes to be a really great way to iterate on the product.

And so we're going through the sprint right now, where it's pretty all hands on deck.

And so I get to hang out with my team quite a bit.

Yeah, answering escalations, talking to, talking to and thinking about go to market and product marketing is a really, really big component, especially for a product that overall is still very on in its life cycle.

Product mark, like the marketing is a really big piece of it. And especially, I think for a developer product, that's another line that's really blurred.

Because you don't market to developers, I think the traditional way of like, look at, like, these great things that we can promise.

It's like a tutorial might be a really great asset for marketing, because someone was actually able to get their hands dirty and like, see the value for themselves.

So yeah, marketing is how I spend a bit of my time.

I don't know, let me take a look at my calendar. It's a whole episode unto itself, right?

We talked a lot about how we build a product for a different set of use cases, but like how we go to market, how we market it, how we sell it is, there's a lot of interesting stories there.

I know. Also, quite a bit of it is I think, so with the analytics example, actually the very, very, very initial iteration of that, just to prevent us from overbilling customers is like me generating these analytics manually for customers.

And so I think you do spend a good chunk of your time kind of on an ongoing basis, just plugging the gaps of your own product and getting your hands dirty.

So right now, yeah, spending a bit of time with our documentation and polishing that up.

Yeah. We got four minutes left.

What are you excited about? What's coming up for workers? What am I excited about?

The tricky question, like, I don't want to spoil anything. I thought you were gonna say you're not excited about anything, in which case we have other problems, but no, we would have, you're too excited as a problem is what you're saying.

Too excited about too many things. I don't know. What am I excited about?

I'm excited about, so one thing I've been thinking about a lot is like the software development journey.

I mentioned it earlier in the context of like a local development environment.

But yeah, we've been spending a bunch of time thinking about like, how do people deploy code?

And I don't know the, I think the best way to illustrate it is kind of like, you choose your technology, you do some local development and figure out like, okay, this line of code does like, what I expected to do, you write some tests, to make sure that it does what you expected to do every single time from now until forever, right?

Then you kind of put something into QA, where you send it to your PM or like other designers and make sure it works how they expect it to work, kind of end to end.

Then you push the big prod, like send to prod button, and then you observe and see what happens and go and fix things.

And so one of the things that I've been really excited about is like, we've been picking off like small bits of that chain and improving all of them.

So like this past quarter, we have Wrangler dev and alpha and we're shipping like really major improvements to it and like, probably going to promote to GA sometime soon, which addresses local environment, we shipped Wrangler tail, which allows you to see your production logs, ship analytics, which also kind of helps demystify the later parts of it.

So I think kind of like connecting this full chain, it's kind of like this self progress bar, that thinking about all of it turning green is what's been really exciting me.

Well, so but do you have an end in sight?

Like it was the progress bar going to fill and it's all green. And then we go home or like, how do we what?

It sounds like we have you have some line of sight to what's coming, but like, how far does that bar stretch?

And how are you adding to that?

I think you can take it in all sorts of interesting directions. So what I think is, like, interesting that's coming out of Microsoft now is like GitHub code spaces, right is looking at their chain of like, they now own the source control and they introduce code spaces.

And so now you spend the time that you're like writing the actual code there too.

So I think there's always like interesting directions to grow in.

And the other thing too, is we have the new markets team and the distributed data teams that are also building stuff out that I think the workflow has had a lot of successes, like you ship an MVP and you get some feedback on it, but it's never going to be polished.

And I think in the serverless space, it is really, really important to get things really polished, because otherwise you're not providing that much more value than having to set things up yourself.

So yeah, I expect them to introduce kind of new points into the journey that will continue to iterate on.

I don't think the bar is ever fully complete.

I for one, I'm super excited to see what comes next. Thank you so much for joining Rita.

And hopefully, we'll have you on again soon. That sounds good. Thanks so much for having me.

Thank you all to everyone that watched.