What's New with Product (Intern Edition)

Hi, I'm Jen Taylor, Chief Product Officer at Cloudflare and I'm here with my partner in crime, Usman, for latest from product and engineering.

Hi, Usman. Hi, Jen. How are you?

I'm Usman Muzaffar, Cloudflare's Head of Engineering. It's good to see you. We were just saying we haven't done one of these in a while together.

We've gotten very...

I missed you. I missed you. I missed everyone. I missed this. It's like my favorite thing.

Yeah, I had to dig up my lava lamp background to get it ready for Cloudflare I'm really thrilled that we have three people who we've not had on this on this segment before and who actually are interns on the product management team.

And I thought we'd let them each say hi and then we'll get a chance to talk to each of them.

So Alex, Justin, Amrita, in that order, I guess, say hi.

Hello. Yeah, I'm a PM on the research team. Awesome. And I'm Justin. I'm working on PageShield.

My name is Amrita and I'm a PM intern on the Magic Transit team this summer.

Excellent. OK, so we're going to start with Alex. Yeah. Yeah.

Yeah. Let's start with Alex. Alex, what is it when you say you're an intern on the research team?

What does that mean? What are you doing? Yeah, so the project I'm working on is translating some of the research the team did to detect multi-user IP addresses and turning that into a production service that can be integrated with some of Cloudflare's product teams.

So, for example, cloud management the firewall team.

Got it. What does that mean to have a multi-user IP address system? What does that mean?

Yeah, so there can be different types of multi-user IP addresses.

So one of them would be carrier grade network address translators. So these might be things that an ISP uses for their cellular tower or for like an entire apartment building, for example.

What makes this a tricky problem for us and other security companies is being able to identify the difference between when a bunch of people in an apartment building search up a site versus a malicious bot sending many requests in a short period of time.

And so we're hoping that by integrating this capability into Cloudflare's products, we'll be able to better distinguish those different cases and unblock legitimate traffic.

That is good.

It's a really, it's an interesting problem, right? Because like, so I remember when NAT, which is, you know, network address translation, we're all sharing the same IP address.

Same thing happens in our house actually, right? And so each of us have one external IP address and every device inside your house or your business has a private address, which means that from the outside world, it looks like one number, right?

It's not that different from the telephone exchange, honestly, where you have one gateway number.

But if you're in the business of studying and classifying traffic, now, how are you supposed to know, wait, is this legitimately just hundreds of people who are all doing different things?

Or is this one bad actor deliberately trying to master? And so like trying to figure out what side of that on matters a great deal.

So Alex, like, so how, what were some of the, what did this problem look like?

Tell us a little bit about like, what was interesting or hard about this?

What were the tools? You show up at Cloudflare, you're saying, this is a problem, problem makes sense.

You know, we can describe it in a couple seconds.

So what's the, you know, what are the tools on your workbench on day one?

How did you, what was it like being a product management, product manager intern to help think through how should this work?

Yeah, so definitely as a PM intern, I wanted to first kind of understand the technical aspects of kind of the existing work for research you did.

So scheduled a bunch of chats with members of the research team, research leads, and research engineers, and kind of looked at both like the public sources and the internal sources of data that were being leveraged.

I think going from there, the tools really were just, yeah, leveraging Cloudflare's like great teams and speaking with the PMs for a bunch of different teams to see like how this might be used and incorporated into their products.

So yeah, I guess biggest tool was probably Google Meet and just being able to hop on calls quickly and get that info.

How did you know who to talk to?

I guess I just followed the rabbit trail. I had a couple of recommendations to start for other PMs.

I thought management seemed to be kind of a low -hanging fruit to go after immediately, but then I would always ask like who else would you recommend I speak with?

And that led to a lot of interesting conversations. What surprised you most over the course of your internship?

I think, let's see, a couple things.

First, everyone was super receptive and everyone works really quickly and ships things really quickly as well.

So that was really surprising coming from previous internships.

Another thing was, I guess, how diverse each team was in terms of both team composition as well as how each team was run.

At my previous internship, each team was very focused on the methodology and Agile Scrum and kind of following those rituals.

And I think here is some teams that are a lot more flexible and other teams that are a lot more procedural.

And so I think being able to see so many different teams achieving success in different ways was really cool.

Given what you saw this summer at Cloudflare, what advice would you give to Cloudflare?

Advice to Cloudflare? What Jen's really asking is can you help us do our job better?

Yeah, I mean, I think the biggest advice would probably be from a student perspective, getting, I guess, the brand out to more people.

I think just even talking to my friends.

They've heard of Amazon, they've heard of other huge tech companies, but very, very few of them, even my CS major friends have heard of Cloudflare.

So I think being able to get the word out, publish what Cloudflare, broadcast what Cloudflare is doing to that student audience, that would be, I think, good for hiring and maintaining that type one.

That's awesome. Yeah, you know, the diversity in not just in composition, but in process, this is something that is quite intentional.

A line we try to use a lot is autonomy over alignment. It's more important that the teams have the agency and the freedom to do it.

Okay, we're going to go this way and implement this way.

If you're a CS nerd, I like to look at my job as defining interfaces, not implementations.

And so let the teams come up with the implementations that work best for them.

So one question I wanted to ask you, though, Alex, what's some of the most important things you feel like you've learned, not just observations, but how are you a better product manager, a better analyst, a better engineer?

How has this experience helped you? Yeah, so I think especially coming from previous software engineering internships, one thing that I really learned and improved on during the summer was like how to drive meetings and come into a meeting prepared with an agenda and questions and points to tackle.

Maintaining like an organized note-taking system to pull insights from later and just making sure things were kind of everything was addressed in the timing.

I think previously I didn't have as much experience driving meetings, more on kind of the sitting in and just contributing when needed kind of side of things.

So I think that was a good experience this summer as a PM intern.

Spoken like a true product manager.

And seriously, we're not plus one that higher. That's such a key skill and sort of an unsung skill too, right?

Like you take it for granted when it's done well and you're just kind of like, well, why did that meeting go sideways?

Or, you know, did we actually get anything out of that and then, and that's, and I guess last question, Alex, before we tap off you is what was the most fun?

Like what was the most enjoyable thing? I think last week was a lot of fun being able to visit the New York office currently based out of New Haven, but it was there like for the weekend and some weekdays.

It was really cool to meet another software engineering intern and another fellow PM intern in person and like go out and grab lunch and finally get that like face-to-face contact after entire summer and then some of the internships.

So yeah, that was probably the most fun experience.

That's a great answer. Yeah, totally. I'm glad you had a good productive internship and thanks for all the hard work.

I mean, those are interesting, tough problems that will make Cloudflare a better service by your efforts.

So thanks a lot. Amrita, you've been looking at something completely different this summer.

I have. So like I said earlier, I'm on the Magic Transit team and the project that I've been working on this summer is building an onboarding portal for our Magic Transit customers.

So really exciting. It's a great project that I'm very fortunate to have been able to work on.

Now remind folks again, what is Magic Transit?

So Magic Transit protects entire IP subnets from DDoS attacks, but also enables a full suite of network functions like advanced packet filtering, load balancing, traffic management tools.

So it uses Cloudflare's really large global network and takes in all the traffic through two very key networking protocols, BGP and GRE.

And then once the packets hit our network, we filter them, send the good traffic on toward the origin and ingest the bad traffic and keep it away from our customers.

Got it. So you're working on an onboarding portal.

What's the problem you were trying to solve with that? Right. So today the Magic Transit onboarding process is definitely quite time consuming and extensive and it takes about two weeks to complete as of right now.

So we start with a kickoff call with our customers and then going through a number of different stages like scope configuration, tunnel configuration, pre-flight checks and edge configuration stages and eventually end with a go live customer call.

So as you can imagine, that takes a lot of hands-on work throughout numerous teams within Cloudflare, but also a lot of time from our customers.

So Magic Transit is growing and we realized that building this portal is necessary to support that growth and make Magic Transit and this process much more scalable.

Yeah. This is one of our most exciting products that has come out in the last two years.

And I remember when the engineers were first explaining it to me, they had to draw like five different diagrams to get like how interesting and how we expose one range and the tunnel is asymmetrical and it goes back.

But one thing that was obvious to us right from the beginning is, wait, how do you turn this sucker on?

Like this is not just downloading an app to your phone.

This is not installing a piece of software. This is not even just changing a couple of DNS records.

This is actually changing your address on the Internet and IP addresses that big customers own and manage and are theirs to own.

They are effectively saying, actually Cloudflare can answer for this.

And that is exactly as complicated as it sounds, which is no surprise that it took a couple of weeks to turn on.

But when you also have a hit product, we've got to streamline that.

So this is the problem. Like, okay, like this thing takes two weeks.

We need it to go as fast as possible. How did you approach it? Like what were some of the inputs you thought through and what is the shape of the solution that came out of it?

Right. So similar to Alex, I started off my time here just meeting with as many people as I can across all sorts of different teams that touch Magic Transit, whether that be sales, support, Magic Transit engineers, product manager, implementation managers, just having so many different conversations all centered around the Magic Transit onboarding process.

And that really helped me a lot, just get a better understanding of what this process looks like and how it's evolved over time.

And that shaped into this onboarding portal that will live in the Cloudflare dashboard and centralize any information and actions relating to the onboarding process for our Magic Transit customers.

So I know one of the key things that'll come out of this is it'll give customers more visibility into the process, allow them to see where they're at and keep a better track of how the process is moving along, but also make the requirements of each step much clearer, which will hopefully reduce the need for back and forth communication between the Cloudflare team and customers and reduce the number of support tickets and incorrect setups.

All goodness. It's kind of one of the things we pride ourselves on as a company is taking complicated things and making them simple and easy for our customers to work with and enabling our customers to do more themselves and rely less on people.

So this has been a big hurdle for us, this challenge of Magic Transit onboarding.

So really excited to get the, really excited to get this live and going.

So, but I'm going to ask you the same question I asked Alex.

What surprised you most during your internship here? I would say the biggest thing was how much autonomy I had with the work that I did.

I did not expect. Welcome to Cloudflare, grab a shovel, start digging.

There's a lot of work to do. I feel like it wasn't like I was thrown in the deep end.

I had enough support to be able to accomplish the tasks that I needed to, but being a rising junior in college and being the sole product manager on a project like this is pretty incredible and an experience that I feel very fortunate to have gone through.

So I feel like because of that, I've learned so much about what it's like to be a product manager in the industry, have had to make product decisions by myself.

And like Alex said, lead meetings by myself that I didn't expect to have to do as an intern.

So I think being able to do that all independently, but with having the necessary support that I need from my manager and my mentor and everyone on the Cloudflare team that I've come in contact with has helped me learn I think as much as possible from this experience.

What advice would you give other interns? I would say I guess just buy into your time at the company you're at.

If you're at Cloudflare, something that I've really enjoyed is I've set up numerous coffee chats every week with people across teams, whether that be marketing or sales or other product managers, engineers, and that's really helped me get a better feel for Cloudflare's culture, but also all of the work that goes on here.

And I think one of the greatest parts of Cloudflare is how welcoming and open everyone is to getting to know each other.

And even as an intern, especially in such a remote environment, it's nice to have those coffee chats throughout the day just to get to know each other, talk about what work you're doing.

And I would say that's my biggest advice during an internship and in terms of seeking out and going through the recruiting process.

I know it's stressful. I definitely felt the stress of recruiting, but just trying to network as much as possible, you know, contact alumni from your school that might be a company of interest.

I know that's something that I did when looking into Cloudflare and the specific alum that I talked to was extremely helpful in navigating me throughout the interview process.

So I'd say that was probably my biggest pieces of advice.

I wanted to ask one thing that I've often hear from people who are students and young folks who are early in their careers.

This product management thing, what actually is your output? So you're not selling it, you're not coding it, you're not necessarily supporting it.

So what is it that actually comes out of you?

So like, what, to use the word, what is your deliverable?

What is the thing that you actually put on paper that gets consumed?

Can you talk a little bit like, how do you know that you've actually done your job?

What is the thing that you produce? What are you connecting? What information is going through you and where you're making decisions?

How do those get documented and what happens next?

Right, I'd say like the biggest deliverable in my mind for a product manager is obviously the product once it ships, but before that is your product requirements document.

So that was something new for me to be introduced to.

But right after I went through those information gathering calls I mentioned, I spent some time drafting all the requirements and business goals and customer use cases for this portal.

And then went through a review of that with a team that I would be working on to actually lead the design and development of the portal.

So then that helps guide the development and, you know, engineers or design can refer back to that if they have any questions about where we're looking for this specific project to head towards.

But then obviously after that, I think the role of a product manager is just to collaborate with engineers and design and whoever else is working on the project and making sure that everything's running smoothly and nothing goes up in flames and that, you know, we can ship the project and get it to our customers as quickly and as...

Yeah, another line we like to use at Cloudflare is, you know, there's just as many bugs in the spec in the requirements doc as the code.

You just can't see them, right? And so that's all the more reason to get as many eyeballs as possible on that PRD as early as possible and have all the debates and have all the arm-wrestling matches to be like, okay, if we do it this way, what is the impact?

If we do it that way, are we locking ourselves in?

And I think that's actually something we do well at Cloudflare, which is, you know, it's dangerous.

You can tilt toward bureaucracy if you spend too much time on the requirements doc.

I've seen, you know, in the old days that used to be called the waterfall model, but I think we've struck the right balance.

And I think knowing that there's a place where we write all this down.

And I guess last question for you, same thing we were asking Alex is what was most fun and what did you learn and what was most fun?

I think the biggest point I learned, this was my first product management internship within industry.

And I've learned a lot about, like I said, what it's like to be a product manager.

And I think similar to Alex, I previously have only had software engineering internships, realized that was 100% not for me.

And this summer, I've been able to experience what being a true PM is like.

And I definitely think this is something more up my alley. But I'd say the most fun thing is just being able to meet so many people, even though we're all working remotely.

I know, with Justin and Alex, we have a weekly product manager intern hangout where we just get to chat about work and...

No one ever invited me to that.

You're not an internist, man. You're not an internist. I didn't qualify as a PM intern.

That's about right for me. We'll add you to the next invite. But then also just all the wonderful people that work at Cloudflare across teams.

I've been fortunate to talk to people all over the company and I've enjoyed every conversation and learned so much from everyone that I worked with because Cloudflare is a place with a lot of smart people.

So I'd say that was the best part. You're lucky in that way.

How about you, Justin? Justin, what have you been working on?

Yeah. So I've been working on a product called PageShield, which is within the client-side security space.

So surfacing threats and helping our customers tackle them.

What made tackling this hard or easy?

Why now? What was going on that was the genesis of this and why hadn't we done it so far?

Yeah. And this totally goes back to the larger theme of how web applications have evolved over the last 10 years.

So we have a lot of application security that's really great at protecting origins.

We've got WAF. We have rotating encryption of very sensitive information in databases.

So it has become much more difficult for malicious actors to access application origins.

The challenge is that while application origins have become much more locked down, the client has become much more vulnerable because we see specialization in JavaScript vendors.

You have third parties that have other third parties that have other third parties.

And we find that for dependencies like marketing analytics, payments, customer success, that folks are just injecting into their web app, that those present a lot of a combinatorial explosion of problems, right?

We see some applications have hundreds or thousands of JavaScript dependencies that they're not aware of.

So in expectation, some actors have figured out that a couple of these dependencies might not be fully secure.

And by compromising those dependencies, they can control the logic that executes on the front end, which is inherently dangerous if that front end is handling sensitive information.

Got it. Now, so you've actually, you've been working on this project now for kind of an extended period of time.

How has your role as a product manager grown and changed and evolved as you've grown and evolved with the product?

Oh, that's a great question.

I think that when I first started at Cloudflare, and this would be June, 2020, we were in an extremely exploratory stage, right?

And we're always exploring how to make the product better at PageShield.

But we were really starting from zero in terms of what is the best formulation or client-side security product, what approaches are feasible for an engineering side, and what do customers actually want, right?

So we spent the first several months just figuring out what we can deliver, and also what will help customers actually catch these attacks and address them.

And then after that, we shifted gears into execution, right?

So we were able to launch PageShield during security week, onboard customers, right?

And once your product goes live and you have active customers, the balance between internal and external work changes significantly from when you're just doing research and proof of concepts internally.

When you think about that sort of shift for you, that shift from sort of the internal collaboration and development to the customer work, what are some of the key things you learned as a product management intern in engaging and working with customers?

Yeah, what I've learned is that we have such a diverse array of customers and diverse array of users, right?

Even within PageShield, you have CISOs who are the buyers, right?

And then you have security managers who are setting the product up, you have analysts who are using the product day to day, right?

And each of these folks have different needs, right?

So when approaching them, it's really important to sort of have a diverse array of folks that you're actually talking to, right?

Because if you talk to only CISOs, you might come up with a product that might be really easy to sell and have folks pay for, but it might not be great for their teams to actually implement and gain value from.

So this has been something we've been really deliberate about is just understanding every different kind of user and how we build for all of them so that the whole organization can gain value.

So Justin, what does the customer actually see? So as a script monitor and these kinds of PageShields, so you turn this thing on, it's watching to see, hey, wait a second, this website didn't pull in suspicious.js.

Why is it suddenly loading this?

Where did this thing come from? How do they even get alerted to that?

Where do they look to see it? How do they know that? And the developers go, wait a minute, actually, I wrote suspicious.js.

Sorry, I shouldn't have called it that.

But how do you tell the difference between something really, really concerning and something that actually that's just a wacky new JavaScript library that we're all in love with?

Yeah. So we have two approaches on a high level, right?

One is reported malicious sources of malware. And another is our internal threat research, right?

And my teammates on the data science team have done an incredible job creating new cybersecurity understanding of what malicious JavaScript looks like.

But in the first bucket, when we talk about reported malicious, Cloudflare has this very deep and extremely valuable kind of understanding of what the threat landscape looks like across the Internet, right?

These are threat feeds that we integrate across a ton of products at Cloudflare.

And it fits really nicely into PageShield as well, where we can tell our customers, hey, you have Magecart.com.js loading on your application.

And this has been reported to be malicious by the security community.

So this is definitely something that you should look into and confirm that it's either not affecting most of your users or all of your users, or that you launch a threat investigation in response and actually get rid of that malicious file.

That's the first bucket. And then the second bucket is what can we do net new that the security community even hasn't really found these files yet, right?

And this is looking for markers of maliciousness within files, right?

Evidence of things like data exfiltration and touching your DOM when you shouldn't be touching the DOM.

And so we're combining net new internal research with consensus by the security community to get well-rounded understanding of threats.

There's another question I want to ask you before we wrap.

You have a blog. You're a byline blog by Justin Zhao on Cloudflare .com.

You know, blog.Cloudflare.com is something that comes up a lot. And you in Canada talk about it.

So talk about the experience of how does that work in broad strokes of Cloudflare.

How do you get an article on blog.Cloudflare.com? And how did that process work?

Who helped you with that? And talk a little bit about that. Yeah, we started work on our blog around the time when we were positive that we were launching PageShield during security week, right?

Which happened around third week of March.

So I probably began outlining the blog post sometime mid -February.

And socialized internally, socialized with MST, socialized with Pat, my manager, and my manager's manager.

And really understand, okay, what are we trying to communicate about the client -side security space in general?

And what we're doing to solve for that.

And then after that, you enter the more formal blog process where JGC looks at every single blog post, right?

And gives them some great feedback there.

And finally, during security week, we were able to launch the blog post on blog.Cloudflare and got tremendous feedback.

Yeah, I think, yeah, you outlined exactly how it works.

I often get this question, like, how come you guys are so good at blogging?

It's part of the product life cycle. And there's a lot of effort goes into it.

And we track it as a deliverable just like anything else. So I think that was a really great story.

Justin, most fun thing about all of this? Shipping software, man.

The thing I love about product in general is that there's nothing like knocking a ship off of PM dev and then that has a customer impact.

Your team is that much further ahead and then moving on to the next piece after that.

Shipping is like driving down a freeway in a convertible at 60 miles an hour with the top down.

It's just like, it's exhilarating.

It's fun. That's great. I think we're close to time. And I just wanted to thank all of you for joining us on Cloudflare TV and sharing your experiences.

And of course, forget that, way bigger thanks for all the effort you did to help ship product.

That's really what this is all about. And Amrita, as the head of engineering, obviously, it breaks my heart a little bit that you said that you did engineering and you think PM is better.

But then I look at Jen's expression and I can't argue with either.

So if we see you back in the PM organization, that's awesome too.

Any other thoughts or closing things here? I just want to echo Usman's thanks specifically to the three of you for your contribution and effort of the course, your internships.

Interns are a key way that we build and deliver product at Cloudflare.

You guys bring in energy, you bring enthusiasm, and you also bring a fresh perspective.

I find every time I work with interns, I learn something myself about being a product manager.

So thank you for everything you're doing to ship great product to our customers and helping us grow as an organization.

So I hope to see you all again soon. Yeah. Thanks, everybody.

All right. Thanks, all. Thanks for watching, folks. Bye. Bye.

Bye.