Product Managers in Parks Having Pints

Good afternoon everyone. My name is Garrett Galow. I'm a Director of Product here at Cloudflare and I'm excited to bring you the second edition of Product Managers in Parks Having Pints.

Today I'm drinking a lovely refreshing Topo Chico which if you aren't from maybe the southern part of the United States you don't know about it's a mineral water, carbonated mineral water, that is undecidedly the best one in existence and I will defend that until the day that I die.

More importantly I am joined by Melissa Niu who's a Product Manager at Lyft and I will actually allow her to introduce herself to all of you.

Hey everyone, thanks Garrett.

I'm Melissa. I'm currently a Product Manager at Lyft and today I have a pint of Deschutes Black Beet Porter.

It's very rich and flavorful and you know a good alternative to coffee in the afternoons.

Yes, yes. People don't know that I'm much a coffee fiend myself though I've been actually much more on the controlled amount of it recently.

Before we sort of get into the main parts of today, as per usual on the show, since we are you know having this conversation in parks you'll see both of us are in different parks so I invite you to try and figure out where we are at the end of the show.

We will reveal the places that we are.

Feel free to send guesses to livestudio at Cloudflaretv.com and we'll receive those here to check at the end.

Before we get into the main topic for today, Melissa tell me a little bit about sort of your journey into how did you get into product management?

Sort of what does your career look like to where you've gotten today?

Yeah absolutely. So in undergrad I sort of stumbled my way into computer science so I come from a more technical background but pretty early on through some internships and just exposure to technology I realized that I cared a lot more about the problems we were solving than how you know how to actually build the solutions.

So that led me to product management as a role because I realized that as a PM you get to identify and solve people's problems and not just think about the architecture and the solution itself while that's obviously equally important.

So since graduation I've been in product my entire career working at Microsoft Glassdoor and now Lyft.

Over time realized that working on consumer technology was really where I thrived and you know the problem space I was interested in and so that's what brings me to where I am today.

Awesome yeah I think it'll be really interesting today in our conversation about you know how does some of this stuff vary if you're working on consumer products versus enterprise or business to business products.

So sort of if you joined us last time the conversation we had was around customer conversations so really how do product managers talk with customers of various types and understand what it is they might need what are their problems that they're having.

Sort of as a good continuation of that discussion today we're going to talk about sort of problem validation in some cases even product validation.

So how do you sort of use all of those things to understand and come up with you know really the problem statement of what you and the team that you work with as a product manager are going to try and solve.

So I guess to kick this off sort of Melissa you know how do you think about problem validation right.

I know last time we talked about you know you're talking to a lot of different customers each customer specific needs they might vary they might conflict.

How do you sort of think about taking those together into a singular problem statement that you can really rally a team around to really dig into.

Yeah I think the first thing is just identifying potential issues right it might not be the one you want to focus on but just really getting deep in the problem space and a lot of that is just being exposed to the product.

So obviously working at Lyft part of that is easy you know just being in a Lyft you get to experience the product itself you get to see kind of how different people interact with the product.

I think one of the challenges I learned early on was I actually only worked on the driver side of the platform since joining Lyft and I've never personally driven for Lyft and so immediately I realized you know when I'm in a Lyft rather than just being on my phone or being absorbed by other things I started paying attention to what the driver is actually doing what was their screen like what were their interactions you know what were challenges they might be having.

So that's definitely one thing that you can do is just really using the product especially if it's a part of the product that you normally don't interact with a lot and then on top of that I think you know just thinking about interacting with other people who are exposed to potential problem areas that can be talking to customer success managers it can be looking at online forums the subreddits on you know on Lyft and Lyft drivers are full of very opinionated people obviously you want to take everything with a grain of salt but those are all good places to start identifying potential issues and I think through all of that you get a good balance of qual and quant.

You know the qualitative pieces being those things like user feedback and talking to users and the quantitative pieces being more you know around data and metrics and the parts of the business that you might notice are struggling or have you know you notice some changes in some of the data and you have more of one than the other you want to try and go out of your way to see the one that you're lacking.

So if you do notice for example that you know a driver is having issues like completing a ride right then maybe you dig into the data and you ask your data scientist hey how often do we see ride completion drop off or how how often do we see that this button is tapped more than four or five times because clearly it looks you know that it wasn't responsive or whatever it may be and that's a really great way to get a more holistic view of the product and the potential gaps there might be.

I really like the point you make sort of the balance of qualitative and quantitative aspects to it because you know I think there's what's good reason there's a big push across you know companies especially in SaaS or software based companies of you know qualitative measurement like can we measure these things can we measure all the different aspects of our products you used but you know there's an essence of you know you miss some of the the raw feedback and really emotion around certain things when you only think of things from a data format.

I know here at Clubflare we think a lot about you know we have different parts of our business obviously like our enterprise customers we can talk to directly very easily but our sort of broader more consumer type customers like we don't as easily talk to them and so we really have to balance those things ourselves.

One of the things I wanted to go a little bit on is you mentioned sort of like the you know using your own product aspect and there's sort of a mantra that is sometimes said in product management of like you aren't your own customer but yet you can learn a lot from being a customer of your product.

How do you at different times balance sort of that like firsthand experience of using products and stuff that you worked on to try and understand versus maybe other mechanisms to understand like user journeys and user understanding of the product.

Yeah absolutely I totally agree with that.

I think you know being in tech and being exposed to tech as much as we are we have a very different view of how things are supposed to work.

One thing I distinctly remember was when my dad tried to use Lyft for the first time when he was visiting and you know I was explaining things like they were super obvious he was like how was I supposed to know that and so a lot of times you know just those reminders help you ground yourself a little bit and the realities of the fact that while you may have you know been part of the journey as the product has evolved for many people they're being exposed to v5 v6 v7 of your product where you've now you know pushed out so many different changes since the one where it was simple and you know maybe straightforward at that point.

So I think one thing you know I always encourage people to do is do it early when you first join a company you know oftentimes that's when you've been exposed the least.

One thing we try and do isn't we encourage folks who are new to the driver side of Lyft for example to go and sign up and be a driver even if they don't actually end up driving going through the onboarding process is usually something no one's done before and just going through that makes you realize like oh these are all the steps that people have to go through and so being aware of you know when the best times are to be exposed and have that firsthand experience is really important but then I think it's important to get a diversity of opinions when you are reaching out for qualitative or even quantitative measurements.

So you know one thing is if you are reaching out for user studies for example think about what your you know demographic may be and try and make sure that the people you're recruiting are representative of that.

So for example if you know 80% of your customer base lives in urban areas you know try and make sure you're recruiting folks who are in urban areas or if you're doing a user study on congestion or navigation and high density areas if you're only recruiting people from suburbs you're not going to get the type of you know feedback that you're perhaps looking for.

So that's definitely one way out of many right to make sure that you're not anchoring your problem space and your solutions on kind of your individual perspectives on the world and on the product itself.

Yeah you mentioned you know we use Cloudflare ourselves a lot but you know we also have like the the benefit of being Cloudflare so we can sort of often get around like bumps in the road to use a turn of phrase that like a customer might not be able to and so like we're very much reminded when we sort of find diverse customers that are using things in different ways.

I think the you know we a lot of our customers are technical and so we often think about you know oh they understand concepts but be you know similar to your you know story of your dad trying to use lift for the first time right there's a lot of people that come to Cloudflare because they they know they need something but they they don't know enough of the underlying you know terminology or things like that to try and do it and that's like a constant thing we're trying to think about of how do we how do we portray things in a way that's understandable to to everyone.

So I sort of want to go from that and so you know we talked about you know you know recruiting diverse people in order to to understand qualitative measures quantitative data how do you then sort of think about okay well we we have all this input data whether you know it's for people and forums or you know from customer success people or whether it's from our user metrics how do we start to combine that into a way to really understand what might be the most important problems people are facing.

Yeah that's a great question you know I think as you start digging into the quant quant and I think we talked a little bit about you know how do you quant like how do you then evaluate the trade-offs between problems that you're having and I think one thing I think about a lot is how do you weigh you know low frequency but high pain problems against high frequency versus low pain problems.

I think oftentimes you see a lot of both and it's really hard to make that trade-off right so for example maybe something happens daily but the end result is you just have to wait an extra second for a button to load and then it works.

You know that's high frequency perhaps but relatively low pain obviously it's annoying and it you know maybe decreases the amount of trust that your customer has but it's not necessarily a deal-breaker versus you know maybe something happens weekly but it crashes the entire application you just start the entire process over while that may be lower frequency it's super high pain and you know maybe the resolution to the problem is even made more painful because now they have to perhaps reach out to you know an actual support person or what-have-you so I think trying to really balance those two can be really challenging and I don't think there's you know a net magical formula and sometimes you have to use your product intuition but at least being aware of the fact that you are making these trade-offs and that you're not just picking you know the most frequent issue or the most painful but that you're looking at a balance of both can be really important and then I think at that point it's also about starting some of the solution process as well because some problems they may be the worst problems but maybe they're not in your control right like for example you know with lift navigation and tunnels super hard problem you know drivers have really low GPS signals sometimes we're like oh we're rerouting you oh we're trying to send you a ride you can't receive it but at some point we have to stop and say like are we best suited to solve this problem even if that is the biggest impact area we can make maybe we decide no this is not you know something that we are well scoped or well resourced to solve so in that case maybe you know you pick something else and so as you start you know some of that goes into the solution piece but thinking about at least the first level of that I think can also help you see if there are any low-hanging fruits where it's maybe you know just a little bit of work to solve this button not loading issue or whatever it may be right or adding an extra confirmation before you do something extra destructive because you know the extra pain of going through a confirmation model is still you know way better than accidentally doing something catastrophic so you as you go through this phase I think it's important to not just look at the problem space but also you know the ease of the solutions and sometimes it's a little bit of a back -and-forth right maybe you pick the most important problem and you ideate on solutions and maybe that doesn't yield you anything that you think is possible you go back to the drawing board and it's definitely an iterative process I think another thing too which is a slightly different way to look at it is also concerning like the root of the problem right is it user error is it expectation setting is it you know the technology itself and I think that will also help you figure out whether you know your team or whether this is the right type of problem to be solving or whether the problem can be solved in a totally different way right if it's just about expectations because users don't understand what's actually going to happen when they take an action because they don't realize there's no undo button for example and this is a one-way street then maybe you can actually prevent the problem itself by you know setting the right expectations and so I think taking all that together can help you identify which ones are the right you know engineering problems to solve versus perhaps alternate solutions that can remove the problem altogether yeah I think you make you know a really good point about you know the not just what you know your example of like frequency versus pain of the palm is like a common you know kind of framework that some people think of these things and and think the that extra dimension of you know how you know how might it look to solve this from how likely is it we could solve this problem in an easy or difficult fashion I think is you know a really important point that a lot of people don't actually think about first because you get stuck in that like do I solve all the crashes or do I solve all the little minor glitches without really thinking about like the value of solving those problems right what's gonna be with along with the cost there's example of like for Cloudflare where it's like we would ask people you know kind of on the aim of you know being being a customer right if someone interviewed the Cloudflare very often we would ask them to you know sign up for Cloudflare because it was free and I go through the onboarding flow and like you know give us feedback like tell us what you think you know what's good what could be improved and all of those sorts of things and you know one of the most common things we got was you know changing name servers for a domain is not a simple process and people were like yeah that's like kind of an arduous process it's really hard for us to control kind of like the tunnel thing where it's like it's something that's done at third parties we don't control it and actually like our sort of the the back pocket solution we always had was you know oh if only we had our own registrar where you could just register your domain with Cloudflare then you don't have to worry about the name service we'll take care of it for you and that eventually became a reality right then sort of you know for a few years when I first started right that was like consistent problem where the pain was actually pretty high there but the solution was also very costly to build in a certain sense it took a lot of investment and so it was something that we you know waited for a while to choose how to do I think it's a really important point yeah you know some of those solutions like you said we're we're you know at first glance yeah this feels impossible this feels you know way out there I think those are still really important conversations to have because one it sets you know some anchor for where you want to be and then two oftentimes you do realize like maybe there are steps you can take in that direction right like maybe we can't solve the network connectivity you know a tunnel issue but at least identifying that means we you know should be aware when drivers are in a tunnel and maybe when we know they're in a tunnel we try and not reroute them because we know you know signal is bad or we try and not send them rides because likely they could be any part of the tunnel or what have you and oftentimes that means you get to short and medium term you know solves that can kind of pave the way to the long-term solution that eventually solves you know all of the right problems yeah I sort of wanted to spend a little more time talking about one of the last things you mentioned was sort of like getting at what the root of the problem is if it's like I think user error is one of the examples you gave or other aspects how do you think some of those affect this sort of process because I think you know it's a good point of light obviously something that you know is like a crash is very much in you know your control as you know going and finding the fix of that but something that is maybe you know missed expectations you know presents a much fuzzier picture in a certain way how do you try and you know think about waiting those in some capacity yeah that's oftentimes yeah it's a little bit hard right because some of those are just so drastically different I think weighing them you know there's kind of different levels one is weighing the different problems all together right and I think the challenge there is oftentimes there isn't one common metric or one common thing that you can weigh right how do you weigh two crashes per week versus 15,000 accidental little button clicks or whatever right they're hard to trade off and oftentimes you need to align on some common metric or something that you know is representative of the value you can gain from solving a problem and then when it comes to like you know actual potential solutions I think it's very specific to your specific team and the resources you have right if you're engineering constrained or if you have a tight deadline and you know that because you know trying to get something in the app store it takes you know at least a week of fake time etc then you have to get more creative right of other sessions or other things you can do to solve it in shorter period of time versus if you have you know a lot of engineers but you have you know no one who is able to actually help on some design or implementing or whatever it may be then maybe you skew the solution in a different way I mean that's where you know when you're building a team roadmap or you're thinking about what you're doing for a quarter that's where the balance comes in right you'll probably have a laundry list or you know some list of all the problems and all the solutions to all the work that you can do and even if it's staffed rent you know from one to a hundred you might not tackle the first three just because the combination of resources and timing and alignment and whatever it may be you know isn't right for for what it is you're trying to solve in terms of actually figure out what the route is I think the most important thing is just asking why I know in a lot of our engineering retros when a sub happens or whatnot right you like the five wise right like why did this happen okay why did that happen and why did that happen right and usually you get to like the true root of the problem and when we talk to customers and when we go through these exercises we ask ourselves the same thing right so I think earlier I mentioned an example of you know drivers accidentally ending a ride early right and you know sometimes you ask why like well was an accidental tap like what happened well why did you tap that oh I was trying to go you know and do this and eventually you figure out that like some of the root of it is a combo of them right it's that one some of you know as a driver progresses through the core ride flow sometimes we ask for confirmation sometimes we don't which means we're not setting the right expectations for a driver on when they're supposed to tap once versus twice right you know make some of it is the language itself saying drop off Garrett does that mean I'm now headed to drop off gear or does that mean I've already dropped off gear some language can also you know cause potential confusion especially when you think about internationalization and translating those you know strings some of it is not you know is the technology piece right like if you're in a low cell service area for example or in the tunnels that I keep referring back to you know if you tap on it and nothing happens you think maybe it didn't register so you keep tapping and now maybe we've registered multiple taps and now we've progressed you through you know three steps of the ride so oftentimes it is a combo of all of the above and oftentimes you can solve the problem in four or five different ways right but getting the root of the problem of like why this is happening and why you know it's accidentally happening can can help you at least figure out what are some possible options and eventually you know what are possible solutions yeah I think that makes sense it also is crazy because I'm just thinking about how so many of the problems you have to solve in the context of someone is literally driving a car and operating a motor vehicle and like that is not a thing that exists at all in my world of like when people are using our product they are generally sitting at a computer probably not even on their phones because most of the time if things are going well you don't need to try and you know change security settings on your edge of network you know on the fly and so it's like you have a there's a completely different operating model in which you know your customers aren't focused on the task at hand necessarily because they are you know driving a car having a conversation with the passengers right and so there's all these other things that are getting in the way of like you know very clear signal of what's happening right whereas I think most of the time for our customers you know they are fairly focused on the thing that's trying to do to a certain extent there are always distractions happening when someone's using any product or service great point and that's you know and that's something we think about a lot and it was the first time I really had to think about it which is you know putting I'm putting your users in the context of whatever their normal context is right oftentimes you think of user research as you and I sitting across the table in some office right and me showing you prototypes or me just asking you questions and there are a lot of times we'll do our user research in a car or we'll ask the driver to be driving and we put our prototype on the car and we ask them to like interact with it because oftentimes things that are really easy when we're sitting down holding a phone here you know become very hard when you're doing this and trying to do this at the same time and you know to your point exactly right there are gonna be distraction there are gonna be other things and that's something what you consider right we were in a prototyping session once and you know a few notifications and text messages came through and it totally blocked the navigation instruction that we were trying to demo and you know those are things you don't necessarily think about when you're designing in the confines or problem you know even ideating in the confines of you know an office space or in a design studio or on a whiteboard and so a lot of times it takes takes that extra step to actually identify oh maybe it wasn't a technology problem at all it was the fact that our design made it so that any notification that came in blocked the most important information you know and that would then force us to reconsider how we surface our information and when to make sure that it's both safe and relevant to the drivers so that we're not giving them you know just excess information at the wrong time yeah I feel like there's some sort of statement ignorance like if you're not user testing in the environment in which your users operate you're basically wasting your time maybe that's a harsh way of putting it but it's probably a fairly accurate way of putting it we won't have a ton of time to get into this but I want to just talk a little bit about so we've kind of gone through this process we've we've come up with we found all these problems things we could be solving for our customers things we could be doing let's say you know you talked about having the stack rank of all the things you could be doing let's say you know you've identified here's you know one of the most important ones we're gonna work on I don't spend a few minutes hanging a little bit how do you think through sort of that ideation we kind of hinted at it in some places of like you know understand the cost of doing something can help you inform whether you should solve the problem or not well what what does some of that ideation look like to you to try and understand how you might better solve a problem yeah I think the biggest thing is just going broad we oftentimes do brainstorming sessions with you know our entire extended team right engineering's designers data scientists like what have you like anyone who you know who wants to is welcome to come and just going really broad with the ideas can oftentimes yield really interesting results and interesting solutions that we've never thought about before and then I think it's also just prototyping and getting getting some ideas in front of users early before you've really invested in it I think you know when you're 90% of the way through to the finish line it's really hard to say no let's not do their last 10% because we don't think it's the right product to build and you know at that point it's too little too late so no matter what fidelity it's in like just getting in front of users I Microsoft we used to just go around the cafeteria with our prototype and times big hey can we chat with you for five minutes like you probably use this product or even if you don't that's fine lift it's you know going to our driver hubs and while drivers are waiting for their cars to be serviced we're like hey do you have five minutes you know just getting quick feedback and even though you know take everything with a grain of salt again but I think that's really important and then I think too is just you know once you start thinking through the actual solutions going through and I think some people call it the ice framework it's like impact confidence and effort right your times your confidence divided by your effort so if it's something that you have you know high impact but low confidence and a ton of effort you know maybe that's not worth it for you to invest six months of time but you're not confident that's gonna give you the impact you want right obviously it's not a perfect formula but at least evaluating that to see you know are there quick wins are there quick MVP things that you can ship or just ways of validating whether or not your solution or whether or not you've been headed in the right direction absolutely yeah that's something we focus on a lot of club players you know how can we test the smallest thing possible with customers whether it's something we actually put in their hands fully or just a test so we can validate that we move in the right direction keep doing that I we could definitely keep time with this for a long time but before we get to the end of our time wanted to ask you the question we're gonna ask every time which is what is your hill question so what is your hill as a product manager that you are willing to die on the opinion that whether popular or not you hold so dearly that you will defend it till the end of time yeah I think for me it's that experience is just user experience is just as important as business impact and I think you know it's sometimes hard to quantify right how do you quantify that this is a better experience or that this experience needs to be improved but I think it's really important because over time that's how you just turn and lose you know lose users and while I'm no growth PM I think every person who's worked in growth before will tell you that you know it's very expensive losing you know it's very expensive losing users and it's very expensive gaining new ones and your best bet is really retaining the users you have and oftentimes you know we don't think enough about user experience when we can see these quick business ones and that's really hard to walk back from yeah I think you can probably find plenty of companies that have sort of you know deteriorated that user experience over time and then things start going up you know I'm a run and they're like what happened what happened in that point it it's it's really too late to go back yeah so in the last little bit of time we have let's talk about where are you right now Melissa I am behind me you'll see the painted ladies I am in Alamo Square Park here in San Francisco where I spent many weekends since shelter in place just enjoying a little bit of perfect yeah that's actually not far from where I live in San Francisco so I've been there myself as to where I am I'm in Yosemite National Park some people call it Yosemite if you've seen any recent news somewhere I'm actually haven't been but plan to actually visit for the first time in about a month or so so really excited for that and with that that sort of brings us to this time so thank you very much Melissa really appreciate the time coming and talking with me today and thanks to everyone watching yeah thanks for having me see y'all next time