Product Stories with Patrick

Hello and welcome to Product Stories on Cloudflare TV. I'm your host Patrick Donahue and today I'm joined by Aaron Pacheco of Acquia.

Welcome Chico, how's it going?

Great, how are you doing? Doing well, thanks. Thanks for joining in this first segment here on Product Stories.

Each week my hope is to talk to a product leader at top technology companies around the world to get a sense for what it's like building products at their organization, some of the challenges that they've encountered and sort of how do they develop strategies to drive change throughout their organization.

So today Chico, I'd love to chat with you about maybe a recent product you've been working on.

Is there anything that comes to mind that would make sense to chat about?

Yeah, I think the best one is a feature we've been working on on the Acquia Cloud Site Factory platform related to SSL management scale.

So would it be helpful if I provided a little context about the ACSF product?

Yeah, sure and actually I forgot to ask you if you could maybe start sort of just telling a little bit about Acquia.

We've been, you know, Cloudflare and Acquia have been great partners over the years.

Maybe you could share just a little bit about what your business is and then jump into sort of the background as you mentioned.

Absolutely, yeah. So Acquia is the world's leading platform for managing and building digital experiences.

So another way of saying that is we help organizations adopt an open source technology called Drupal.

It's a content management system and it helps people develop sites and non-proprietary software and then expand them to change the way that they do business.

So we've seen a lot of EDUs, financial organizations, government organizations, pretty much anyone of any scale can adopt Drupal.

Again, it's non-proprietary, it's open source and we help them be successful by giving them a, you know, optimized cloud platform to run on.

It's secure, it's compliant and then we provide a lot of tools and services to help with marketing, again security, performance and pretty much anything along those lines.

Cool. And so what was the platform that you were mentioning that you were, you know, the new feature that you were working on?

What is that?

Is that the platform you just described and if so can you kind of tell us a little bit about the, you know, the problem you're trying to solve with it?

Oh, absolutely.

So as noted, you know, we're a digital platform, we host thousands of different sites and organizations and what we've discovered over the years is that there are a lot of organizations that don't need to build a custom site for every single site that they need to run.

There's a need for what some might call brochureware sites or templatized sites and customers have the need to spin up dozens, hundreds, sometimes even thousands of sites that maybe don't share the exact same content but they need to look and feel similar, they need to be easy to maintain, to update, to modify and to scale, right?

So we've developed a platform called Acquacloud Site Factory that makes it easy for customers to create templatized sites at any scale.

So, you know, at that use case there's varying types of complexity that we have to work with and the one that we've been solving recently is related to SSL management scale.

So as you know, the web is really moving towards a place where everything needs to be over HTTPS.

It's very important to have everything encrypted and that's an easy enough use case for single sites, right?

You go to a vendor, you either buy a certificate, you use Let's Encrypt, you get a certificate, you install it and you're off to the races.

But there's additional complexity when you're working with hundreds or thousands of sites that we needed to work around.

So, you know, the use case that came to us was, hey, I'm a customer, I've got hundreds again or thousands of sites and the specific need is we need multiple certificates and that can come down to two different use cases.

One is as a security -minded organization, I want a different certificate for every domain because I want to reduce the blast radius in the unlikely event that a certificate is compromised.

So you don't want to risk a certificate with hundreds or thousands of domains on it being accidentally exposed by mishandling or whatever on the end users side and then somebody taking that certificate and installing it elsewhere, right?

So a lot of organizations are saying we want a certificate per domain or certificate per business unit so that we're not combining too many into one, which is a totally valid use case.

And then on the other side, you have a general Internet standard around the number of domains that you can reasonably fit onto an SAN or a UC certificate.

Those are certificates that can have up to, depending on your vendor, 150 or 250 domains.

Once you start getting beyond that, the amount of time it takes to perform, you know, decryption, encryption and general handling of the certificate grows longer and longer.

So everyone's kind of agreed, let's try and keep it under 250 or 150 domains, which again poses a problem for customers at scale who have hundreds of domains or thousands of domains to manage.

So this posed an interesting problem to Acquia.

You know, our platform was built around, you know, single environment, single certificate, and now the question is how do we operate that at scale?

And so, I mean, it's an interesting story how we came to solve the problem.

So we knew that there was a need. We knew that our larger customers were finding this and some of our largest accounts were saying, Acquia, we need your help to solve this problem.

But, you know, there's technology implications, there's roadmap implications.

So we had to sit down and, you know, as it became more important to our largest accounts, we had a trade-off decision that had to be made.

We could solve the problem one way and it would be solved faster, it would be good, or we could take a little bit of extra time and solve it a different way and it would actually unblock additional features for those customers down the road.

So I want to just interrupt for one second.

So I want to understand kind of how are you doing this today?

What was the sort of the status quo that you were kind of faced with and what does your process look like and kind of what was challenging about that process, whether time-consuming, costly, etc.?

Like what was the starting point that you were faced with?

Yeah, absolutely. So, you know, as you can imagine, you know, an organization of our size, we have a number of competing priorities.

So the question is, you know, it's twofold. It's when do we put something like this in the roadmap, which is, I mean, that is the bane of every product manager's existence is when, where will it fit, how soon can we get it?

So there was a question of where it fits in the roadmap, who's going to do the work and, you know, how exactly do we solve this problem?

And, you know, there was a lot of pressure for us to develop something again for our largest accounts.

So the motivation there, you know, when you look at it just purely in the sense of mathematics is, well, if there's a way to solve this problem well, and it satisfies that use case, let's do it.

But doing that, you know, as I came onto the project and was kind of evaluating the approaches, it would require us to build what would be called a siloed solution.

So that's a solution that solves one specific use case, but it doesn't satisfy other use cases down the road.

It's very, very isolated.

So, you know, me coming into this situation, the way I looked at it is, well, if we do this, this way, it's good, it solves a problem, but does it help us down the road?

Isn't the best possible solution we can offer our customers? Or do we develop one that's a little more advanced, takes a little more time?

But then the issue becomes, you know, customer expectation management.

You know, when the customers are saying, I need this feature by X date, it adds a complication to the math.

And you say, okay, well, are we going to do what will make customers happy now?

Or are we going to do what will make customers happy later? So, you know, it was an interesting position early on in that project where, you know, it was a pivot point.

And, you know, I had to use a little bit of internal political capital, but say, basically, like, can you guys just give me one sprint, give me two weeks, let me look at this with some experts, let me reassess the approach.

And here is the potential value long term, if we do this a slightly different way.

And, you know, we went through the exercise, and we designed an alternate solution.

And it's like, this will take, you know, a quarter or so extra, but it'll be really, really good.

And it'll help us unlock a number of additional features. So then once you have kind of company buy in on that, then the question becomes customer expectation management.

So you know, I had to go about that process. I'm really curious, as someone who is building products myself and facing similar trade offs.

You've got the people that needed yesterday, and then you've got the people that want a whole bunch more features, and they're willing to wait a little bit.

How did you kind of navigate that those conversations and those expectations?

Yeah, with customers?

Yeah, with customers. Yeah. So it was a combination of things.

So I actually, you know, taking over a product fresh, I'm sure this is a position a number of product managers have dealt with, you know, you, you're in one part of the organization, or maybe you're outside the org, you come in, you have your priorities handed to you.

And it's like, alright, here's your roadmap, run with it.

And it was it was an interesting position to be in, you know, I had to speak with a number of accounts at that at that phase of my joining that specific product line at Acquia.

And I wanted to gather feedback from different customers and understand what is the scope of work that you would love to see delivered on this product?

What are the features you want addressed? What are the bugs you want to address?

What are the things you want us to work on? And that gave me a representative sampling of what is important to all the customers on the platform and why.

And I was able to say, you know what, if it was kind of twofold, if I make sure that certain quick win features are prioritized, over the course of the other larger projects lifespan, we could create a narrative of consistent victories and consistent wins that keep our customers happy and solve some of their problems until the big bang comes up.

And so in that way, we were able to make sure that our streamlined, I'm sorry, our baseline accounts were all happy with the work that we were doing, we were releasing new things they were excited about.

For our larger accounts, it was a more tricky conversation, because, you know, we're going to them, they're saying, I have a problem now, we need you to solve it.

And we essentially had to, it had to be kind of a good faith discussion, which was, look, here's my vision for how this product evolves.

Here are all the benefits that we could provide you if we do this, this way, versus doing it that way.

And, you know, we were very fortunate in the sense that, you know, our largest accounts were very understanding.

They said, look, we'd love to have this problem solved now. But if you can promise us you'll get one, you'll one solve this in a reasonable amount of time, and two, that you're going to solve other problems for us as a result of this work, then we're going to give you the time you need to do this right.

But we need you to commit, we can't, you know, and so that gave us kind of the breathing room we needed to get the project started.

So two questions on that. So do you find that that sort of transparency and laying it all out for the customers and sort of reading them into your plans is a helpful part of the process?

Do you think that that makes it more likely that you're going to get, you know, accomplish what you want to get accomplished?

You know, I've seen, I've seen it both ways. You know, I've worked with product managers who, you know, they like to kind of take more of a marketing spin on things and, you know, make sure that everything is kind of what I would call a little bit light and fluffy.

And then I've seen the product managers who are like, look, let me level with you.

And, you know, as a person who is a customer myself of different organizations, you know, I find that I actually prefer the approach of transparency, it helps build trust, and maybe it shows a little bit of what's going on behind the curtains.

But at the end of the day, I found that customers are more willing to trust and reason with you if you level with them than if you're spinning a situation and, you know, trying to provide unnecessary air cover.

So that's the approach I try to take. It's so far a knock on wood, it hasn't bitten me.

But, you know, I'm really grateful for the fact that we've been able to build strong relationships with our customers using that methodology.

Yeah, I think that's a good point is the building the relationships, right?

This is a repeat game business, you're not in their buildings, not your consultant, and you're coming in and doing one project for a company, you know, you're trying to be that trusted advisor on whatever offering you have and deliver, you know, wins over a long period of time.

And so I think you can demonstrate, you know, that you're going to deliver what you promise, you gain a lot of credibility, and you're able to kind of do that again and again.

One thing you mentioned about kind of the commit, what does that what does that look like?

I think, you know, different organizations, you've got pressure from from the sales side to try to write something into a contract.

I mean, how do you guys think about that at Acquia?

And how do you navigate this sort of the balance of wanting flexibility and knowing that, frankly, stuff can go wrong on the engineering side versus, you know, customers wanting a specific date?

Yeah, no, that's, um, that is the other bane of a product managers existence, right?

Every date is final until it's not.

So, you know, software development is tough. I think, you know, those of us who have been in it for a while, we know that even the best laid plans, like you can be green on a project until the last minute, and then Oh, no, there's a bug, you know.

So we went into this kind of with eyes wide open.

And we took kind of a two pronged approach. The first was we had to get a general what we call a t shirt size, what is the rough level of effort?

Do we have a good RFC or request for comment with an architectural plan of how we'll solve the problem.

And that gives us a rough idea of the scope. And then we get all the teams in a room and say, how long would it take you to do this?

And what is the sequencing?

Sequencing is a thing that people often forget about. It's one thing to get everyone in a room and say, all right, this team can do this, and this one that and this and that and this and that.

But if you don't think of the sequencing, and you don't put buffer in for the sequencing, then what you end up with is, oh, I didn't realize that this team's two months of work came after that team's two months of work.

And now it's actually four months, not two, right? So so we had to have those conversations, we had to get a rough plan, we said, Alright, this team is going to execute in this order, then this team's going to execute, and then we'll unblock these teams.

We also had to factor in down to what we call downstream impact, which is if we build features on certain teams, there is an impact to the customer success organization, the operations teams.

So we had to look at how it would impact documentation, tooling and automated mechanisms.

So all those are factored in.

And what we did with our customers, as we said, here's our proposed approach, we'd like to go down this route, give us one month to get to get a timeline to you.

Again, they were very accommodating. They said, you have, you know, 30 days, come back to us, let us know what the timeline is.

So we went through all those grooming exercises on multiple teams, and said, Alright, we think on this project, we can have it have the back end work done in 90 days, we can have the front end work done in 30 days.

And then I look at that. And I say, Okay, based off of unknowns and complexities, and you have to plan for everything, right?

Basically, what is the what is a reasonable buffer on this project?

So I think in a not imperfect world, as a general rule of thumb, I always try to say a project will take twice as long as you think.

On this one, I just said, I'm going to give it a 50% buffer.

So I said, look, it's a roughly quarter long project, I'm going to give it six weeks of buffer.

So if we were expecting it, would it be out on May 1, or, you know, by May 1, I'm going to give it anywhere from June 1 to June 15.

And, you know, we ended up actually really close to those timelines, you know, six months out to be that close in your timelines is, you know, on a project this complex, we had eight teams collaborating on the solution.

So, you know, we will end up having been four weeks off in our estimate, which on a project that's a six month scale is actually not that bad.

But yeah, and that includes I mean, we have released versions of the service early for testing purposes.

And we're like, Oh, little bug here, little bug there.

But we're able to smooth those out, streamline them, and get them out.

And now, you know, when I'm having those, those conversations with the customer on the verge of release, they're like, look, we get it, we see light at the end of the tunnel, take the time you need, make sure it's polished and get it out to us.

And, you know, they're excited, they want the solution, and they're willing to wait the extra week or two for it.

Those are the best products that the customers are really just pulling out of your hands, right as it's coming off the engineering board.

So one of the things that you said that kind of struck me is there's eight teams involved, like, how did you keep them all organized and working together and rowing in the right direction?

And you mentioned there are some interdependencies, like, how do you manage that?

That seems like a lot to coordinate and keep straight. It is. It's a huge challenge.

You know, in our organization, we're fortunate enough to have not just product managers, but also product owners and project managers.

And we have very clear delineation of what those roles are.

So the product manager, you know, I help set the priorities and explain the business impact.

And then we leverage project managers to help do the coordination between teams, and the teams are run by product owners.

So, you know, the first part was getting the product owners and their corresponding tech leads aligned on, we're going to do our chunk of work in this order.

And with this timeline, and a little bit of buffer, and at a certain point in the project, team two will be unblocked, and they can proceed.

And then, you know, you coordinate with that PO and that tech lead.

What we found that worked really well was, again, having the RFC up front where all the tech leads aligned on the approach.

Where we found some struggles was in the kind of the day-to-day communication when a bug pops up in mid -development.

If that causes a story to carry over between sprints, now the timeline's off by a two-week sprint, right?

Unless you can pull it in. So you need to make sure there's tight communications between the team on a given project.

Now, these are all teams with their own distinct scrums. So we got to the point midway through the project where we said, you know what, there are so many teams involved at this point, and team A relies on team B, but team B needs to get a bug fix out through team C, which unblocks team D.

We got to the point where we said we need to start doing weekly, almost a scrum of scrums, where the teams all get into a room together for 30 minutes.

And we literally walked through a smart sheet and said, all right, how is this part of the project going?

How about this one?

How about this one? Do we understand the dates, the dependencies, the order of operations, and any of the downstream impact?

You know, this is where we had to factor in, again, the documentation, the testing cycles with specific customers, the phase rollout to different types of accounts, and the impact to internal tools.

And then it just becomes like a spider web where you're like, all right, I have to follow this thread, get that done.

And having a project manager and doing those routine syncs and holding people accountable for these are the timelines you commit to, do you understand the delay here has a cascade effect down the rest of the delivery path.

And then, again, just ongoing communications and alignment.

That's great. That sounds like a great technique.

It seems like it's working well for you. One thing you said that struck me also is, or I think people need to be reminded of sometimes, is the impact on the internal tooling.

And so I think for us, we want to get products in customers' hands as soon as possible.

In some cases, that means giving them functionality before the tooling internally is matching the external experience.

But clearly, you want to make sure that you're not leaving a lot of debt there over time.

And so I think those can be hidden costs that cascade throughout an organization if everyone managing it internally is taking X number of minutes longer times hundreds of people.

Is that something that you think about as a product team and make sure it gets delivered alongside the project or shortly after?

Yeah, absolutely.

This is something that's near and dear to my heart because almost nine years ago now, I started with Acquia when it was like a rocket ship startup.

So we grew so fast.

And as we grew and acquired new companies and built new products and everything, there was just new features, new functionality, new needs coming at us left and right.

And I started on the customer support team. I was on the front lines having to diagnose issues, having to solve problems, get configuration changes scheduled.

And we had to build, it was almost like frontier automation. We had to build a lot of different tools and come up with new systems and processes for making sure that we could diagnose anything, we could configure anything, we could do it quickly, get a customer solution fast, get them their answers.

So like I said, it's a topic near and dear to my heart.

And it's something that we keep in mind now, years later, on any project we build.

We have this thing called the definition of done. I'm sure it's not unique to Acquia.

But our definition of done includes, have you documented it?

Is it providing value to customers? Has it increased the burden on any downstream team internally?

Or have they been enabled? Have they been trained? Are their systems updated?

So that's something that we had in mind early on. Now, complexity comes in when the implementation of a project changes over time.

And that's one of the issues we faced with this, is the final implementation is going to vary from what we originally had envisioned.

So we have to work with, now late into the game, our automation teams to make sure that they can adapt to the new methodologies that are different than what we had originally planned.

So we're working with them actively right now to make sure, okay, can you pull this data?

Can you do this? Can you do that? And in the interim, again, we're trying to be good neighbors to our customer success teams.

So we have to provide them with documentation and insights and different analytics that say, look, here's how you get this information now.

And once the solution comes out, here's how your life will change.

And just making sure that they feel comfortable, updated every step of the way, they know the timelines, they know the caveats, that they're informed and enabled.

One of the things that I found interesting at Cloudflare, I've been here a little over five years, is that when I started on the product team, we did sort of, it was product and engineering for the most part, right?

And we did, I apologize to all the engineers I've worked with historically on the UI side, mock-ups ourselves.

I was firing up Allsomic and kind of going crazy there.

And then we started this product design team, and then we started a product content experience team.

And so some of the responsibilities that a smaller company would fall on product management, you bring in the experts, right?

And you're doing, you're specializing in those functions.

Do you do something similar at Acquia?

Do you have teams you work with on the documentation side or from a front-end design side?

How do you think about the auxiliary functions related to product management?

Yeah. My philosophy is bring people in early and often. So, let the docs team know in advance that a thing is coming.

And maybe it's a quick win, it's something they can knock out quickly, or maybe it isn't, but let them know in advance and that way they can help shape the delivery of the solution.

There are times, as a product manager, you're a person, you don't have a full view of everything that's going to be impacted.

You want to have a full view, but you can't.

So, bringing in the docs team early, I found, helps us identify things that maybe we didn't think of, which they'll be like, hey, have you thought about this document over here in a corner that recommends X that now won't be possible?

Oh, I didn't even know that was the thing. So, now you account for that and it either gets pulled into your requirements or it gets pulled into your project plan.

For downstream automation teams, again, it's we bring them in early and often, make sure they understand how their lives are going to change and how the downstream lives are going to change and make sure that they can plan their roadmap accordingly because they're automation teams, they're slightly outside of product.

And then on design, what we've learned over the years is you can't build an RFC and then go to the UX team and say, hey, can you design this?

One, they don't like that.

Two, but you end up kind of missing the point. If you're not doing UX design and research up front, then you're missing the requirements that will dictate the implementation details.

So, even though this specific feature didn't require many UX changes at all, involve them early, let them validate that it doesn't change the user experience.

Otherwise, you end up in a predicament, which I've seen in the past, where you design a solution, you're getting ready to roll it out, the UX team looks at it and says, this is completely untenable.

You didn't account for X, Y, and Z.

There's APIs you just didn't know that you needed to power the front end.

I think that's a really great point because even when I was doing the designs myself, I would do those early as part of the product requirements document because I always found having and now partnering with the design team early on, partnering with them, you get a sense of, okay, what does the experience look like for the customer?

That can help you scope what you actually have to build in the back end.

I think you mentioned stuff that might be missing, but in some of these cases, if you're not going to expose something to a customer, we have this rule that everything in the UI is calling an API that is the same that customers would call directly, you might not actually need to build all of that stuff.

I think that's a good segue into my question around milestones.

How do you think about what you build?

I want to go back to the project that you were talking about specifically.

You're in this project now. Did you break up the deliverables and try to say, okay, we have to hit this date that's six months from now across eight teams.

Obviously, there's an end state you're trying to work to, but do you think about breaking up for interim deliverables or do you put stuff in front of customers as you go?

How do you think about that? It's always a trade-off. Any release of customer-facing functionality comes with a degree of additional steps required.

There's enablement, there's documentation, the automation, everything.

When we were originally envisioning this rollout, we were saying, all right, it's actually two features in one.

We've got to make self -service SSL certificates available to a certain segment of the customer base that are more complex.

We also have to enable the concept of multi-certificates.

We were originally going to just merge those into one project and release them all at once.

Then we found a second use case unrelated to those two, but unblocked by them.

They were like, you know what? We can actually release three features at once.

That was our original plan, but then as we got closer to the original release date, we realized, oh, there's a little complexity on use case A, a little unexpected complexity on B, but C is still looking good.

At that point, we pivoted and said, let's try rolling this out in phases, in milestones.

We did it and it actually worked really well. We learned some by using the methodology of incremental rollout to customers.

We said, all right, roll this out to 1,000.

We see something that we need to address on one or two. All right, let's adjust the code.

Yep. Okay, good. Then you roll it out to 1,000 more and so on and so forth.

That's allowed us to learn a lot of lessons like, okay, there's this one edge case that 12 customers use.

Then there's this one edge case that three customers use.

You build that into the logic so that when you do finally release everything, it's nice and polished and there's no impact to the larger organization.

Yeah, that's really smart. I think we at Cloudflare spent a long time trying to architect a gradual rollout system.

We can send stuff to specific pops of the 200 plus pops or we can send stuff to specific customer bases.

Then you always have the early adopters that raise their hand and say, I don't care if it barely works.

I want to test it and they're super valuable, I find. Have you found with specific customers that you gravitate to one that gives you really precise, honest feedback and helps you develop the product?

I think we call them in some cases like founding customers for certain products.

Well, we're lucky that pretty much any customer we reach out to is always ready to provide ample feedback.

We're lucky.

Pretty much any account we have that has a technical account manager, we have weekly touch points with them or bi-weekly touch points.

Depending on the feature we need to release, we can reach out to our TAM organization and they say, you know what, this customer is a good fit.

Because they already have touch points with those accounts, we can hop on a call with them and our customers are awesome.

They love our products and they're like, here are all the things you could do to make our dreams come true even more.

They're great with their feedback like that.

They're almost always willing to just jump in and like, yeah, let me test that.

Let me try this early. We're very fortunate in that regard. That's great.

Chico, we're actually running up on time here. This was really interesting. I feel like we could have gone double the length.

Maybe next week, I'll do an hour instead.

Thank you so much for your insights. I think those listening at home interested in product management, hopefully learn some stuff and really appreciate your time and your partnership, Acquia and Koffler.

Thanks again. Absolutely.

Thank you. Take care, everyone.