When hiring product managers, product designers and directors in both areas, the majority of the time I get asked, “How will you measure my success?” A fair question. And I always answer: “By looking at success of your product.” But how do you actually measure success of the product?
Measuring product success
It really comes down to understanding the long-term health metrics of the product itself and the ability to measure the success of individual releases. When you dive into product management books, you’ll always read that bad companies celebrate releases and good companies celebrate outcomes. But, if your company and way of working falls into this first category and you’ve become a feature factory, how do you move towards becoming an outcome-based product organization?
It all starts with understanding why you are developing your product. What’s the problem you’re trying to solve? You can talk to customers and hear them saying they have a problem, but for every problem that you’re trying a solve, there’s behavior you’ll be able to measure.
In my previous blog, you can see that reporting is hard to use, because people run it multiple times and it takes them quite a lot of time to set up their filters. That’s clearly a bad behavior and that’s clearly something you want to change. You need to be able to measure the change in behavior.
Why is releasing a change in your product not enough? If you just release and you consider your job done, it’s almost as though you’re arrogant – you’re convinced that you’re providing the right solution to the problem, and you trust your gut so much you don’t even bother measuring the effect of the release. That’s not good enough.
Measure behavior, not adoption
If you measure activation, that’s a bit better. That at least guarantees that someone tried using what your team has just released. But again, did it solve the problem? The fact that someone tried clicking a new button once doesn’t mean that their overall behavior has changed.
If you measure adoption, it’s once again a step closer. But what adoption really means is that users are using your new release repeatedly. But is it solving the problem you wanted to solve, or is it solving a different problem? If it’s helping with the right problem, is it helping enough?
You don’t really know until you measure the change in the behavior of your users. If they ran the report five times before seeing the right numbers, are they now running it only a single time? Or are they still running it multiple times? This is your only success metric. You’ll need several releases to get there, and that’s fine. But you really need to solve the problem and change how your user behaves.
Understanding the overall impact of your product onto the user and their business is then only an aggregation of all your releases. But you need to understand each and every single individual release. If you don’t, no one else does. They might feel it, but… what’s the price you can ask them to pay for a feeling? You need to be able to prove your impact, and measuring tangible success is the only way how.
