Growth: thoughts on metrics

Data-driven

Instead of excessively collecting metrics, you should follow a data-driven approach. When you manage a product in a data-driven way, you run experiments that generate data that feed your decisions for the next experiments. It is this simple, and it seems to be an excellent way to manage a product. But there are a few problems with this approach.

  • Easy versus most relevant: There’s a natural tendency to measure what is easy to measure, and that can happen to the detriment of measuring what is most relevant. For instance, what is easier to measure: clicks in a message from an e-mail marketing campaign or the user perception (curiosity, happiness, disdain, etc.) when getting this message from the e-mail marketing campaign? Before taking action over the existent data, it is always good to ask if these are the best data for making the decision.
  • Local optimization: Another concern in relation to decisions based on data is the hazard of sticking to the improvements focused on local optimization. That is, you make incremental improvements in your product focusing on improving a given metric, but you don’t realize that if you make a more radical change you can get a considerable increase in this metric, larger than the largest increase obtained from these incremental improvements.
  • Quality of data: It is necessary to understand the quality of data, that is, how they are obtained and processed before you get them into the analysis. Are the results statistically relevant? You must have heard of A/B tests, right? It is a test in which you build two versions of something. Next, you set apart the traffic in these two versions and start to measure the behavior on the two versions according to a given goal, such as clicks on the Buy button. After running this test for a while, you will know which of the two versions reaches the goal more often. Now try to run an A/A test, that is, an A/B test in which the two versions of the page are identical. After running the test, check out the results. Chances are you will find one of the A versions of your page reaching the goal more times than the other one. How is that possible? Statistical variations, seasonality, and unpredictable effects. Every data can be subject to statistical variations and that is why, in quantitative data, it is always good to understand the size of the sample and how it was collected. The season when the data sample was taken also affects the data quality: time of the day, day of the week, day of the month, period of the year, and so on. Data are photography with a date and time set, and the same datum collected minutes later can give you different information.
  • Quality acknowledgment: Not only from quantitative data lives the product management. Au contraire, the product manager must use different tools to learn more about the user, the problem afflicting this user, the context in which this happens, and what motivates having this problem solved. Many of these tools present qualitative data, that is, exploratory research data, through which it is seeking to understand reasons, motivations, and opinions. This kind of understanding is very difficult, if not impossible, to obtain in quantitative data analysis.

Data-informed

Instead of data-driven, we need to be data-informed, in other words, using data as one more input to decision-making, not the only input. Take the experience, the intuition, the judgment, and the qualitative information into consideration, along with the metrics to increase the quality of decision-making.

A little bit on A/B tests

There are two great tools for taking A/B tests — the Visual Website Optimizer and the Optimizely. I used the services from Visual Website Optimizer, which gives you one free month to test some hypotheses about the home of ContaCal:

Analysis paralysis

Lastly, aside from these precautions, it is necessary to take care with the analysis paralysis effect, that is, analyzing data all the time and not taking any action. As seen in the picture from xkcd.com (2014), analysis paralysis can cost you a lot:

Final thoughts

The metrics are not the reason for the existence of your product. The users, their problems, and the strategic goals of the business are the reasons for its existence. In other words, metrics are a mean and not the end, the results, and the goal. Therefore, use them more like one of the tools to help you to drive your product in the right direction.

Product leadership coaching

I’ve been helping several product leaders (CPOs, heads of product, CTOs, CEOs, and tech founders) extract more value and results from their digital products. Check here how I can help you and your company.

Digital Product Management Books

Do you work with digital products? Do you want to know more about how to manage a digital product to increase its chances of success, solve its user’s problems and achieve the company objectives? Check out my Digital Product Management bundle with my 3 books where I share what I learned during my 30+ years of experience in creating and managing digital products:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Joca Torres

Joca Torres

2.5K Followers

Digital product development advisor, coach, and board member. Also an open water swimmer and ukulelist.