Growth: thoughts on metrics

Measuring is good. Metrics are essential for knowing deeply your product and for making more accurate decisions. But you must be very careful. The excess of metrics can get in your way and overshadow the full knowledge of your product.

Data-driven

Instead of excessively collecting metrics, you should follow a data-driven approach. When you manage a product in a data-driven way, you run experiments that generate data that feed your decisions for the next experiments. It is this simple, and it seems to be an excellent way to manage a product. But there are a few problems with this approach.

• Easy versus most relevant: There’s a natural tendency to measure what is easy to measure, and that can happen to the detriment of measuring what is most relevant. For instance, what is easier to measure: clicks in a message from an e-mail marketing campaign or the user perception (curiosity, happiness, disdain, etc.) when getting this message from the e-mail marketing campaign? Before taking action over the existent data, it is always good to ask if these are the best data for making the decision.
• Local optimization: Another concern in relation to decisions based on data is the hazard of sticking to the improvements focused on local optimization. That is, you make incremental improvements in your product focusing on improving a given metric, but you don’t realize that if you make a more radical change you can get a considerable increase in this metric, larger than the largest increase obtained from these incremental improvements.

• Quality of data: It is necessary to understand the quality of data, that is, how they are obtained and processed before you get them into the analysis. Are the results statistically relevant? You must have heard of A/B tests, right? It is a test in which you build two versions of something. Next, you set apart the traffic in these two versions and start to measure the behavior on the two versions according to a given goal, such as clicks on the Buy button. After running this test for a while, you will know which of the two versions reaches the goal more often. Now try to run an A/A test, that is, an A/B test in which the two versions of the page are identical. After running the test, check out the results. Chances are you will find one of the A versions of your page reaching the goal more times than the other one. How is that possible? Statistical variations, seasonality, and unpredictable effects. Every data can be subject to statistical variations and that is why, in quantitative data, it is always good to understand the size of the sample and how it was collected. The season when the data sample was taken also affects the data quality: time of the day, day of the week, day of the month, period of the year, and so on. Data are photography with a date and time set, and the same datum collected minutes later can give you different information.
• Quality acknowledgment: Not only from quantitative data lives the product management. Au contraire, the product manager must use different tools to learn more about the user, the problem afflicting this user, the context in which this happens, and what motivates having this problem solved. Many of these tools present qualitative data, that is, exploratory research data, through which it is seeking to understand reasons, motivations, and opinions. This kind of understanding is very difficult, if not impossible, to obtain in quantitative data analysis.

Data-informed

Instead of data-driven, we need to be data-informed, in other words, using data as one more input to decision-making, not the only input. Take the experience, the intuition, the judgment, and the qualitative information into consideration, along with the metrics to increase the quality of decision-making.

One of the best examples I can quote has to do with the website hosting product from Locaweb. Through the years, in a reasonably informed way, but always counting on a lot of intuition, we altered our hosting plans in order to have more space on disk, data transfer, and the number of sites that could be hosted in each plan sold. In 2011 we noticed that more than 90% of our clients were choosing the basic plan because it attended to the needs of most people who needed a website.

We wanted that the largest plans would play a bigger role in sales, but with the limits we had, there was no motivation for the clients to buy them. We thought of changing the plans for new subscriptions, decreasing the limits to incentive clients to get bigger plans. However, as this was a significant and very sensible change, we brought in a consulting expert in pricing that helped us collect and analyze several data, to suggest which was the best chance to be done.

We implemented the changes suggested in 2012. There was a little variation in the distribution of plans, but the number of plans contracted per month didn’t change. Moreover, it even decreased a little, which resulted in no alteration in the amount of monthly revenue. That is, we spent time and money collecting and analyzing data that made us take a decision that didn’t change the company’s result. Maybe if we had defined the changes more intuitively, we would have saved money and would know the result of the decision more quickly.

A little bit on A/B tests

There are two great tools for taking A/B tests — the Visual Website Optimizer and the Optimizely. I used the services from Visual Website Optimizer, which gives you one free month to test some hypotheses about the home of ContaCal:

In less than 30 minutes, I was able to create 4 versions and began to run the test. I decided to test two things. One was for the color of the create account button if it was going to make a difference in the number of people who would click on it:

The second test: if changing the explanatory video for a photo would increase or decrease the number of people who would click on create account button:

The result was:

After seeing this result, I got the impression that if I put the green button with the healthy family picture, I was going to increase the conversion even more. So, I decided to run this test and the result was:

Therefore, take care, appearances can be deceiving! Make experiments before taking conclusions!

Analysis paralysis

Lastly, aside from these precautions, it is necessary to take care with the analysis paralysis effect, that is, analyzing data all the time and not taking any action. As seen in the picture from xkcd.com (2014), analysis paralysis can cost you a lot:

Final thoughts

The metrics are not the reason for the existence of your product. The users, their problems, and the strategic goals of the business are the reasons for its existence. In other words, metrics are a mean and not the end, the results, and the goal. Therefore, use them more like one of the tools to help you to drive your product in the right direction.

With this, we close the theme on the growth stage of a software product life cycle. We understood how to deal with client feedback, what it is, and how to prioritize a roadmap. We also saw several types of metrics, including the conversion funnel, engagement, churn, global and individual financial metrics, revenue and negative churn, NPS, the loyalty metrics, and we also approached some considerations on metrics.

In the next chapter, we will better understand the next stage of a software product life cycle: maturity.

Product leadership coaching

I’ve been helping several product leaders (CPOs, heads of product, CTOs, CEOs, and tech founders) extract more value and results from their digital products. Check here how I can help you and your company.

Digital Product Management Books

Do you work with digital products? Do you want to know more about how to manage a digital product to increase its chances of success, solve its user’s problems and achieve the company objectives? Check out my Digital Product Management bundle with my 3 books where I share what I learned during my 30+ years of experience in creating and managing digital products:

• Startup Guide: How startups and established companies can create profitable digital products
• Product Management: How to increase the chances of success of your digital product
• Leading Product Development: The art and science of managing product teams

You can also acquire the books individually, by clicking on their titles above.