Metrics are the foundation of testing, but determining what to measure is one our most complex tasks. Before you can analyze the numbers and apply what you’ve learned, you must first determine the metrics that matter most.
At Brooks Bell, a cross-functional group representing all departments meets regularly to share knowledge and explore issues in testing. Recently, we discussed secondary metrics and how they matter. Here, we’re sharing each group’s distinct perspective on the issue:
The client team’s perspective
In any organization, there are multiple teams that have varying goals and interests. SEM focuses on clicks, creative focuses on imagery, and marketing focuses on messaging.
Despite these competing interests, there is always an overarching (and hopefully unified) goal. In testing, we call this single goal the primary key performance indicator, or KPI.
Common KPIs include:
- Average order value
- Revenue per visitor
- Form completion rate
Ultimately, a KPI is a single metric. But our clients often want, rightly, to broaden their perspective beyond that metric. That’s where secondary metrics come in.
Secondary metrics add context and color. They may give us insight into visitor behavior or clues as to why a test lost. They may affirm the performance of the KPI, or they may call the KPI into question.
So how much credence should you give secondary metrics in each test?
It’s easy to become lost in the narrative when looking for the moral of the story—especially when each team has a unique perspective. The question of credence is ultimately answered by statistical rigor and certitude.
As a client team, we simultaneously seek to add as much color to our story as possible while relying on our analysts to ensure our story finds grounding in solid data.
The analytics team’s perspective
Like the client team, we believe that secondary metrics generally shouldn’t be used to determine the winner of a test, but they should help fill out the story—the “why”—and explain the user behavior behind the change in the KPI.
Using secondary metrics this way helps to continuously paint a picture of your users and ensure that every test is a learning experience, even if the KPI is not improved.
To help illustrate the importance of secondary metrics on storytelling, consider a test emphasizing a sale.
The hypothesis is that the promotion will cause more visitors to place orders, increasing the KPI of revenue per visit. But when the results come in, they show a loss in revenue per visit.
We turn to secondary metrics to help tell the “why.” The order rate shows that, as expected, more visitors placed orders. Because the sale was emphasized, the orders placed typically contained more sale items—so overall order value took a large hit.
If you looked at the KPI with blinders on, you’d think that emphasizing a sale is worse. If you look at the secondary metrics, you’ll see what is truly going on behind the scenes and how users are reacting to the difference.
With that said, as an analytics team, we do feel that there are some circumstances where secondary metrics can serve as points of decision.
A common example of this is testing a business-driven site redesign. The primary goal may be to ensure that the new experience doesn’t underperform the current experience (in other words, a flat test would be a win). If the KPI is unaffected by the new experience, it may be appropriate to consider secondary metrics to determine the better of two equivalently performing experiences.
The creative team’s perspective
An important part of any creative team is to push boundaries and come up with new, innovative ideas to pursue. Therefore, our perspective may be different from both the client and analyst teams. In our view, secondary metrics are imperative to take into account—and are often even more interesting than the KPI.
Whether it’s to more accurately understand the user’s behavior or to inspire new test hypotheses, secondary metrics are incredibly valuable. The insights provided by secondary metrics can drastically increase the quality of our testing strategies going forward.
Secondary metrics can be critically important for understanding user behavior. The way the user interacts with these metrics directly affects the KPI, especially depending on how the emphasis of these metrics is shifted.
For example, think about the obvious correlation between cart additions and orders. Cart additions indicate an intent to purchase, right? But if you look holistically at this metric, a more complex picture of user behavior emerges.
For example, we’ve seen many retail sites where cart addition rates are very high, but the order rate is similar to other sites. From this arises key questions about the user:
- Is she purely in a browsing state?
- Is she there for price comparison?
- Does she actually have intent to purchase on the site at this current time?
All of these questions are reasoned with secondary metrics, and by starting with cart additions, you can formulate hypotheses about user behavior. Ultimately, you can test these hypotheses, which can lead to a better understanding of the page and your customers.
How do you use secondary metrics to interpret test results or inform future test ideas? Share your comments below.