DATA-DRIVEN CMO is an ongoing series on the Brooks Bell Blog that focuses on topics for the modern day data-driven CMO.
“Flat” results are simply a matter of perspective. And by “perspective,” I don’t mean taking a different look at the same result, but rather digging deeper to find a metric that was influenced.
That is a statement that I have been mulling over for many months. In a world of rapid, iterative testing, and clearly defined conversion metrics, it’s not uncommon to hear the phrase, “The test results were flat” before proceeding on to the next test idea. As if a flat result meant the idea was inconclusive and not meant to explored further.
In our world of clearly defined success metrics that our strategic tests try to lift, the term “flat” is fine. And although it may be “fine,” as marketers we owe it to ourselves to believe that flat is not enough.
Prior to Pythagoras’ theory that the earth was in fact spherical, not flat, people lived comfortably through their day thinking the Earth was flat. But it doesn’t mean they had a fuller understanding of the Earth as a whole—only of their immediate surroundings.
That is the same trap we fall into by judging the success of our tests strictly by the metric we established for the test. In many cases, when you dig deeper into the data of a flat test, you can discover many points of differentiation. This can be simply in a conversion point at a different part of the funnel—or in more advanced segmentation examples, trends by demographic.
This type of analysis can sometimes take more time than we are comfortable with. I am not advocating slowing down your test velocity to run your analysis. What I am suggesting is to continue to dig through that data until you find a point of differentiation. That piece of learning could pay off in the future.