For anyone who has considered testing, this should sound familiar: “What should I test?” Congratulations, you’ve arrived at major strategic cross-roads for your company.
Essentially deciding what to test means that you’re choosing between RESULTS or KNOWLEDGE.
Most people would say that they would choose results, right? Results are ALWAYS the short-term and long-term goal, no matter how you define it.
RESULTS seems like the obvious answer, until you realize that there is an enormous risk as well. Let me explain. With testing, the RESULTS options usually involves designing a completely new campaign or experience. It’s ‘swinging for the fence’, throwing every new strategy, relying on your intuition and experience that the new strategy is superior. When you do this, you’re naturally incorporating dozens of new elements – new layout, new copy, new eyepath, new marketing strategy. It’s not clear which of these new elements (if any) are drivers of response…it’s all a guess, but it could be a huge win.
But, what people don’t realize that that there are two issues with this: 1) if it is an big win against your control, it’s hard to know what the next step should be in your test plan, because you have no idea which element was the driver of response. 2) If it’s a big loss, you also have no idea which element was the driver. You may have incorporated 9 awesome new elements, but one really terrible element that totally messes up the test. At that point, you usually need to revert to the control and start over. So, in short, the RESULTS option is simply really risky. It’s often a great option, as long as you’re willing to stomach the chance that it might not work.
KNOWLEDGE is the less risky alternative. This is the scenario that is more disciplined, slower and more careful. In this testing case, you only change one or two elements at a time, therefore isolating each driver of response. The great part is that you can prove over time what works and what doesn’t. Once you have information, you can back up all of your decisions, gain better customer intelligence and limit downside risk for the company. It’s a great option if you have lots of time and traffic to spare.
In fact, the big risk (outside of time and traffic) is that you’ll have more inconclusive tests, thereby wasting time, resources and incurring opportunity costs. This can happen more easily by simply choosing elements that have probably have no meaningful impact on your goal (like seriously, when have we ever made an important purchase when the Buy Now button was Red rather than Blue?)
We are faced with this question all the time for our clients. We try to apply both approaches wherever we can. For clients that are just getting started, we like to recommend a 3-Cell test : the control, a brand new (RESULTS) option, and an element test off the control (KNOWLEDGE). If there’s not enough traffic for a 3-Cell test, we usually start with the RESULTS option to get a quick win, and then build in element tests from that point forward (KNOWLEDGE).