With a good testing tool—and the skills and experience to use it—creating tests should be relatively painless. Even more, creating tests should be fun. But the brainstorming process can easily get sidetracked, hijacked to serve individual or department interests, or mired in tactical testing that doesn’t provide much in the way of insights. Creating a testing roadmap is a great way to direct this process, but sometimes an idea is proposed and you just have to say “no.” And let’s face it: Saying no is hard.
But with good reasoning, it doesn’t have to be. To make the job of saying no easier—and to keep the entire process on track and efficient—it helps to have a framework in place to guide ideation. Here are five reasons to say no to a test idea:
1. It Doesn’t Relate to a KPI
Probably the most important source of inspiration for any test idea is its relevance to your Key Performance Indicators. KPIs are common guideposts for all kinds of business processes, and testing is no different. The first question to ask, then, is whether a test variation could contribute to the knowledge or performance of a KPI.
2. It Won’t Help You Learn
Another important question is: What will we learn from the test? This simple question actually addresses a few important features of a test variation. First of all, it requires that a test will actually produce insights—meaning the performance lift or fall will be attributable to a specific change from the control. Changing too many important elements at once can produce large performance improvements, but it can be impossible to isolate and repeat the improvements. Another feature of this question requires that a test will produce results relevant to the guiding hypothesis. Again, without answering this question, any lift can be very hard to repeat—and, ultimately, “the goal of testing is learning, not winning.” (Get this and other key insights from Click Summit 2013.)
3. It’s Too Difficult to Create
Of course, one of the ultimate measures of a test idea is how difficult and resource intensive it will be to create and run. Simple copy changes on a menu or page headline are one thing, but launching a multivariate test on a fully redesigned homepage is something else entirely. And often a test idea that has a weak connection to KPIs or questionable learning potential will simply not be worth the effort of implementation—even if it seems relatively straightforward.
4. It Will Take Too Long to Run
Similarly, not every page on every site is suitable for testing. For a test to reach confidence—the point at which estimates can be considered reliable—it needs visitors. If a page receives small or irregular traffic, it can take a very long time to achieve a sample capable of producing confidence and ultimately, statistical significance. A long-running test slows velocity and blocks progress. Sometimes, a test on a low-traffic portion of a site can be useful, but that low traffic is also an excellent reason to say “no” in favor of more viable ideas.
5. It Will Be Impossible to Implement
Finally, you must consider whether the variation, if successful, could even be implemented on your site. Picture an e-commerce retailer, for example, who wants to test videos that accompany product photography. It’s possible that even if adding videos proves more successful, it would be too costly and time consuming to actually implement—imagine shooting video for an entire inventory at once! It turns out that there may be some things better left unknown.
Saying no—especially to an idea—can be very difficult. But sometimes, it is essential for keeping the testing roadmap on track and your test queue flowing. By using grounded reasoning, it’s possible to reject an idea while at the same time encouraging focused ideation in the future.
Have you had to reject a test idea? Share your experience in the comments below!