We have updated our Privacy Policy and Privacy Options

Got It

When is a Complicated, Difficult-to-Execute Test Worth the Effort?


Sometimes, during a brainstorming session, an incredible test idea emerges and the developer or analyst in the room sighs deeply. It’s not that he or she feels the idea isn’t great—rather, the thought of having to develop a complex variation at the edge of feasibility or analyze a complicated combination of variations has them thinking about available hours and resources. Then, there’s the project manager who remembers the last complex test that was developed, launched, and returned a flat result.

Indeed, the level of effort required is an important consideration when prioritizing test ideas—but a high degree of expected effort is not a good reason to disqualify an idea completely. Of course, investing more time and energy into a test, of course, can’t guarantee a big win—and it’s not necessary for achieving learning, either. As a result, it’s important to consider the potential benefits carefully. Here are five reasons it might be worth investing extra time and resources into a test idea: Tweet_this

1. It Will Mitigate the Risk of a Larger Decision

Imagine a retailer considering a new recommendation engine. Such a move requires a huge investment—and represents a sizable switching cost—both in terms of development time and the actual expense of the tool or service. There are a lot of factors to consider when making such a decision, of course, and testing can’t address all of them. However, a test, albeit a complex one, could reduce uncertainty around one aspect—whether any of the options under consideration will actually increase the performance of a website. The effort will be high, but by testing several recommendation engines against one another, the optimization team will be able to help make the decision more data-informed.

2. It Could Generate Unique Insights

When preparing a whole-site redesign, most organizations choose to launch the update all at once and hope for the best. Not only is this approach not necessary, it’s usually not a good idea. Though there are many ways to stay data-driven during a redesign, testing lots of ideas that may never make the final concepts can be a frustrating misapplication of resources. One alternative requires much more effort, but can produce a useful baseline for a redesign.

Conducting a multivariate test to evaluate each element of the redesign at once, for example, requires a lot of development and would be time consuming to analyze. It will provide a granular understanding of how users are interacting with each new element of the site and—perhaps more importantly—the impact these elements have in combination with one another.

3. It Will Make Future Testing or Implementation Easier

One of the biggest struggles successful testing teams face with the organization as a whole, is getting their winning variations implemented into the base code of the site. Sometimes, the test design itself can have an impact on whether a winning variation will be implemented or not.

Though it often adds to the time and effort required, building the variation to the specifications required by production, or executing a QA rigorous enough to match the baseline requirements for implementation ensures the full value of winners will be realized. Getting variations implemented also makes testing in the future easier, since developers will no longer have to worry about maintaining an ever-expanding codebase within the testing tool.

4. It Will Dramatically Increase Support for Testing

In some organizations, testing is valued from the highest levels. In most organizations, however, support is more limited. Even when the testing program enjoys support of an executive sponsor, other groups may feel threatened or remain skeptical. Involving these skeptics in the process is a great way to build support.

If a good idea comes out of such a collaboration, it may be a worth pursuing it even if the effort will be greater than usual. Doing so will require extra resources, but solidifying the relationship will contribute to the long-term health and influence of the testing program.

5. It Could Create Huge Monetary Value

Understanding the potential impact of a test is essential for effective test prioritization. Using a conversion value calculation—one that considers total traffic volume and conversion rate—will allow you to compare pages and ideas on a similar scale. When expected or potential impact is high, it’s obviously worth the extra effort development, QA, or analysis may require. It’s important, however, that the test is still designed to produce clear learning, not just value, in case performance does not improve as dramatically as anticipated.

Maintaining velocity is important for the momentum of a testing program. But sometimes, taking a little more time and drawing on more resources to execute a test that requires a high level of effort is worthwhile. There’s risk with such tests, of course, but by weighing the value of the result—especially when it contributes in one of the ways described above—can help you determine whether the effort is justified.

Brooks Bell helps top brands profit from A/B testing, through end-to-end testing, personalization, and optimization services. We work with clients to effectively leverage data, creating a better understanding of customer segments and leading to more relevant digital customer experiences while maximizing ROI for optimization programs. Find out more about our services.