We have updated our Privacy Policy and Privacy Options

Got It

10 Considerations for Choosing a Testing Tool


With dozens of testing tools on the market—all offering different sets of features, at different prices—it seems the biggest hurdle to starting a testing program is finding the right platform. Though the decision is always going to be a tough one, these 10 considerations can help guide the process.

1. Cost

Let’s face it: The first concern when choosing any product or service is the cost. It’s no different when it comes to choosing a testing tool. The challenge is finding a tool that fits your needs and your budget. And it’s not always as simple as comparing posted rates.

Every testing tool company seems to price their products differently. Some offer monthly rates, others annual. Some charge by site traffic, others by the number of tests run. Deciding which fits your budget depends on your goals and situation just as much as the amount you’re willing to spend.

It is also common to offer multiple pricing tiers—each boasting progressively more features. Though the base price of one tool may initially appear cheaper, upgrading to a tier that offers the functionality you need could put it on par with competitors.

2. Development Requirements

Along with cost comes the development requirement. Some tools offer impressive power—but this power typically requires the skills of knowledgeable programmers and an IT department with the time and motivation to collaborate.

Other tools offer easy implementation and sometimes even minimal development services. This is tempting for small organizations, new testing programs, or marketers without IT support. However, the easy implementation sometimes sacrifices customizability, performance, and more advanced features.

3. Customizability

Some tools make it very easy to design and launch simple tests, but more complex variations can require extensive coding or may not even be possible. Customization—or a tool’s ability to work with your site and support your testing goals—is an important consideration.

The caveat, of course, is that with greater customization comes an increased need for development, IT, and creative support—even from the most robust packages.

4. Ease of Use

Once a tool is purchased and implemented, the fun can begin. The fun, in this case, is testing: Creating variations, measuring their impact, and learning about your customers. But if the tool is too complex, this stage becomes more of a chore than a joy.

Ease of use is an essential consideration, especially for those new to testing. As you and your team become more comfortable and experienced, you will no doubt need a robust testing platform that is infinitely customizable and can deploy any type of test imaginable.

But when you’re just starting out, it’s important to achieve a basic threshold of test velocity. This is where learning happens and the set up and deployment of tests needs to be easy enough that your tool facilitates this discovery instead of inhibits it.

5. Versatility

When it comes to testing tool versatility, the big dichotomy is between A/B/n and multivariate testing. Most people focus on—and certainly start with—A/B/n tests, in which visitors are served one page or another. The pages can have lots of variations, but only one of the defined combinations can be served. In multivariate testing, pages are dynamically constructed and can offer a more complex combination of variations.

However, these are only two of the many testing options available. Other testing methods include full or fractional factorial tests, segment tests, and behaviorally targeted tests. Some testing tools offer display ad testing, email testing, video testing, mobile testing, and more. It is important to ensure your testing tool can accommodate the methodologies and platforms you hope to work with.

6. Targeting and Segmentation

Targeting and segmentation are related to a tool’s versatility, but they are such important features that they deserve a discussion of their own.

Targeting, specifically behavioral targeting is a feature that analyzes visitor behavior—things like referrer source, time on page, history, etc.—and dynamically serves test variations in response. This allows you to gain specific insights into customer behavior, needs, and motivation.

Segmentation is the ability to divide visitors by specific characteristics. Typically, this includes things like geographic region, time of day, or other qualities.

These features work together and, though admittedly advanced, become increasingly important as testing programs develop.

7. Reporting

Any testing tool worth considering will supply reports of test performance and the final results. How easy it is to understand these reports—and whether they fit into your current analytics reporting system—will vary from tool to tool. Moreover, there is often variability in the amount of data that is stored and the form it in which it is presented.

Other things to consider are whether the tool allows for the creation of custom reports, whether data can be downloaded as a spreadsheet or PDF, and how the performance of variations is tracked.

8. Integration with Other Tools

Closely linked with reporting is another important consideration: Whether a testing tool can work with other related platforms already in place. Usually, users want to integrate their testing tools with existing analytics packages—allowing them to combine data into comprehensive reports.

However, there are lots of other combinations to consider. Whether a testing tool can work with your CMS, email provider, an e-commerce platform, or shopping cart plugin, for example, could be essential for your testing program.

These integrations usually take place through an API, which may require the skills of a developer to implement.

9. Support and Service

With any technology product, support is an important consideration. The cost and complexity of a testing tool means that support is even more essential. As with other features and services, different vendors offer different levels and types of support. Some include extensive, personalized training, strategy consultations, developer services, and more—though often at an additional cost.

For organizations with limited IT or developer support in house, however, the extra cost of extensive, hands-on support could be a wise investment.

In addition to support, access to services should also be given some thought. Whether using a third party service like Brooks Bell, or a service package bundled with your tool, having a team of experts available to execute your testing program can be the difference between losing an investment and making a profit from a testing platform. This is particularly true when adopting testing for the first time.

10. Unique Requirements

Of course, every organization is different and as such, their requirements for a testing tool will differ. Things to consider here include site architectures, internal support and politics, reporting structures, and goals for the testing program. Many testing tool vendors have a personalized sales process during which any specific issues, concerns, and requirements can be discussed.

Choosing a testing tool is a big decision but by carefully weighing your organization’s requirements, resources, and goals the long list of candidates will quickly become manageable.

The 10 considerations above will help you find the right tool for your organization and don’t forget to download the 20 Questions to Ask When Choosing a Testing Tool, too!