We have updated our Privacy Policy and Privacy Options

Got It

How Rigorous Should Your Test QA Process Be?


Before launching a test, it’s critical that the entire campaign is thoroughly reviewed and checked for errors—preferably by several different people with different expertise. But how thorough does QA really have to be?

At Brooks Bell, we use a proprietary 120-step QA process with the goal of pushing the number of busted and broken tests to zero. This is a core measure of program performance and success that supports other more commonly considered metrics like win rate, learning rate, and test velocity. Ultimately, a rigorous approach to QA helps increase the reliability of testing and, as a result, its value within an organization.

However, not every testing team has the resources to execute such an extensive QA process and for others, the goal may be something besides smoothly run tests. Because of this, there’s a need to evaluate the actual capabilities and requirements of test QA.

Small Teams Launching Lots of Tests

For small teams launching lots of tests, the resources thorough QA requires may not be available. As a result, the process must be prioritized. While most test QA tends to focus on the design and interface characteristics of each variation, errors in measurement implementation, metric firing, event tracking, page flicker, and other analytics and development issues are more common. In addition, testing programs tend to face similar problems over and over.

In response to this, limited QA should focus resources to ensure common errors are double-checked and all metrics and events are firing and tracking properly. This will never be as effective as a comprehensive QA process and as a result the program bust rate should be monitored closely and reported regularly. A significant shift in quality may demand a realignment of priorities and focus.

Programs Rapidly Increasing Test Velocity

For programs working to increase test velocity, establishing a thorough QA process is critical to ensuring program growth is effective and sustainable. Speeding the ideation, creative, and development process attracts a lot of the focus during growth phases, but if QA isn’t scaled both in terms of rigor and resources the results from more testing will become increasingly unreliable.

If this sounds like your situation, get started with our Campaign QA Checklist.

Testing Focused on Long-term Product Improvement

For some testing programs, the goal is to improve the usability and performance of a product—be it a software service, app, or the usability experience of a website—over the long-term. To do this, tests must launch frequently, build iteratively, and—perhaps most importantly—provide winning improvements that can be formally implemented.

In this scenario, the QA process should strive to at least match the baseline requirements for implementation. Often, these requirements are defined by an IT or development group which may or may not be involved in day-to-day testing. Cross-organization communication, then, is critically important to ensure any winning test variation has already cleared the basic requirements for implementation. Doing so will enable increases in velocity and lead to greater gains from each win.

Whether your testing program is big or small, brand new or mature, you need to be doing QA. Matching effort to your resources and goals helps ensure testing remains trustworthy, dependable, and scalable.