When you make the jump to multivariate testing (MVT), there are suddenly a lot of new concepts to sort out. Will you use a full factorial design, which can produce dozens, if not hundreds, of variations? Or will you use a partial-factorial approach like the Taguchi method to influence a more efficient experimental design? It sounds complicated—and it can be—but that doesn’t necessarily mean it’s better.
Multivariate testing is a tool—like any other method. And just like a hammer, there are some applications in which it excels, like pounding nails, and some in which it is completely inappropriate, like driving screws.
When comparing MVT to A/B testing, there is another helpful analogy. Imagine a great sculptor, like Michelangelo, creating a statue of a human form. He begins by removing large chunks of marble, carving the block into a rough outline of the human shape. Then, using a set of medium chisels, he refines this outline to a more representative form. Only after this initial work has been done, can the sculptor use the finest chisels to shape each detail with precision.
This process illustrates the relationship between MVT and A/B testing. The latter helps marketers and analysts define broad segments, uncover fundamental optimization strategies, and shape the roadmap for large-scale optimization efforts. Once the ideal outline for a page has been defined through A/B testing, MVT can be used to establish the best combination of fine details—button shapes and colors, minor shifts in element placement, small tweaks to messaging, and so on.
There is another important distinction between MVT and A/B methodologies: The potential for learning from each test. When executed properly, A/B testing evaluates a specific hypothesis, providing a deeper understanding of user behavior and psychology whether variations win or lose. Often, these insights can inspire tests elsewhere on the site—and even marketing or promotional campaigns. An MVT test using the Taguchi method, on the other hand, provides insight into the interaction effect between variations in multiple parts of a site, helping you understand which changes are most effective when used in combination.
So, how do you know if it’s time to use multivariate testing? Here are a few guidelines:
1. You have some understanding of your users
MVT provides an excellent opportunity to compare the performance of individual elements, but it doesn’t reveal as much about the needs, desires, or psychology of your users. Designing variations that target specific psychometrics, then comparing their performance through A/B testing is the best way to gain these insights.
2. You’re confident the design is effective
With MVT, it’s difficult—if not impossible—to test dramatically different approaches to a page design. Before starting an MVT, then, it’s important to have some data establishing the page is a reasonable foundation for granular optimization.
3. You have substantial traffic to the page
With dozens and even hundreds of variations, MVT requires a substantial amount of traffic to reach confidence within a reasonable period of time. For most sites, this means few pages past the homepage will be suitable candidates.
Multivariate testing is a powerful optimization tool. But, like any tool, it has strengths and limitations. It’s important not to rush into MVT just because it seems more complex or offers more simultaneous variations. Instead, develop a few sound A/B test ideas, prioritize your queue, and gain some learnings and wins before jumping in.