We have updated our Privacy Policy and Privacy Options

Got It

The Parallels of March Madness and A/B Split Testing


Living in the Triangle area of North Carolina with respected collegiate athletic programs in schools like Duke, UNC and NC State (I’m a Wolfpack Girl), I can’t help but draw parallels between testing and March Madness at Brooks Bell. Our testing behavior is oddly similar to our March Madness behavior. There’s office chatter, favored challengers, constant stat checks and (I’m not going to lie) a little smack talk. And then it struck me – many of the principles of testing and The Big Dance are the same.

Survival of the fittest

March Madness is optimization at its best. With every game, the losing team is eliminated and the winner goes on to the next round to play another day.

In testing, the winning challenger becomes the new control experience – thus moving to the next round. From there a new challenger is chosen, designed to expose any possible weaknesses of the former winner. And the testing cycle continues, iteration upon iteration.

Good matchups are key

It’s the reason that we’ve coined phrases like Sweet 16, Elite Eight and the Final Four. There’s excitement as the playing field gets smaller and smaller. The elimination of the weaker teams ensures a great matchup for the championship game. The likelihood of an underperforming team making it all the way is slim. This means that the final challengers are on an equal playing field, which will hopefully give fans the heart-pounding, nail-biting excitement they crave.

In testing, challengers with only small element variations make for the best, cleanest tests with the highest learnings. By isolating the differences between challengers, you can identify their strengths and weaknesses. Wins can be attributed to a specific element to use going forward, and the losses – even though they are losses – provide great learnings.

Upsets happen

Though there are assumptions about what team will win, you never really know until the game is over. In basketball and testing, upsets are a reality. They make it interesting and they teach us that assumptions are merely an educated guess.

This leads into the next principle…

There are come-from-behind victories.

In the words of Animal House’s Bluto, “It’s not over until we decide it is.”   Though a test or game is shaping up one way, it could take a turn.

In testing, it’s imperative to let your conversion data normalize. Ensure that you have enough data points and that the data has reached confidence. Calling a winner too soon is an understandable, common mistake made by eager marketers. However, it is risky business to make big decisions based upon data that has not reached confidence levels. Adobe® Test&Target™ provides a confidence indicator, which is a very helpful tool.

Please note that the confidence calculation does not consider the number of data points. For example, if there are a total of 10 conversions in 10 visits, it will display 100% confidence. However, you need more than 10 data points to truly be confident in the result. Marketers just need to keep the number of data points in mind.

And finally,

There’s one winner.

The NCAA Champion is the team that has the highest number on the scoreboard when the final buzzer sounds. Testing isn’t as cut and dry.

In testing, it’s important to know your key performance indicators (KPI) before the start of any test. It’s also important for a marketer to determine if you are measuring conversion rate, average order value (AOV), revenue per visit (RPV) or sales.

For example, a challenger that offers a discount will often have a higher conversion rate but a lower AOV and RPV. It’s important to know your KPI before beginning a test so that you are monitoring the correct data—and so you know who the clear winner is.

As you can see, the principles of the NCAA Tournament and online testing are very similar. Add in the excitement, assumptions and smack talk and you’ve got Testing Madness – our favorite season at Brooks Bell.

Now, if only I had written this post last week when my Wolfpack was still playing…