We have updated our Privacy Policy and Privacy Options

Got It

A/B Split Testing: Where Everyone’s a Winner


Our society thrives on the principle that someone wins and someone loses.  It’s the focus of professional sports as well as our nightly reality TV shows.  We identify the strongest contenders, the underdogs and our favorites.  We love the drama and suspense that builds as the final seconds tick away.  We just can’t seem to get enough of it.  It’s a thirst that’s lasted the test of time and shows no signs of slowing.

And a tie?  Well, that’s no fun.  So that we can clearly define who will be dubbed “the winner” and “the loser,” we’ve created sudden death overtime, exciting shootouts and extra innings.  Why bother getting invested if nobody wins or loses?  (I wonder what they would do if there was a tie on The Bachelor / The Bachelorette?)

Testing is actually very similar.  We want to find the winner – the cell that will lead us to reaching our conversion goals.  However, in testing, flats (sort of like ties) and losses can be very valuable.

Clearly a win is the best outcome.  Winning and knowing why you won is ideal, but the truth is that many marketers are winning and not knowing why – as long as they get the big W.  Our priority is tying all results to learnings, so our tests always focus on that piece along with striving to win.

But, what’s next best?  A loss or a flat?  We had this discussion over lunch with Kurt Heinemann, the CMO of Monetate. It’s really interesting because even the Brooks Bell team is split on the answer!

First, let’s talk about losses.  From our perspective, there are two types of losses:

1. Losses with Learnings – The challenger lost, but you structured the test so that you know exactly why.  This is the best kind of loss because you can use that knowledge to create a stronger challenger cell next time.

2. Losses without Learnings – The challenger lost and you didn’t structure the test so that you could pinpoint the reason.  This is the worst kind of loss because you have no idea what went wrong.

Losing is not good for the bottom line; that’s obvious.  However, the silver lining is that great learnings can come from a loss if the test was strategized properly.  Kurt inspired me quoting Thomas Edison: “I have not failed.  I’ve just found 10,000 ways that won’t work.”  This is a perfect illustration of the impact that testing losses can have.

A losing challenger cell that was strategized with learnings in mind can tell you which levers make an impact and which don’t.  Perhaps the method that the challenger used was ineffective, but you’ve identified that it matters and that the control’s methods should be re-evaluated.  This is knowledge that is applicable throughout the testing roadmap, which is valuable information.

Now, let’s talk about flats. The equivalent of the dreaded tie. And the bad thing? There’s no OT in testing. We’re left longing to determine the winner. Some conversion experts may look at this as the worst possible outcome because a clear winner and loser didn’t emerge. Some opinions are that you’d rather lose gloriously and know why as opposed to ending a test with flat results. However, one thing was clarified that should get recognition. The flat results let you know that the element tested does not matter.

For example, you test the timing of an image carousel on the homepage and the results are flat across the board. You weren’t able to find an optimal timing solution, however, you learned that the timing does not matter to your users. This is a learning that you can reflect on when creating your testing roadmap.

So, let’s turn it over to you to decide.  What is your order of preference?

  • Win with Learning
  • Win without Learning
  • Loss with Learning
  • Loss without Learning
  • Flat with Learning
  • Flat without Learning

As mentioned before, this is something debated in our office regularly. Opinions are split on losses vs. flats. However, we all agree that anything that results in clear, definitive learning is imperative, and that wins are awesome.  With a focused, disciplined testing process you won’t walk away empty handed – win, loss or flat.

Brought to you by Brian Shampnois, Suzi Tripp, and Naoshi Yamauchi