We have updated our Privacy Policy and Privacy Options

Got It

Gorillas in the Midst: 4 Things to Look for When Your A/B Split Test Runs Flat


Our team has recently been stumped by the results of an Adobe® Test&TargetTM campaign we are running for one of our clients.  This test has been running for a while, and it showed a clear winner for the first couple of months.  Then it suddenly and inexplicably flattened out.  There is no testing-related reason for the change.  Nothing has changed with either the control or the test variations during the testing timeline.  So what’s causing the sudden change?

This testing puzzle reminded us of the Selective Attention exercise – better known as the “Invisible Gorilla” demonstration.  Take a look at this video of the exercise…we’ll wait.

Did you see it?  No?  Look again.

About halfway through the video, a gorilla walks through the frame.  Don’t worry, you’re not alone if you missed it.  Most people don’t even notice it because they’re so busy watching the ball.

Have we been so busy watching conversions that we’ve missed a glaring anomaly in our data?  Was a new email or promotional campaign launched around the time the results flattened out?  Are the traffic numbers consistent?

This test has been a perfect reminder for us that it’s always important to check for invisible gorillas in reporting.  Here are four simple reminders of what you should be monitoring in Adobe® Test&TargetTM:

1.     Look at the BIG picture.  Never take what your reporting tool is telling you at face value.  Looking at the long-term trending perspective to see what’s happening over time is crucial to accurate reporting.  Sometimes a clear winner in the beginning of a test may prove to be much weaker after the point of normalization.

2.     Segmentation is your friend.  If you’re seeing confusing results, part of the confusion may be the way different audiences are engaging with your site.  The key to figuring out what’s causing the change is to slice and dice your data to pinpoint where the change occurred.  Set up segments before running your test to be able to look into details later.  Just because you don’t think you’ll need those segments now, doesn’t mean you may not need them later.

3.     Check for outside interference.  You may have set up the perfect Adobe® Test&TargetTM campaign, but testing never takes place within a vacuum.  What’s going on with other active and inactive tests?  Was there a campaign to promote free trials or samples that may have affected sign-up or purchase conversions?

4.     Never stop testing.  Adobe® Test&TargetTM is designed for you to constantly refine your test campaigns.  Whether you have flat results or a clear winner, continue to run iterative tests to determine which specific elements are moving the needle.