We have updated our Privacy Policy and Privacy Options

Got It

Test Results Need To Normalize: Be Patient My Little Grasshopper


Estimated Read Time: 3 min

What’s In It For You: You might save yourself some embarrassment or from mistaken happiness.

Keep reading if:

  • You’re kind of impatient
  • You love challenges
  • You love winning
  • You’ve at least heard of landing page testing (A/B or MVT)

If you run online tests on landing pages, regardless of which tools you use such as Google Website Optimizer, Test&Target, Visual Website Optimizer, etc., you have to let your tests run for more than three or four days REGARDLESS of if the testing tools claim +160% in conversion rate at 99% confidence on the first day.

First, think about what you’re testing. Are a few simple modifications to your homepage really going to create a +160% lift in conversion rate? Assuming your homepage is the highest viewed page, are you really seeing a record-breaking sales day because of this +160% lift? As exciting as it is to see these too-good-to-be-true lifts, don’t pop the champagne bottle yet. Just like most things in life, lift doesn’t come easy.

Below is a great example of an actual test currently running and why to keep the lid on the bubbly for a little while. Here’s the first three days of reporting for a control/A/B/C landing page test.

Day One:

Day Two:

Day Three:

The reason why you need to be patient and not immediately shut off the losing experiences (cells) is that the data needs to normalize. Not getting into technical detail, the testing tools take some time to get the results under control.

Here’s a good example of how tests normalize over time. It’s a bit wild at first, but usually settles down.

What you need to know:

Monitor closely, let the tests ride and normalize. I recommend at least four or five days.

A few quotes from this post I want to claim!

“Just like most things in life, lift doesn’t come easy.”
“Let the tests ride and normalize.”