We have updated our Privacy Policy and Privacy Options

Got It

The Golden Ticket of Testing: Understanding Confidence

Share

Running a campaign that you are excited about is like being a kid in a candy store.  The kid walks in and marvels at all of the candy on display, they see the particular type of candy they were hoping to find and they can hardly wait to dive in.

When running a campaign, you see the data coming in every minute, it’s confirming your assumptions and you desperately want to call a winner and implement your new brilliant learnings.

But, not so fast…

It’s easy to get caught up in the excitement of testing, but it’s very important to ensure that the campaign has reached confidence.  Adobe® Test&Target™ provides confidence levels and confidence intervals for each experience to use as a helpful guide.

Willy Wonka

When you view reports in Test&Target, you select what cell should be used as the Control.  All Challengers are measured against that Control.  You’ll be able to evaluate the lift provided to conversion rate, average order value, revenue per visit and/or sales (our examples will focus on conversion rate).  The report will show the metric data, the lift percentage and the confidence indicator in an emphasized box for easy evaluation.

Reporting

The confidence level is displayed as a bar chart in the confidence column of the campaign report.  There are four bars that indicate the level of confidence the experience has achieved.  The more bars, the higher the confidence.  To see the exact confidence level, just hover over the bar chart. The confidence level is higher when the data proves that the results are repeatable and were not a fluke.

Confidence Level

Here’s where it gets tricky… A high confidence level alone in lift does not necessarily mean you should pull the trigger for the winning experience.  For example, if a campaign has just launched and has received only 25 conversions – the confidence level for one experience may show 90%+.  However, a total of 25 data points does not provide a clear picture of how this test will perform long-term.  In general, 100 conversion points per experience and running the test for at least a week is our recommendation.   So, what’s a conversion expert to do?

That’s where confidence intervals come in!

Confidence Interval

The confidence interval is the range that the confidence level refers to.   It’s very important for conversion experts to get into the habit of monitoring the confidence level in relation to the confidence interval.  You can do so by practicing the exercise of saying “we are __% confident that the conversion rate will be __% to __%” if this test is repeated (you can substitute any other metric).  If it sounds ridiculous, it probably is.  Meaning that if the range is incredibly wide or includes a negative number, the information really isn’t that useful and the test should continue to run to obtain more data points.

An ideal confidence interval range is small, with no overlap among experiences.  How do you achieve this?  Larger data sets, which are provided over time.

Think you’re ready to analyze a test?  Give it a try!  (See answers below.)

Problem to Solve

1. What is the confidence level?

2. What is the confidence interval?

3. How many total conversions are there?

4. Is there a clear winner?  If not / If so, why?

It’s easy to get caught up in the excitement of testing, but it is important to look at the whole picture before declaring a winner.   Don’t go all Veruca Salt, “but, I want it nowwww.” Trust us, good data is worth the wait!!

Brought to you by Brooks Bell’s very own Suzi Tripp and Naoshi Yamauchi!

 

 

 

Answers:

1. What is the confidence level?  95.79%

2. What is the confidence interval?  -4.74% to 37.6%

3. How many total conversions are there?  826

4. Is this test ready to call?  If not / If so, why?  No, because the confidence interval is too wide.  You never want to call a winner with an interval that includes a negative number.

Categories