We have updated our Privacy Policy and Privacy Options

Got It

Going to Extremes: How One Bad Data Apple Can Ruin the Whole Conversion Bunch


You’ve heard the saying that “one bad apple can spoil the whole bunch.”  It’s the fundamental idea that one thing can have a negative impact on its surroundings.  With conversion data, extreme orders can be the bad apple.  By definition, the word extreme is “of a character or kind farthest removed from the ordinary or average.”  Which, for a conversion expert, can really throw a wrench into things.

To illustrate an example of an “extreme order”, let’s head to the classroom.

There is a classroom of 21 students, all anxiously awaiting their final exam.  One student joined the class very late in the semester, and is not adequately prepared to take the exam.  However, as part of the class, they are required to take the exam.  When testing concludes and the exams are scored, results show that 20 students’ scores are distributed evenly among the curve.  The student who joined the class late gets a very low score, which was expected due to their unique circumstances.  How will this impact the overall average score?

If this unusually low grade is considered when determining the average score, it will drag the average down.  This situation is out of the norm because the low scoring student isn’t on the same playing field as the others, so including the extremely low score muddies the water.  So that it does not skew the data, this score should be removed from the data set.

This often happens in conversion data.  For example, companies may want to separate consumer purchases from larger retailer purchases.  Adobe® Test&TargetTM uses the example of an individual athlete purchasing a uniform vs. a coach purchasing uniforms for an entire team.  They come through the same conversion path, but their average order value (AOV) is very different.  As you can see, this is a situation where the marketer may want to isolate the consumer data to get an idea of what the average consumer is purchasing.  An extreme order is defined as more than +/- 3 standard deviations from the AOV.  What does that mean?  Here’s an illustration*:

Normal distribution (dark blue) accounts for 68.27% of the set.  Two standard deviations (medium and dark blue) account for 95.45% of the set.  Three standard deviations (light, medium and dark blue) account for 99.73%.  When filtering extreme orders in Test&Target, anything that falls outside of the light blue area would be flagged as extreme.  Its order value is replaced with the AOV of the experience (excluding extreme orders). This will affect AOV and revenue per visit (RPV), but does not impact the conversion rate.

With Test&Target, you can filter extreme orders so they don’t affect your campaign results.  The great news is that the default report will capture their data, but you can filter them when analyzing the data.  This gives you the opportunity for a side-by-side comparison to see how extreme orders impact your conversion data.

To filter extreme orders from your reports:

1.      Open a campaign and click the Reports tab.

2.      Click Extreme Order Filter.

3.      Click Exclude.

4.      Click Show.

The moral of this testing story is: don’t let a few bad data points spoil the analysis. Filter out extreme orders so that you are able to accurately analyze the conversion data.

Removing the extreme data from your life gives you room for extreme couponing, extreme makeovers, extreme sports, etc.  After all, some extremes are worth keeping around.

Brought to you by Brooks Bell conversion experts Brian Shampnois and Suzi Tripp!

*Standard Deviation Chart