We have updated our Privacy Policy and Privacy Options

Got It

Segment Surprises! How Digging Deeper Can Turn the Testing Tables


We recently completed a test for an e-commerce client with some very interesting results.  The campaign focused on the color of the Add to Cart button.  What we expected to be a simple, straightforward test ended up provided some insightful results—results that underlined the importance of segment behavior.

The control has a navy blue “Add to Cart” CTA button, which is consistent with the rest of the site’s branding.  We believed that we could increase conversions by using a button color that really pops and draws the visitor’s eye to the intended CTA, so we tested an orange and a green button against the control.  We don’t usually focus on button color tests, because they are often too simple to gain learnings on conversion motivation.  However, when we experience a quieter period during a particular testing program, that’s the perfect time to run some of the simpler tests.

Surprisingly, both the orange and the green button versions lost to the control in average order value and add to cart rate.  We were curious, though, to see if new visitors showed a different preference— and did they ever. New visitors showed a 13.3% lift in average order value with the orange version of the button.  That meant a difference of over $25 in average order value!  What if we’d stopped at the loss in the overall test without digging deeper?  We would have optimized for the larger returning visitor audience, but we could have missed out on a big opportunity to optimize new visitor conversions.

This test was a great reminder to us of the simple importance of taking a close look at segment behavior when analyzing results.  It was also a reminder of yet another benefit of Monetate. While most testing tools require you to set up the segments you might want to track in the initial test setup, Monetate allows you to run custom reports with segments you want to evaluate after a test is already launched.  We’ve talked before about how awesome Monetate’s recyclable segment setup is, but it’s also important to acknowledge the power of being able to look into a segment’s results without the pressure of having to set up the segments from the get-go.

Blog Post 8.14.12 screenshot

New vs. Returning visitors should be one of the key segments you target in any test, but this was a perfect example of a situation where we were surprised by the segments behavior.  Being able to dig deeper into both of these segments after the test launched made the difference in understanding the revenue impact of this test. It also proved that setting up an iterative test to optimize conversions for new and returning visitors can bring home a big win.