We have updated our Privacy Policy and Privacy Options

Got It

How Testing Platforms Are Like Gym Memberships


One late evening at the Adobe/Omniture Summit 2010 After Party, I was watching a surprise Beastie Boys performance by Josh James, (former CEO of Omniture) and Brett Error, when a good friend of mine and fellow optimization expert, Brion Hickey, whispered in my ear, “Testing tools are like gym memberships. Hundreds of companies sign up, but no one uses them.”

The comment struck me as an insightful explanation to weird trends I was seeing at the time. It appeared that hundreds of companies were signing up for testing tools such as Adobe Test&Target, Sitespect, Google Website Optimizer and Webtrends Optimize, but I wasn’t seeing those companies report the positive results I expected to hear. Finally, an explanation!

This was 18 months ago and a lot can change in that timeframe. Adobe and rest of the testing platforms have had increased success driving usage since then (although I still haven’t seen reams of case studies yet) and they get more powerful and easier to use all the time.

However, the underlying theme of Brion’s comment still holds truth; in fact, getting companies to start testing is a lot like getting people to start exercising at a gym.

We all know we need to do it. There are tons of health, productivity and stress management benefits from exercising. But yet, only 6% of the population works out daily. The same thing goes for testing. We know now that it’s accountable, disciplined, objective, data-driven, proven, customer-centric, and is the #1 way to improve conversions. But yet, most of us don’t do it. Why??

The reason is the same for testing and exercising: they are both way harder than they look.

We approach testing the same way we approach exercising – we go and buy the equipment. That’s the easy and fun part. But that’s how we end up with all sorts of useless thighmaster gadgets… and idle testing tools.

We forget about the hard part: changing our behavior and habits. Like exercising, testing needs a daily commitment and focus. It has a true cost in time, team resources and budget.  Most companies have underestimated the commitment in headcount, time, process, and increased teamwork. On top of that, testing is often a shift in team mentality, how employee job performance is measured, and even company culture. While the testing tool is an essential part of the testing equation, it will only work as part of a broader shift in company focus, goals and measurement to a truly data-driven marketing culture.

When building your testing program buying a license to a testing tool alone isn’t going to instantly get you a successful program. Thighmasters don’t immediately give amazing thighs, either.