We have updated our Privacy Policy and Privacy Options

Got It

Click 2012 Opening Remarks


Hi everybody! And welcome!

I’m Brooks Bell, your host and chair of the Click Summit, as well as the founder and CEO of Brooks Bell. I am delighted to be surrounded by so many amazing testing experts, analysts and marketers.

I want to thank all of you for joining me here today. You guys are the main attraction of the summit. It’s the opportunity to participate in meaningful dialog with you that really thrills me. I especially want to call out David Chivers, Jesse Lipson, Trevor Claiborne, Bill Gassman,  Brion Hickey, Amin Makhani and Ashish Vij who have been part of every Click Summit since we founded this in 2010. As you guys know, we’ve come a long way.

I want to thank our exclusive headline sponsor, Google, for sponsoring Click a third year in a row. Google has been incredibly supportive since the beginning, and has truly made it possible for us to make the Click Summit a reality.

Let’s talk a little bit about what’s happening in testing and optimization.

It’s an exciting time to be part of the optimization industry.

More than ever, I tell people we do A/B split testing, and they actually know what I’m talking about.  For the first time, testing and optimization has finally started to earn a separate line item in marketing budgets and product development roadmaps. I’ve also noticed that testing is finally getting recognized as a role worthy of full-time resources, as evidenced by a new trend in job titles like “Director of Optimization” and “Testing Analyst.”

We’re entering a golden age for testing platforms, too. A few years ago, the only options were Offermatica, Optimost and Google Website Optimizer. And things have totally changed. I recently checked WhichTestWon.com and noticed that there are now 43 testing platforms to choose from. And, some of the more polished new entrances like Monetate, Optimizely, and Maxymiser are growing like crazy.

I can understand why. Conceptually, A/B split testing is easy to understand. It’s just splitting traffic into separate experiences and comparing the results of each. And it takes simple math to see how a 10- 20% lift in conversions would positively impact the business. I think the allure of this concept is what’s fueling the manic growth of the industry. Companies have been mesmerized by the idea that if they test the color of their button or change their headline, they might magically hit all of their revenue goals.

However, the testing experts in the room know that this is often an unrealistic hope. We know that hope is not a strategy. For those of you who have actually built an enterprise testing program from the ground up, you know that the reality of testing is vastly more complex and nuanced. Thankfully, companies are at least trying testing out anyway.

But there’s a still a big problem in the testing world. A lot of these new companies simply aren’t seeing results quickly enough and are getting discouraged. There are a few reasons for this.

First, it’s difficult to integrate all their data sources, so they’re not doing that and their data isn’t complete, and so it’s probably not the right data for the program.

Second, it’s still difficult to choose the right thing to test, because that’s where the real expertise and experience comes in. This is a big deal because choosing the right thing to test is the most important part of testing. If you choose the wrong thing, you are about to invest weeks of time and tons of resources on a futile effort, only to have it come back flat.

The third thing is that it takes lots of coordination of resources and a lot of time to run a test. Between getting in line for IT and creative resources, and waiting for a test to reach significance, a test can take a month or more to run. For companies who aren’t prepared for that, it gets frustrating quickly. Furthermore, results are more elusive than people are often led to believe. Between our own experience and through interviews with other enterprise testing experts, the standard for a statistically significant WIN is 1 in 4 tests. On average, 2 out of 4 tests are flat or a loss. And 1 is usually broken.

Lastly, between the expenses of the platform and the expense of the consultants, testing is proving to be a pretty expensive tactic. The cost of ownership is high, and it’s hard to see results unless you’re ALL IN. And right now, most companies are not ALL IN.

So, if we put it all together, 1 in 4 tests win, and a lot of companies only get 1 test out a month. That means they’ll only win 3 times the whole year. And, they may have just invested $100-$200k in resources and headcount. And so that just feels like something’s wrong with the picture. And so that’s why they’re walking away.

This reminds me a lot of mobile industry 5 years ago. I remember a company named Windwire, which had an amazing technology for monetizing the advertising inventory on the mobile web. Their solution clearly addressed an obvious opportunity in the industry. It was a wireless ad network, and was even instrumental in setting the standards in wireless advertising. Unfortunately, this was 2001 and they were way ahead of their time. They flamed out in 2002. How do I know this? I worked there as an intern in college and saw it first-hand.

Like testing, mobile has had huge promise for years, but until recently, the technology wasn’t quite right, the culture wasn’t there, and the investment didn’t quite make financial sense for most.

Now, we all know that mobile has finally arrived. And what it took was a progressive company that thought differently. It created an incredibly usable and accessible smartphone technology, negotiated some innovative ways to promote adoption, like unlimited data usage, and created a new culture around the app store. After years of trying, the mobile industry finally clicked with consumers and companies. Testing is just a couple of steps behind mobile, and we need to approach the challenge with similarly innovative thinking to help companies bring testing deeply into their organization. Google took the first step around 6 years ago with Google Analytics by making data accessible and free, and we need to continue to build on that foundation that they established.

Intuit has proven that testing can be successfully incorporated into a large, established organization. Intuit just finished running hundreds of tests for TurboTax this past accounting season. Intuit’s CEO, Scott Cook has a fundamental belief in the value of scientific experimentation. Scott believes that ideal company culture is where the experiment is king. Contrast this to most traditional companies, where the org chart is king. I watched Scott speak on this issue at Techonomy, a tech conference last fall in Tuscon. He talked at length on the value of scientific experimentation in organizations and how it’s the “management structure of the future,” where junior people can have broad power to run experiments and let their ideas prove themselves. It clarified for me that the ultimate success in testing doesn’t depend on the latest features of a testing tool. Instead, it depends on a much deeper shift in management, politics, decision making and organizational culture.  Dylan Lewis, one of the testing rockstars from Intuit, is here at Click and is going to speak on Intuit’s approach to testing during his session this afternoon.

The good news is that this is already starting to happen on its own. The next generation of companies and most start-ups have testing in their DNA. Zynga is a great example, especially given last year’s billion dollar IPO. I recently watched a video of Mark Pincus, their CEO speak to a class at Stanford. He explained the concept of “ghetto testing,” where they test every feature before they build it into a game. The way they do it is to create a landing page that promotes a feature before it exists and then they measure how many people try to sign up for it. If they get a good response then they actually go out and build the feature. Mark also said:

We built a data warehouse with a testing platform so we’re running several hundred tests at any given time for every one of our games.  And no single user has more than one test. So, we love tests.  When we see that it moves our metrics in a considerable way, that’s when we take it to be a full feature roll-out.

While we all can’t become a Zynga overnight, it reinforces how testing, when deeply integrated into your organization, can really move the needle.

What we need to do today, as a community of testing experts, is to figure out what needs to click for the rest of us to help testing finally realize its promise for established companies, who have a lot of organizational baggage to overcome. Some of the key issues to discuss today are corporate politics, testing process, better approaches to choosing what and where to test, and of course leveraging data and technology behind it all.

And I am eager to learn from all of your perspectives on how to bring the testing dream alive within our own companies.