We have updated our Privacy Policy and Privacy Options

Got It

Brooks Bell’s Opening Remarks at Click Summit 2013: ‘Let’s Focus on Learning’


The following remarks were made by Brooks Bell at the fourth annual Click Summit on Wednesday, May 15, 2013 at Clift Hotel in San Francisco.

Brooks Bell Click Summit 2013

Welcome to Click!

Four years ago on this stage, I found myself … absolutely terrified.

It wasn’t a fear of public speaking — I’ve done a lot of that.

It was because I felt the stakes were higher than ever before. I was looking in the eyes of dozens of people to whom I had promised this would be a world-class event — despite it having a format and an agenda that was almost unheard of at the time.

And it was standing here, kicking things off, that I realized how much was on the line. I was about to send everyone into Click’s first “conversation” session. There was a real chance it would be 75 horrible minutes of awkward discomfort.

I was about to know if this was going to be an epic success or an epic failure.

Now, four Clicks later, fear is pretty much the furthest emotion imaginable from what I feel right now. Today, I’m not afraid — I’m excited, because I know what’s in store for you today. It’s almost like getting to watch each of you open a present!

The format of Click is uncommon because the value of this event is completely driven by the attendees, and we have 85 top testing individuals from some of the best-known consumer brands in the world.

The main reason this event continues to grow is because of the incredible talent, experience and expertise in the room. Each of you sitting here is going to contribute to the conversation today. And that’s really special.

Thank you for being here today. Thank you for sharing your stories. And thank you for sharing your ideas.

I want to thank Monetate for sponsoring Click this year. Monetate has been incredibly supportive of us, and believes in the bright future of testing and optimization. In a few minutes, I’m going to invite Monetate’s VP of marketing, Blair Lyon, to join me onstage to share his vision.

But first, I’d like to share some of my thoughts on this past year.

After last year’s Click Summit, we published 20 “Clickaways” of the most concrete takeaways from our conversations. I reviewed them recently, and we really covered a lot of ground! We discussed process, content, reporting and personalization. But the overriding theme was about culture.

At Brooks Bell, we’ve witnessed many of you do the tough work of building your testing culture over the past year. It’s a complex business — organizing your data, getting the right process in place, improving your experiment design and, of course, gaining buy-in from your executives and colleagues.

But an equally complex issue that hasn’t been as well-addressed at Click is having a clear approach to deciding what to test in the first place.

One reason for this is that testing is often used only as a way to reduce risk.

Often, when a big company project has been approved — say, a new website design or application feature — the last step before launch might be to test it. By the time testing is considered, everything is fully baked and ready to launch. So the question of choosing what to test is totally irrelevant because testing is just to confirm the market demand.

Essentially, testing is increasingly being used an alternative to beta rollouts. And this is a great use for testing. Sophisticated companies like Dell regularly leverage testing in this capacity and have used their experiment results to inform important strategic moves.

Other companies could benefit from the risk-reduction testing approach. On April 8, the day that J.C. Penney CEO Ron Johnson’s dismissal was announced, the New York Times revealed that:

“Mr. Johnson favored bold actions, and put the plans on pricing, marketing and merchandise into place without doing any small-scale tests.”

We can only imagine how things might’ve been different for J.C. Penney if Ron Johnson had performed some market tests beforehand.

While using testing as a risk-reducing tactic is a huge step forward for many companies, it’s still more or less missing the biggest value that testing can bring to the table. And that’s using testing with site optimization in mind.

With an optimization mindset, testing is a forward-looking vehicle to adapt the site to connect with your customers. It changes the focus from “Is there a market for that?” to “How can I make the most relevant, most personalized, effective site experience for my customers? How do I learn what my customers want from my company? How do I learn who my customers really are? How are my customers’ needs changing?”

Of course, there are more ways to answer these questions than testing, but testing can be a very powerful tool in this quest.  This is how I think of optimization, and this is why I am eternally excited about what testing can do.

In an optimization-oriented testing program, choosing what to test is critical. It’s the foundation for everything thing else.

Choosing what to test is a question my team has been quietly thinking about — and experimenting with — for the last three years. We’ve been on a crazy mission to try to discover the source of all good ideas — the Fountain of Winning Tests. We’ve wanted to learn the secret to crafting the perfect test idea.

And today, I’m going to say… that we still haven’t found it.

While we haven’t located the Fountain of Winning Tests, my team has experienced the pros and cons of many different ideation approaches over the years. I’m excited to announce that we’ve organized these into a framework to help us know where we stand and to help us to continue to evolve our thinking.

Today, I am sharing the Five-Stage Testing Ideation Model. We’ve been through all of these stages ourselves; they build on each other, and having a taste for each stage gives us the context for the next one.

Stage 1: Pre-testing and Risk-Reduction

Stage 1 is actually the mindset that often precedes testing completely. It’s the mindset that the more experienced and often more senior people are, the better their taste is, and the more likely they are to accurately predict performance.

I read a great quote from a book titled Four Steps to the Epiphany by a well-known Silicon Valley guru, Steve Blank: “In traditional companies, CMOs are hired for what they know, not what they can learn.”

Essentially, it’s the unspoken belief that many marketing directors have: “If I like it, it’s gonna work.”

Of course, your CMO’s intuition is probably pretty good. A lot people make it to the executive suite because they’re smart and their judgment is sound. So, there’s a reason this approach has worked well for decades.

But relying on this type of thinking can also easily lead to the hell that J.C. Penney is living through. Risk-reduction testing is the most natural evolution for companies with a Stage 1 culture, where testing is an add-on to just validate the market rather than a strategic priority of its own. It’s a great way to start to see the value of testing, but it can’t be called optimization at this stage.

Stage 2: The Buckshot Approach

The Buckshot Approach is usually the stage where a company discovers a testing tool for the first time, gets really excited about the possibility and starts to willy-nilly try out different images, headlines and layouts to see what happens.

What’s awesome about this stage is that the company has discovered that their conversion rate isn’t a number that’s fixed in stone. However, by looking at their testing tool as a shiny new object, it’s easy for companies to get distracted by what’s easy to test, partially because they have no idea what they should test.

Stage 3: Best Practice Land

Best Practice Land is definitely better than the Buckshot Approach, and there are a lot of winning tests and more effective designs due to applying best practices. Entire companies in the conversion industry exist through teaching best practices alone. And there is a certain a seductive allure that a conversion expert in the room can look at your landing page for 15 seconds and diagnose 20 things that if you would just fix are a surefire way to lift your conversion rate.

There are a couple of things that are interesting about this ideation stage: First, it’s not focused on iteration. The idea is that you apply the best practices once, you get a massive win and you’re done!

Second, the majority of best practices, such as lock symbols, are rooted in consumer psychology, but at this stage companies don’t fully understand the principle behind it. They go straight to the tactic itself. That’s the premise of the 20-second live landing page audit — with this, the conversion expert wouldn’t need to know anything about your customers, your data, your strategy or your goals to give you a decent tactical solution on the spot.

I’m not saying this approach doesn’t have merit — and I am embarrassed to admit there are still tests for which we pull out a best practice without deeper thought — but it only works to a point.

Stage 4: Applied Psychology

At this stage, we adopt a much stronger appreciation for a consumer’s point of view. For instance, we’ll introduce an applied psychology framework that has big themes like loss aversion and social proof, and use this to come up with better ideas about what might work.

In fact, Applied Psychology is the approach we’ve been using over the past year. It’s a great way to help us understand the “why” behind best practices, and build better judgment for when and where they’re likely to be effective.

Being at the stage of Applied Psychology is really good. It produces winning tests and even some good learnings.

But it’s lacking, because it’s based on universal psychology frameworks. We’re still assuming that there is a universal best experience for all consumers.

Stage 5: Discovering the Customer

This stage represents the next evolution of ideation. It’s where we stop thinking about consumers and start thinking about customers. We stop thinking about choosing what to test and instead choose whom to test. So instead of a starting with a psychology principle, we start with a segment.

Actually, a better word for segment, I think, is “customer group.” The customer groups become the basis for our testing program. And this gets us thinking about differing needs and differing solutions.

Deciding how to segment your data is important, but not that easy. Is it as simple as new versus returning? Is it the product pages they are showing an affinity for? Is it the time of day they are visiting the site that reveals their needs? Are their other qualitative tools like call logs or marketing research available to give us clues to where to start?

There are tons of interesting tests in here and increasingly powerful technology on the market to give us a leg up.

Once we have identified some potential groups, we need to consider the likely differences in their needs.

For a financial company, they might have two main customer groups: young professionals with a higher risk profile who want to maximize their nest egg over the course of their career; and older, wealthier retirees who care more about protecting their assets than growing them.

Having different needs mean they probably lean toward different products, respond to different value propositions and probably even require different amounts of information to make a buying decision. Testing is the best path to incrementally uncover the right solutions for each group.

The Discovering the Customer approach still leverages a psychology framework, but introduces a new layer of complexity and opportunity.

Rather than trying to get a quick win, the new focus of the program is to build an increasingly accurate picture of who our customer groups are, what they need and where they’re getting hung up on the site.  To take this approach to its logical conclusion, we need to make a change to what testing success looks like to us.

Instead of focusing on winning, we need to focus on learning.

The good news is that as we focus our test ideas on learning, our tests will have increasingly likely chances of winning.

This is where I think the exciting future of optimization lies for all of us — in using testing not to just double-check our strategy to avoid J.C. Penney-style collapse, but to connect with our many customer groups, discover their unstated needs, and add human emotion to our increasingly complex data.

The Five-Stage Ideation Model is one idea I’d like to contribute to our conversations, and it’s just one perspective on test ideation. The conversations led by Will Close from Dell, Jesse Lipson from Citrix, Grant Hosford from Idealab and Victor Foreman from Orbitz also touch on ideation and could explore alternative approaches. And maybe one of them has actually found the Fountain of Winning Tests.

But in addition to ideation, we have a solid list of topics to discuss today, from culture, to mobile, to multi-channel, to process and even modeling. I hope I’ve warmed you up to thinking about how to evolve our testing approach, but we’re just getting started.