After a string of good hands, your stack is nearly three times as high as your opponents’. You hold a pair of queens–not terrible, not great–and it’s your turn to call, raise, or fold. What do you do?
Now imagine a reverse scenario: You’ve had a bad run of cards, and now your little stack is the shortest on the table by far. You’re holding the same pair of queens, and it’s time to make a decision. Is your strategy different this time around?
The problem with testing like LinkedIn
When many companies think about building a first-class testing program, they often look to big-name brands like LinkedIn, Bing, and Facebook. These companies are experimentation machines. Data is in their DNA. Everything gets tested. Their culture, understanding of experimentation, skillsets, and processes are enviable.
So why wouldn’t you want to come up with ideas like them?
Because those companies are like the chip leaders in a poker game. The testing stakes are low because they can afford to run thousands of tests per year. Can you?
The main reasons most companies can’t run a high volume of experiments per year:
- They don’t have enough traffic and conversion points
- Testing is expensive
Two factors attribute to the expense: the cost of the testing platform and total human hours.
Total human hours includes the actual hours dedicated by anyone involved in launching the experiment as well as the opportunity cost of not using those resource hours to work on other parts of the business.
Do you know how long it takes to run experiments at your company? Think about ideation time, mocks, approvals, campaign setup, development, QA, analysis, reporting, sharing, and implementing. Especially for bigger organizations, this number can be quite large and frightening.
In other words, when most companies launch experiments, the stakes are high.
Fewer tests, bigger insights
Companies like LinkedIn, Bing, and Amazon have the traffic, culture, and infrastructure that allows them to run a high volume of tests quickly and efficiently.
Rather than spend a lot of time on strategy, they often run tests to find small incremental lifts.
A team member at one of the above companies recently admitted to me, “I think that the rigor around test planning, gaining insights, and iterating isn’t the best [at our company]. We give up some of that for volume.”
So if you can’t play the volume game, what should you focus on?
You can never be certain if a test will win or not, but you can improve the chances with great research, a solid ideation methodology, and creating a collaborative environment with others to generate ideas.
Make all the hours for those involved in the experiment count. Make every test count.
Spend time developing a process for test ideation, development, and analysis. Create a culture that emphasizes learning, not merely accruing wins.
In the end, no matter how high your stack is, you can reap big rewards if you play your hand strategically.
This post was written Naoshi Yamauchi, the chief performance officer (CPO) at Brooks Bell Inc. In this role, he leads the analyst and engineering teams, oversees ideation and strategy, and refines processes and methodologies. Naoshi is focused on increasing win rate, pushing velocity, and deepening learning through testing for all Brooks Bell clients. He received an MBA in marketing and innovation management from North Carolina State University.
Brooks Bell helps top brands profit from A/B testing, through end-to-end testing, personalization, and optimization services. We work with clients to effectively leverage data, creating a better understanding of customer segments and leading to more relevant digital customer experiences while maximizing ROI for optimization programs. Find out more about our services.