Over my past six years in optimization, I’ve seen a shift in the way experimentation strategies are created. First, it was all gut and intuition. Next, it was all about best practices. Today, it’s all about quantitative and qualitative data. But more advanced experimentation programs laying the foundation for tomorrow’s testing are integrating behavioral economics (BE) into their ideation process. Wondering how this works?
Brooks Bell and Irrational Labs are teaming up to tell you what you need to know about harnessing the power of BE in your program. Get to know about our partnership in our intro post or catch up on some of our surprising test results if you haven’t checked them out yet.
This post is all about bringing quantitative data, qualitative data and behavioral economics together to create a powerhouse ideation methodology.
As the co-founder of Irrational Labs, Kristen Berman helps companies and non-profit organizations understand and leverage BE to increase the health, wealth and happiness of their users. Their approach relies heavily on BE principles – with the help of quantitative and qualitative data.
As the lead of experimentation strategy at Brooks Bell, I help our retail, on-demand service, entertainment and non-profit clients build strategies to increase conversion through a better understanding of their users. Our approach relies heavily on quantitative and qualitative data – guided by behavioral economics alongside our taxonomy.
Our two approaches overlap greatly, with slight deviations based upon the nuances of our businesses. Check out these two methods to see if they apply to your program. And if you need some more customized support,
let us know so we can help.
The Brooks Bell Way:
- Pre-Strategy Data Alignment
We look at three primary data sources before beginning an experiment.
- Our analysts pull all available quantitative data and specify our primary and secondary metrics.
- Our UX team generates qualitative findings from internal usability inspections or external user research.
- Our development team performs a technical audit to inform the team of any limitations. Our team aligns on this internally before kicking off our strategies.
- User Needs and Problem Identification
We begin our ideation process by understanding user needs. We have a series of questions we answer to help us step out of our power-user mindset and into the shoes of a prospective customer. From there, we define what data point led us to uncover a specific problem.
- Experiment Ideation
To create a solution to that specific problem, we consult our “ideation toolbox” which consists of past test learnings, our proprietary pain points and tactics and behavioral economics principles. We define the solution in detail, as well as specifying the learnings.
- Strategy War Room
We each pitch our ideas in our Strategy War Room, which serves as our strategy command center. We ask questions and make modifications as we go, and we ultimately create a prioritized list of experiments that we execute, analyze and document.
- Learn and Iterate!
Rinse and repeat! We keep the cycle going, ensuring that each experiment is designed with even more information than the last and that we’re using an iterative process.
The Irrational Labs Way:
Literature review: More often than not, we’re designing for a problem that already has been researched. If this is the case, we suggest reading literature on the problem first and then coming up with a few reasonable hypotheses that could drive behavioral change. There is typically a fair amount of research that already exists, so be sure to give yourself time for this step!
Note: Very rarely are we walking blind, without existing research to learn from. But if you have no idea where to start on a concept (designing for another country or completely new technology), we suggest starting with qualitative research to get a basic lay of the land.
Behavioral Mapping: The details are everything. In order to understand how to change behavior we must carefully examine the environment that the behavior is occurring. We identify the key behavior we are looking to change and map out the steps required to complete the behavior. When possible, we layer on data insights to this map like ‘where are people dropping off?’ and ‘what are people doing at each step?’. Doing this exercise helps us identify specific barriers that we could remove in order to reduce friction to the key behavior as well as identify the benefits we could add to increase motivation to complete the key behavior.
Quantitative: The quantitative method comes next. Instead of interviewing a handful of people, the behavioral approach suggests we first conduct a study with more than a thousand people to help us isolate the key hypothesis. This can look like a simple survey that has multiple conditions, testing a key assumption. Pro tip: Use qualtrics and MTurk to make this step easy!
Qualitative: At this point, talking with current or potential users about different variations you’re considering will prove useful. Have people go through a few different experiences, and watch how they react to them. We always try to include a behavioral measure in our qual studies instead of just listening to what they’re saying. How many seconds did someone stay on a page? Did they ask follow up questions, if yes, how many?
A Controlled Trial (A/B test!)
Now we integrate all of these insights (lit review, quant study and qual feedback) into a controlled experiment. Behavioral science is based on what people do vs. what they say they do. It’s the practice of testing your assumptions by isolating the variable and seeing what happens. This method of testing allows us to claim that X caused Y. Too frequently, we look at correlational findings to make big product decisions. The only way to know if something affects another thing is to do a controlled trial. This scientific process is what behavioral science is built on.
Hopefully, this sheds some light on how you can better use quantitative and qualitative research to further your experimentation! Next month, we’ll talk a little ethics – so put your white hats on!
Have a question for Kristen and the Irrational Labs team? Let me know! Email your questions to us, and we might explore them in a future post!
Suzi Tripp, Sr. Director of Experimentation Strategy
At Brooks Bell, Suzi sets the course of action for impact-driving programs while working to ensure the team is utilizing and advancing our test ideation methodology to incorporate quantitative data, qualitative data, and behavioral economics principles. She has over 14 years of experience in the industry and areas of expertise include strategy, digital, communications, and client service. Suzi has a BS in Business Management with a concentration in marketing from North Carolina State.
Kristen Berman, Co-founder of Irrational Labs, Author, Advisor & Public Speaker
Kristen helps companies and nonprofits understand and leverage behavioral economics to increase the health, wealth and happiness of their users. She also led the behavioral economics group at Google, a group that touches over 26 teams across Google, and hosts ones of the top behavioral change conferences globally, StartupOnomics. She co-authored a series of workbooks called Hacking Human Nature for Good: A practical guide to changing behavior, with Dan Ariely. These workbooks are being used at companies like Google, Intuit, Neflix, Fidelity, Lending Club for business strategy and design work. Before designing, testing and scaling products that use behavioral economics, Kristen was a Sr. product manager at Intuit and camera startup, Lytro. Kristen is an advisor for Loop Commerce, Code For America Accelerator and the Genr8tor Incubator and has spoke at Google, Facebook, Fidelity, Equifax, Stanford, Bay Area Computer Human Interaction seminar and more.