We have updated our Privacy Policy and Privacy Options

Got It

“Oh, BEhave!” – Even Experimentation Experts Get Surprised


In our last post of our brand-new series, “Oh, BEhave!,” we announced that Brooks Bell has partnered with Irrational Labs to bring behavioral economics (BE) into your experiment ideation process. Why? There are decades of experiments that your program could be using to inform your strategies! If you missed it, check it out here.

Brooks Bell and Irrational Labs Share Their
Most Memorably Unexpected Results

We’ve seen the positive influence behavioral economics has made on programs and user experiences, and this series will teach you the basics, provide ideas on how to insert behavioral principles into your process and culture and share some of the surprises we’ve encountered along the way.

Before we get into that information, it’s important to understand that you should approach this with a mindset of experimentation. Unfortunately, there’s no universally perfect Shopping Cart experience template out there guaranteed to win. By understanding behavioral principles combined with quantitative and qualitative data, you can create an informed experience that gives you the best chance for success.

Kristen Berman, co-founder of Irrational Labs, and I recently shared some surprising experiment results we’ve seen. We have over a dozen years of collective experience and have seen many (perhaps thousands of) experiments! And the fact that we’re sometimes surprised is a testament to the fact that you should always test!

Let’s look at two examples:

Experiment #1:
When working with Google, Irrational Labs needed to convince small businesses to add conversion tracking to their AdWords account. Conversion tracking is a highly valuable feature that lets a business know how their ads are performing. It also takes a lot of work to add it to their account; they have to add a piece of code to their site and more. Thus, Google was offering a monetary incentive to help encourage businesses to add it.

Irrational Labs suggested that there may be alternative ways to motivate people other than paying them directly. They tested four different variations against the control:

  • No incentive (Control)
  • $20 Adwords credit
  • Two movies
  • $20 Google play credit
  • $50 Adwords credit

Which one won? Irrational Labs and Google’s intuition was that the two movies or $50 Adwords credit would win.

Wrong! The winner was $20 in Google play credit.

The first lesson here is that testing is critical. Even expert intuitions are incorrect, which is why Kristen loves this experiment. It deeply highlights how critical testing is. Imagine if Google had rolled out the $50 Adwords credit without testing it? They would have lost money and may not have realized it! Second, in hindsight, this result makes sense. The Google Play reward is a more hedonic reward – meaning that they are desirable objects that allow the consumer to feel pleasure, fun and enjoyment. They may even feel like a gift.

The Adwords credit is more utility-based – meaning that it may be perceived as more useful or practical. While helpful for their business, the credit may feel less immediately motivating for the small business owner.

As for those movies? We’re not sure why the “two movie” condition didn’t come out on top over the $20 Google Play credit. It’s possible the two movies we highlighted as examples didn’t hit the mark for many users and thus turned them off to the option.

Experiment #2
For the V Foundation for Cancer Research, donations can save lives. Brooks Bell partnered with the V Foundation to help increase online donations to support cancer research. When determining where to begin, Brooks Bell uncovered that potential donors may be concerned about where their donations go – directly to the cause versus paying overhead expenses, etc. At the V Foundation, 100 percent of direct cash donations go directly to cancer research. This was the benefit that we could offer to alleviate potential concerns about how donations were being used.

In the first experiment, Brooks Bell decided to address this potential concern head-on. On the “About” page, we created a variation that replaced the existing hero image of founder Jim Valvano, with copy stating that “100 percent of direct cash donations go directly to cancer research.” (Here’s some research that backed that idea up.) And what did we see? A double digit increase in donations. This was a big win for the V Foundation, for Brooks Bell and for cancer research.

The next test focused on the sitewide navigation. Naturally, we reflected back on our past test and saw an opportunity to integrate that message into the navigation. Instead of the “Donate Now” button in the navigation, the variation boldly stated “100 percent of donations go to cancer research,” followed by the Donate Now button. Brooks Bell and the V Foundation’s intuition was that it would lead to another positive result.

And what happened? The control’s “Donate Now” CTA in the navigation was a clear winner.

Perhaps it was the way in which the messaging was visually added to the navigation? Perhaps the availability of messaging on the sitewide navigation came across as “sales-y,” even though it was a helpful addition on the “About” page? The Brooks Bell team is still researching this outcome and strategizing for upcoming experiments.

As these examples illustrate, expert
intuition is not always correct
. It’s important to use data and BE to better understand your users and their motivations – and experiment from there. You’re potentially saving yourself from rolling out something undesirable, and you’re saving your users from an experience that wasn’t investigated as thoroughly as it could have been.

Tune in next month for our next post! We’ll be diving into quantitative and qualitative data – and how it can amp up your methodology.

Have a question or idea for Kristen and the Irrational Labs team? Let me know! Email your questions to us, and we might explore them in a future post!

Suzi Tripp, Sr. Director of Experimentation Strategy
At Brooks Bell, Suzi sets the course of action for impact-driving programs while working to ensure the team is utilizing and advancing our test ideation methodology to incorporate quantitative data, qualitative data, and behavioral economics principles. She has over 14 years of experience in the industry and areas of expertise include strategy, digital, communications, and client service. Suzi has a BS in Business Management with a concentration in marketing from North Carolina State.

Kristen Berman, Co-founder of Irrational Labs, Author, Advisor & Public Speaker
Kristen helps companies and nonprofits understand and leverage behavioral economics to increase the health, wealth and happiness of their users.  She also led the behavioral economics group at Google, a group that touches over 26 teams across Google, and hosts ones of the top behavioral change conferences globally, StartupOnomics. She co-authored a series of workbooks called Hacking Human Nature for Good: A practical guide to changing behavior, with Dan Ariely. These workbooks are being used at companies like Google, Intuit, Neflix, Fidelity, Lending Club for business strategy and design work.  Before designing, testing and scaling products that use behavioral economics, Kristen was a Sr. product manager at Intuit and camera startup, Lytro.  Kristen is an advisor for Loop Commerce, Code For America Accelerator and the Genr8tor Incubator and has spoke at Google, Facebook, Fidelity, Equifax, Stanford, Bay Area Computer Human Interaction seminar and more.