A Year in Experimentation: 2016 Learnings and Predictions for 2017

We interviewed the Brooks Bell team to find out their top experimentation and optimization learnings and predictions for 2017.


A Year in Experimentation: 2016 Learnings and Predictions for 2017

This year, shoppers have spent more than $28 billion online since Thanksgiving, representing a 13.5 percent increase from the same period in 2015. Perhaps more interesting, the number of unique online shoppers grew by 16.4 percent, indicating more individuals are making purchases online. This is supported by survey data collected by the National Retailer Federation that found 52 percent of consumers questioned expected to purchase something online before Christmas. This early data points to a trend that shouldn’t surprise anyone: ecommerce has increased in importance over the last year.

While the growth of ecommerce demonstrates a shift in the total share of wallet digital transactions represent, it also suggests that product discovery patterns, means of evaluation, approaches to decision-making, and consumer behavior in general are all shifting. In this context, improving the customer experience demands a process of learning and discovery. “When we started with one client this year,” Suzi Tripp Paugh, Senior Director, Experimentation Strategy at Brooks Bell explained, “we had little to no historical data available to inform our testing strategies.” Without this baseline, the logical approach was to test best practices and big winners from previous clients. “When we tested these best practice ideas, they tanked,” she continued, “it shows there are no guaranteed wins or list of trade secrets to apply. You have to learn about the unique needs of your visitors and improve the experience based on those needs.”

Suzi Tripp headshot“There are no guaranteed wins or list of trade secrets to apply. You have to learn about the unique needs of your visitors and improve the experience based on those needs”
-Suzi Tripp Paugh, Senior Director, Experimentation Strategy

Mobile use patterns, too, have changed. Market research data indicates that 63 percent of adult cell phone owners use their phones to go online and 34 percent of cell Internet users go online mostly using their phones. “We have seen the mobile traffic of our clients increase dramatically this year,” says Brian Shampnois, Director of Analytics at Brooks Bell. “More importantly,” he explains, “the actual rates of purchase on mobile have increased at an even faster pace.” That not only the total volume but also the propensity to purchase via mobile devices has increased “shows people are more comfortable with shopping on mobile devices but also that the entire consumer experience is now mediated through mobile devices.”

brian-shampnois“Rates of purchase on mobile have increased at an even faster pace – people are more comfortable with shopping on mobile devices.”
-Brian Shampnois, Director of Analytics

 

Addressing these substantial changes requires a realignment within many digital businesses. Where advertising and promotions have traditionally driven growth and dominated strategy, experimentation, customer insights, and analytics are gaining prominence. As this focus has changed, it’s become increasingly clear that few businesses have the infrastructure and resources necessary to meet the need. “Analytics is becoming more advanced and should represent a larger share of the budget,” explains Founder and CEO, Brooks Bell, “but at the same time, we’re still seeing businesses focused on improving the page and not the customer experience.” Often, experimentation processes still come into conflict with other business operations. “The ultimate point of an A/B test is to develop causal inference,” Reid Bryant, VP, Analytics and Data Science at Brooks Bell says, “promotions, sales, ad campaigns, and other operations are considered confounding variables which can limit our ability to truly ascribe causality to a result.”

To a certain extent, this is a problem of culture. At the same time, the recognition that testing is an important tool for developing competitive advantage is expanding. “For many of the companies we have the opportunity to work with,” Brian says, “testing has become more commonplace.” Where a few years ago, having anyone with even basic knowledge of testing was considered a huge advantage, today it’s no longer a question of how to get testing off the ground but “how to refine their practice to attain ambitious goals.” To compete in the increasingly online, mobile world, “simply having basic knowledge to launch a test isn’t enough.”

 

Predictions for 2017 

With this substantial change in both consumer behavior and business strategy underway, knowing what to look for in the coming months is incredibly important. “We’re going to start to see the data and analytics infrastructure coalesce,” Brooks explains, “in a way that enables the creation of a clear data strategy that can be activated in real-time.” This establishes the essential groundwork that will make personalization—a perennial prediction—possible moving forward. “When it comes to personalization,” Reid says, “it seems like businesses have greater motivation to pursue it in terms of actual investment and execution.”

Brooks Bell“We’re going to start to see the data and analytics infrastructure coalesce in a way that enables the creation of a clear data strategy that can be activated in real-time.”

-Brooks Bell, Founder and CEO

There will be many benefits to this renewed focus on personalization. “Of course, personalization helps to make experiences more relevant, which we believe is an inherently good thing,” Reid comments, “but it also reduces confounding variables by making experiment conditions specific and constrained.” In doing so, “personalization could increase experimental rigor.”

Reid Bryant“[Personalization] reduces confounding variables by making experiment conditions specific and constrained…[which] could increase experimental rigor.”

-Reid Bryant, VP, Analytics and Data Science

Getting to this point will take some serious work. “People talk about the amount of creative a personalization strategy requires,” Brooks says, “but it takes a lot more data sophistication than people realize.” Systems must be integrated, analyst skills must be elevated, and segments must be behaviorally defined through real-time analysis or complex modeling. The biggest challenge, Brooks explains, is that “solving these problems will look very different across verticals. Personalization for retail is very different than it is for financial services, consumer packaged goods, or any other industry.” Addressing these challenges will be critical if businesses intend to created targeted and tailored experiences in 2017.

Whether it is driven by personalization or improved experiment strategy, testing will continue to provide a more meaningful and productive customer experience. “We’ll see a continued emphasis on needing to prove results from experimentation,” Suzi says, “but learning about the user will become much more important.” As small UX and interface changes have less and less impact on behavior, “we need to start asking deeper questions about the user.”

 

What did your team learn about experimentation and optimization in 2016? Share your learnings below in the comment box.

Categories Analytics, Copy & UX Design, Strategy & Process, Technology & Dev