How To Make A Case for Holiday Website Testing

Either you or the digital pioneers that came before you have already done the hard work to get executive buy-in for testing. But convincing them to let you run tests during the holidays is a whole different animal.

That’s because testing during periods of peak traffic, such as the holidays, is both high-risk and high-reward.

Convincing your higher-ups to test during the holidays requires you to address those risks and highlight the opportunities. To get you started, we’ve outlined a few of the common reasons e-commerce executives are apprehensive about peak season testing and a few talking points to help you change their mind.

Risk #1: Testing during the holidays puts you at risk of sending traffic to variations that may underperform

While this is a valid risk, it disregards the long-term impact of testing in favor of quick wins. Given the competitiveness of the retail space, particularly in the age of Amazon, we are of the belief that those that can afford to test, should.

Holiday traffic is a bit of an anomaly. You’re already attracting different customers with very different mindsets by the very nature of the holidays being a peak shopping period (thanks, consumerism), as well as increased advertising spend.

If you’re not testing during the holidays, you’ll never be able to understand what drives your customers during the holiday season. Any loss may actually generate more value in the future because it reveals insights about how your customers think and behave during the holidays.

Simply put: Testing during the holidays enables you to learn what works for future holiday periods.

There are some very low-risk tests you can run during the holidays that will allow you to glean insights about your customers. These include testing value propositions, featured products, images, return policies and shipping information. 

You can also test different types of urgency messaging to tap into shoppers’ mindsets. For instance, earlier in the season, you could test messaging around finding the perfect gift, whereas later in the season, you could build tests around the number of products left in stock, shipping deadlines and gift wrapping options.

Recommended Read: 5 Testing Tips During the Holidays

Another less-risky way to ensure your website is optimized is by running a multi-armed bandit test.

Multi-armed bandits use machine learning algorithms to dynamically distribute traffic to variations that are performing well and distributing less traffic to variations that are not. These types of tests are perfect for the holiday’s short promotional windows because they can reach a result in a shorter amount of time than a typical A/B test.

While the results of a multi-armed bandit test often don’t provide those game-changing customer insights, it’s a great fall-back option to ensure your website is truly optimized and positioned to capitalize on holiday traffic.

Recommended Read: 4 Strategies to Connect With First Time Holiday Visitors

Risk #2: Making changes to a site during seasonal code freezes puts your site at risk of crashing

True fact: seasonal code freezes are put in place to avoid catastrophes, like your site crashing during periods of peak traffic.

To get around this, many testing platforms have visual editors that allow you to create test variations without touching your site’s code, so long as the changes you make are simple and don’t significantly change how your site functions.

Recommended Read: The Expert’s Guide to Testing During the Holidays

Implementing a winning experience is also challenging during a seasonal code freeze. During the holidays, no two weeks are the same and you have to act quickly. Rather than bothering with implementation, many teams use their testing tool to roll out a winning experience to 100% of traffic, as a short-term workaround. While we don’t always recommend doing this, it is an option.

Risk #3: But what about bugs?

If you really want to blow a few minds, you can reference the Fogg Behavioral Model, which was created by Stanford researcher, BJ Fogg, who also founded Stanford’s Persuasive Technology Lab. 

Basically, Fogg’s research found that as a person’s motivation increases, their concern for ease and usability decreases. In the context of holiday testing, customers during the holidays are highly motivated to buy, therefore they’re less likely to be put off by bugs or experience issues.

But, of course, a site without any bugs is what we’re going for here.

In the months before the holiday season, we typically recommend doubling up on QA and make sure your site is already optimized for page load speed. Building a strategy for testing well in advance of the holidays is also important to get other teams on-board and ensure you have enough manpower to execute your plans.

However, we also understand that many teams don’t have the resources to commit to holiday testing.

Brooks Bell offers seasonal staff augmentation services that enable companies to temporarily bring on Brooks Bell developers and analysts to help you building and launching tests and implementing winners during peak season.

We also offer a Holiday Website Optimization Bootcamp to help companies like yours create a rock-solid plan for testing during the holidays.

Finally, here’s what to do if “No” really means no

If, after all this, if your leadership still is being a grinch about testing during the holidays, there are plenty of other ways you can maximize the holiday season to learn about your customers.

Use the lull in testing to devote resources to conducting usability studies. Hold focus groups to learn more about your customer mindsets and experience needs. You can also interview your sales and customer service teams to learn about the issues customers are coming across.