Truly Understanding Minimum Detectable Effect

Now you know how to Predetermine Your Test Sample Size and How to Reach Statistical Significance & Power within Your Experimentation Programs. Our final blog concludes the series and will expand your understanding of how to use Minimum Detectable Lift, or Minimum Detectable Effect for optimum test design and results.

Any optimization analyst knows that calculating a test duration prior to launch is a necessary step in reaching a desired level of statistical confidence. Minimum Detectable Effect (MDE) – a number that represents the relative minimum improvement you seek to detect over the control – is also critical to determining an accurate test duration. (Note: MDE is also known as Minimum Detectable Lift (MDL), and the two terms can be used interchangeably in testing.)

If you’ve ever been confused by this particular input in test design, you’re not alone.  As an analyst, I even found it confusing when I first started testing.

Test Design & MDE
Theoretically, an A/B test could last forever without reaching a desired level of statistical significance, or confidence.  If the test has little to no impact compared to the control, the test could simply run until someone becomes frustrated and ends it. Similarly, individuals could become impatient with test progression if a website, or specific set of pages, are not attracting enough visitors to collect a sufficient amount of data. Creating a well-established test design from the start minimizes, if not removes, such frustrations because it relies on data to determine test duration.

The simplest way to think of MDE is the following – it is essentially the smallest possible change in your primary KPI that your experiment can detect with statistical certainty.

Ensuring MDE Success
There are often misconceptions associated with MDE because it is not necessarily a number that can be easily determined without historical testing data or upfront analysis.

For clarification, keep these facts top of mind:

  • MDE is not the lift, or conversion increase, that a testing professional wants to see.
  • MDE is not a guess.
  • MDE is an anticipated lift over the control that can be measured with a degree of certainty.
  • MDE is the smallest possible change that would be worth investing the time and money to design the test and implement the change permanently on the site.

The main idea behind MDE is that you will be able to detect a lift down to the specified level at the confidence and significance thresholds you set.  For example, if you set a 1% MDE, you will be able to detect a 1%, 3%, or even 10% change in the KPI.  It is also important to remember lower MDEs require increased traffic to ensure that a smaller lift is truly valid.

If your organization is establishing a new experimentation program, and historical data is not available, Brooks Bell recommends selecting an MDE from the table below.  We have found based on historical analysis, that MDEs vary by page type due to differences in the ability to impact customer behavior at different stages of the online shopping journey.

A 1% increase in Revenue Per Visit from the Cart Page, for example, is going to be much more valuable than a 1% increase in RPV from the Category Page because all revenue goes through the Cart whereas not all revenue passes through a Category Page.

Brooks Bell conducts thousands of online experiments for enterprise-level clients, and collects data from each one. The table below presents website MDE averages by page type.

It is considered best practice to use the lowest possible MDE when designing a test while still running it within an acceptable amount of time (covered in blog 1).  By selecting smaller, more conservative, MDE levels, testing programs can optimize the certainty of testing outcomes, learnings, and ultimately ROI.

 

Shana Braun, Optimization Analyst
At Brooks Bell, Shana specializes in online experimentation evaluation and strategy as well as website analytics and consulting. Shana has worked in the field of data analytics for over a decade, and she actually enjoys it. Her experience spans various data types and verticals including: CPG in-store retail tracking, digital marketing, eCommerce, and conversion optimization.  Shana earned her undergraduate degree from Penn State University and her MBA from Chatham University.