Now you know how to Predetermine Your Test Sample Size and How to Reach Statistical Significance & Power within Your Experimentation Programs. Our final blog concludes the series and will expand your understanding of how to use Minimum Detectable Lift, or Minimum Detectable Effect for optimum test design and results. Any optimization analyst knows that calculating a test duration prior to launch is a necessary step in reaching a desired level of statistical confidence. Minimum Detectable Effect (MDE) – a number that represents the relative minimum improvement you seek to detect over the control – is also critical to determining an accurate test duration. (Note: MDE is also known as Minimum Detectable Lift (MDL), and the two terms can be used interchangeably in testing.) If you’ve ever been confused by this particular input in test design, you’re not alone. As an analyst, I even found it confusing when I first started testing. Test Design & MDE Theoretically, an A/B test could last forever without reaching a desired level of statistical significance, or confidence. If the test has little to no impact compared to the control, the test could simply run until someone becomes frustrated and ends it. Similarly, individuals could become impatient with test progression if a website, or specific set of pages, are not attracting enough visitors to collect a sufficient amount of data. Creating a well-established test design from the start minimizes, if not removes, such frustrations because it relies on data to determine test duration. The simplest way to think of MDE is the following – it is essentially the smallest possible change in your primary KPI that your experiment can detect with statistical certainty. Ensuring MDE Success There are often misconceptions associated with MDE because it is not necessarily a number that can be easily determined without historical testing data or upfront analysis. For clarification, keep these facts top of mind:
- MDE is not the lift, or conversion increase, that a testing professional wants to see.
- MDE is not a guess.
- MDE is an anticipated lift over the control that can be measured with a degree of certainty.
- MDE is the smallest possible change that would be worth investing the time and money to design the test and implement the change permanently on the site.