Early last Sunday, most people were eagerly awaiting the start of a typically popular sporting event. Meanwhile, in Punxsutawney, Pennsylvania a famous rodent was crawling from his burrow and surveying the snowy landscape. It was a sunny morning and, before an estimated 20,000 onlookers, Punxsutawney Phil saw his shadow, casting a prediction for six more weeks of winter weather.
He was not alone in this forecast. Groundhogs including Stormy Marmot in Aurora, Colorado, Staten Island Chuck in New York, and Dover Doug in Pennsylvania all joined the “more winter” verdict. Meanwhile, several other groundhogs from New York to Manitoba fell the other way, forecasting an early spring.
Now, as the Northeast gets slammed with a series of winter storms, it seems as though Phil may have been right. But when the record of these groundhogs is scrutinized, their predictions unravel. Phil’s accuracy bounces between 32 and 39 percent, which is under the threshold of statistical significance—meaning random chance could produce similar results. After analysis of historical data, the NOAA National Climatic Data Center said Punxsutawney Phil “has shown no talent for predicting the arrival of spring…[and] competitor groundhogs across the nation fared no better.”
Forecasting, of course, is hard and weather services should understand that better than anyone. Sometimes, even when an observable storm is impending, various weather models—all based on a complex set of differential equations referencing laws of physics, chemistry, fluid dynamics and more—produce shockingly different predictions. In contrast, a groundhog provides a single, somewhat dubious, data point that is then extrapolated over a six-week period. And before you call the comparison silly, read that last sentence one more time. Starting to sound familiar?
What’s Your Groundhog?
Every organization has a groundhog or two—those persistent vanity metrics that keep coming up as a benchmark of success or failure. Depending on the primary goal, vanity metrics can include measures like hits, raw page views, downloads, or tweets per day. These are metrics that are easily manipulated and highly variable. It is very difficult to link actions to fluctuations in these numbers over a day, week, or even month—and they certainly cannot serve as a seed for a reliable forecast.
Instead of these groundhogs, organizations need to focus on actionable metrics—measures of actual behavior that can drive strategy. Instead of downloads, track active users. Instead of page views, track conversion rate. Eric Reis, author of The Lean Startup, describes actionable metrics like this:
Imagine you add a new feature to your website, and you do it using an A/B split-test in which 50% of customers see the new feature and the other 50% don’t. A few days later, you take a look at the revenue you’ve earned from each set of customers, noticing that group B has 20% higher revenue per-customer. Think of all the decisions you can make: obviously, roll out the feature to 100% of your customers; continue to experiment with more features like this one; and realize that you’ve probably learned something that’s particular valuable to your customers.
“Essentially, all models are wrong,” Statistician George Box famously wrote, “but some are useful.” Groundhog metrics unfortunately are neither accurate nor useful. However, by combining actionable measures with rigorous testing we can develop truly useful analytics that can drive strategy and sustainable growth. Then, maybe, we can let the groundhogs sleep undisturbed until spring.