We have updated our Privacy Policy and Privacy Options

Got It

Multivariate Testing… In a Crockpot?


We usually do online testing in one of two ways: as a simple A/B test that pits a new test concept against the control, or as a multivariate test to measure the effect of numerous possible combinations. So when our Brooks Bell team recently decided to host our first chili cook-off, it was only appropriate that we held this one-of-a-kind culinary smackdown in true testing style.

Our team thrives on competition. It’s what makes us want to always beat the control in our online testing. If we wanted to keep this chili cook-off as a clean A/B test, all contestants could have used the same predetermined ingredients in different ways in order for the judges to choose a winner amongst the entries. However, we chose to take the multivariate approach and allow all contestants to use whatever ingredients they liked best in their chili.

Those who wanted to compete brought a crockpot full of their best chili recipe. Those who didn’t want to compete served as judges– and our very own testing platform was formed. We did need to keep the test clean by establishing a universal scorecard, which included the following judging criteria (or key performance metrics, if you will):

  • Taste – Is the taste pleasing to your palate?  Do you notice the different flavors?
  • Presentation – Is the presentation attractive and pleasing to the eye?  Is there a level of creativity to how it’s presented?
  • Consistency – Is the meat (if applicable) tender?  Are the beans cooked well, or are they overcooked or undercooked?
  • Aroma – Does the aroma smell good to you?  Can you detect different layers of smells?  Is the smell appetizing?
  • Aftertaste – How is the aftertaste?  Does it leave a lingering, pleasant taste or does it finish worse than it started?

We also offered a Heat Bonus – with 1 being super mild and 10 being “en fuego,” judges rated the heat of the chili entry.

For this to be a clean multivariate test, we also had to take some other precautions among the contestants and judges so that judging couldn’t be swayed by bias. All entries were assigned a number and chili tastings were served in a bowl labeled with the corresponding anonymous number. Contestants plated their chili servings with whatever presentation elements they chose to use. Bowls were served and scores were collected by an impartial party. Judges were required to cleanse their palate in between tastings. Yes, it was that intense.

At the end of the day, both A/B and multivariate testing are about determining a winner, and we did just that.  So we’d like to congratulate the following winners of our first Brooks Bell Chili Cook-Off Smackdown:

Gregory Ng – Best Overall Chili

Gregory Ng – Hot Stuff

Suzi Tripp – Best Presentation

To see some pictures of our chili cook-off smackdown, check out our Flickr set. Also, special thanks to Mike, who put together a time-lapse video of the judging: