A/B Testing on the Obama Campaign Winning With Data

Winning with A/B Testing

What impact can testing have?

Testing = constant improvement  Little improvements add up  Improving 1% here and 2% there isn’t a lot at first, but over time it adds up

Tests upon tests upon tests  Subject & draft tests  Full-list tests  Background personalization tests  Review: Every piece of communication was an opportunity to test  A single often had many tests attached

Lessons

Don’t Trust Your Gut Lesson #1

Don’t trust your gut  We don’t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience  There was this thing called the Derby…  If even the experts are bad at predicting a winning message, it shows just how important testing is.

Foster a culture of testing Lesson #2

The culture of testing  Check your ego at the door  Use every opportunity to test something  Compare against yourself, not against your competitors or “the industry”  Are you doing better this month than last month?  Are you doing better than you would have otherwise?

Use data to make the user experience more personal Lesson #3

Big data ≠ big brother  Testing allows you to listen to your user base  Let them tell you what they like  Whether through A/B testing or behavioral segmentation, optimization gives them a better experience  Usually, the interactions that are the most human are the ones that win

Be human!  In general, we founds shorter, less formal s and subject lines did best.  Classic example: “Hey”  When we dropped a mild curse word into a subject line, it usually won  “Hell yes, I like Obamacare”  “Let’s win the damn election”  “Pretty damn cool”

Good segmentation: behavioral  Behavioral segmentation was much more effective than demographic segmentation  Donor vs. non-donor  High-dollar vs. low-dollar  Volunteer status  What issues do people say they care about?  After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results

Conclusions

 Test everything, especially your gut instinct  Foster a culture of testing  Use data to make it personal