1. A/B Testing on the Obama Campaign Winning With Data

2. Winning with A/B Testing

3. What impact can testing have?

4. Testing = constant improvement  Little improvements add up  Improving 1% here and 2% there isn’t a lot at first, but over time it adds up

5. Tests upon tests upon tests  Subject & draft tests  Full-list tests  Background personalization tests  Review: Every piece of communication was an opportunity to test  A single email often had many tests attached

6. Lessons

7. Don’t Trust Your Gut Lesson #1

8. Don’t trust your gut  We don’t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience  There was this thing called the Email Derby…  If even the experts are bad at predicting a winning message, it shows just how important testing is.

9. Foster a culture of testing Lesson #2

10. The culture of testing  Check your ego at the door  Use every opportunity to test something  Compare against yourself, not against your competitors or “the industry”  Are you doing better this month than last month?  Are you doing better than you would have otherwise?

11. Use data to make the user experience more personal Lesson #3

12. Big data ≠ big brother  Testing allows you to listen to your user base  Let them tell you what they like  Whether through A/B testing or behavioral segmentation, optimization gives them a better experience  Usually, the interactions that are the most human are the ones that win

13. Be human!  In general, we founds shorter, less formal emails and subject lines did best.  Classic example: “Hey”  When we dropped a mild curse word into a subject line, it usually won  “Hell yes, I like Obamacare”  “Let’s win the damn election”  “Pretty damn cool”

14. Good segmentation: behavioral  Behavioral segmentation was much more effective than demographic segmentation  Donor vs. non-donor  High-dollar vs. low-dollar  Volunteer status  What issues do people say they care about?  After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results

15. Conclusions

16.  Test everything, especially your gut instinct  Foster a culture of testing  Use data to make it personal