Learning from Obama: Redesigning Analytics

 In 2008, Obama campaign raised $750 million  Would not be enough in 2012 The fundraising challenge Not impressed. $750 million?

The fundraising challenge  But fundraising was proving more difficult in 2012 than in 2008  President less available for fundraising events  In early campaign, we saw average online donation was half of what it had been in 2008  We had to be smarter, and more innovative

Overview  A/B testing in Obama’s digital department  Lessons learned  Don’t trust your gut  Foster a culture of testing  Make it personal

Winning with A/B Testing

Example: Draft language

What impact can testing have? versionSubject linedonorsmoney v1s1Hey263$17,646 v1s2Two things:268$18,830 v1s3Your turn276$22,380 v2s1Hey300$17,644 v2s2My opponent246$13,795 v2s3You decide222$27,185 v3s1Hey370$29,976 v3s2Last night307$16,945 v3s3Stand with me today381$25,881 v4s1Hey444$25,643 v4s2This is my last campaign369$24,759 v4s3[NAME]514$34,308 v5s1Hey353$22,190 v5s2 There won't be many more of these deadlines273$22,405 v5s3What you saw this week263$21,014 v6s1Hey363$25,689 v6s2Let's win.237$17,154 v6s3Midnight deadline352$23,244 Full send (in millions)  $2.2 million additional revenue from sending best draft vs. worst, or $1.5 million additional from sending best vs. average Test sends

Test every element  After testing drafts and subject lines, we would split the remaining list and run additional tests  Example: Unsubscribe language VariationRecipsUnsubs Unsubs per recipient Significant differences in unsubs per recipient 578, %None 578, %Smaller than D4 578, %Smaller than D4 580, %Larger than D3 and D4

No, really. Test every element.  We also were always running tests in the background via personalized content

Then, keep testing  Example: how much should we send? +6 s per week

The results  Campaign raised over one billion dollars  Raised over half a billion dollars online  Over 4 million Americans donated  Recruited tens of thousands of volunteers, publicized thousands of events and rallies  Did I mention raising >$500 million online?  Conservatively, testing probably resulted in ~$200 million in additional revenue

Lessons

Don’t Trust Your Gut Lesson #1

Don’t trust your gut  We don’t have all the answers  Conventional wisdom is often wrong  Long-held best practices are often wrong  You are not your audience  There was this thing called the Derby…  If even the experts are bad at predicting a winning message, it shows just how important testing is.

Experiments: Ugly vs. Pretty  We tried making our s prettier  That failed  So we asked: what about ugly?  Ugly yellow highlighting got us better results

Foster a culture of testing Lesson #2

The culture of testing  Check your ego at the door  Use every opportunity to test something  Compare against yourself, not against your competitors or “the industry”  Are you doing better this month than last month?  Are you doing better than you would have otherwise?

When in doubt, test  In a culture of testing, all questions are answered empirically  Example: With the ugly yellow highlighting, we worried about the novelty factor  Maybe highlighting would only work for a short time before people started ignoring it (or being irritated by it).  We decided to do a multi-stage test across three consecutive s

The ugly highlighting experiment  Experimental design:  Determined through this test that novelty was indeed a factor Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 7 Group 8 First Second Third

Use data to make the user experience more personal Lesson #3

Big data ≠ big brother  Testing allows you to listen to your user base  Let them tell you what they like  Whether through A/B testing or behavioral segmentation, optimization gives them a better experience  Usually, the interactions that are the most human are the ones that win

Be human!  In general, we founds shorter, less formal s and subject lines did best.  Classic example: “Hey”  When we dropped a mild curse word into a subject line, it usually won  “Hell yes, I like Obamacare”  “Let’s win the damn election”  “Pretty damn cool”

Good segmentation: behavioral  Behavioral segmentation was much more effective than demographic segmentation  Donor vs. non-donor  High-dollar vs. low-dollar  Volunteer status  What issues do people say they care about?  After using A/B tests to create a winning message, we could tweak it slightly for various behavioral groups and get better results

Experiments: Personalization  Adding “drop-in sentences” that reference people’s past behavior can increase conversion rates  Example: asking recent donors for more money  Added sentence significantly raised donation rate  Confirmed in several similar experiments …it's going to take a lot more of us to match them. You stepped up recently to help out -- thank you. We all need to dig a little deeper if we're going to win, so I'm asking you to pitch in again. Will you donate $25 or more today? …it's going to take a lot more of us to match them. Will you donate $25 or more today?

Conclusions

 Test everything, especially your gut instinct  Foster a culture of testing  Use data to make it personal