1. CONFIDENTIAL The Wonders of A/B Tests Some info on what you hear those people down in Consumer Insights talking about from time to time…

2. CONFIDENTIAL What are A/B Tests? A randomized experiment between two variables (A and B!) Statistical hypothesis testing… –Is there a significant difference between versions? Goal is to identify potential changes to optimize a product/service, increase exposure/usage, etc

3. CONFIDENTIAL What are A/B Tests? Two versions of a ‘product’ (in our case, a Conversation) are compared –Identical except for one variation One version (A) stays the same (control) and version B is modified in some way (treatment) We (meaning Devon or Andrew) apply a randomization cue to the Conversation –People randomly see version A or B

4. CONFIDENTIAL Fun Fact Time! The term “A/B Test” was coined by Google scientists in 2000 They were trying to determine the optimum number of search results to display on a page…any guesses? Before then, they were known as “split tests” or “bucket tests”

5. CONFIDENTIAL When Should We Do a Test? When an aspect of a Conversation looks like it could be doing better –Is the bail rate at a particular question abnormally high? –Are people not completing the experience? –Are positive survey responses low? –Does something keep coming up in FIB feedback? –Etc. Or when we’re just curious and feel like learning!

6. CONFIDENTIAL What Should We Test? Once we determine some part of a Conversation isn’t performing as well as we’d like, we brainstorm how to test a different version/what variable to change –i.e.: A question has a much higher bail rate than others  Should we change the wording? Adjust the question order? Remove it? There can be multiple variables that can potentially influence our issue –Always have to test one at a time, otherwise you can’t determine which variable was the driving factor –However, you can get around this by setting up an A/B/C…N test, looking at each variable in an individual scenario

7. CONFIDENTIAL The A/B Test is Running, Now What? So we approached Devon or Andrew with a big smile and asked them to program this A/B test into the Conversation The test is now live We analyze and compare both versions and how people interact with each –Any changes in the issue we’re trying to address? Cool! Nothing significant? Consider trying a new variable/approach

8. CONFIDENTIAL What Have We Tested? Shorter Vs. Longer –i.e.: Questions; Introductions; Overall Experience/Conversation FIB (fill in the blank) Vs. Multiple Choice –i.e.: Number of employees Level of Detail –i.e.: “Have you traded before?” vs. “Have you traded financial markets before?” New Questions –i.e.: New first/intro question Language Use –i.e.: “Do you follow?” vs. “Now. Pop quiz!” Optional Early Opt Out –i.e.: To a contact form

9. CONFIDENTIAL What Have We Learned? (Best Practices) Be explicit about why a Conversation is relevant –i.e.: Deloitte OBA—convince veteran OBAs there’s helpful info for them; Dell EquaLogic—new first question Mention incentives –i.e.: Salesforce Edition Recommender—free is motivating Shorter is better—findings are mixed –Tested this in several scenarios, but not always the best option Check for familiarity with content and offer a “skip/jump/contact form” option –i.e.: Dell EquaLogic; Salesforce Recommender

10. CONFIDENTIAL What Have We Learned? (Best Practices) But don’t “invite” people to skip the survey –i.e.: Aetna Individual—less people skipped the survey when not told “of course, we can skip right to your summary” Use multiple choice over FIB whenever possible

11. CONFIDENTIAL Other Tips Devon or Andrew can pull out particular moments/data points/ questions relevant to the A/B test and create a new report with only that data Look at the Conversation in Talkshow –Especially helpful for expired Conversations no longer online –Debug/Debug Path allows you to “preview” the Conversation Don’t be afraid to ask questions! ;)

12. CONFIDENTIAL