1. A/B Testing SaaS Products ProductCamp Provo March 29 th, 2014 Nate Carrier

2. A/B Testing: An Introduction 50/50 randomized split between two experiences Used in ▫Web development ▫Internet marketing ▫SaaS Products

3. Why A/B Testing? Web development and internet marketing ▫What metrics do you try to improve?  Conversion  Sales  ROI (on ad spend)  Engagement SaaS (cloud-based software) ▫Improve user experience (UX) ▫Increase engagement ▫Drive long-term profitability

4. When Should You A/B Test? Before introducing a new feature Small day-to-day improvement When you want to improve the customer exp. When you want to increase sales/subscriptions All the time!

5. How to A/B Test Define goal / question ▫Why are you running the test? Identify metrics ▫How do you identify a successful test? Design test experience Set up data collection ▫Google analytics (very limited), Adobe Marketing Cloud, custom, etc. Analyze data

6. Let’s Analyze Data Test and Control have different # of users! Google Analytics ▫Email: ab@productcampprovo.org ▫Pw: ProductCamp Provo (with the space) ▫Shortcut: A/B Test on Voting Excel Data ▫http://bit.ly/1mdQYJz  Limited data pushed into website database  User, Test Group, Post (at vote level)

7. Some Ideas of What to Look For Difference between Test & Control on: ▫Votes per Visitor ▫Visit duration ▫Sessions voted for (somewhat time consuming)

8. What Insights Have You Found? How do the test and control groups differ? ▫Number of votes per visitor  Test > Control ▫Visit duration  Test < Control ▫Different sessions voted for?

9. Statistical Significance Provides confidence in result ▫Insignificant: diff caused by random variation ▫Significant: most likely caused by something  Analytics and statistics only reveal correlation  Is that a bad thing? Can require more data than we can get Requires more skill to calculate ▫R▫R

10. Connect with Me @nate_carrier linkedin.com/in/natecarrier njcarrier@gmail.com