A/B Testing SaaS Products ProductCamp Provo March 29 th, 2014 Nate Carrier

A/B Testing: An Introduction 50/50 randomized split between two experiences Used in ▫Web development ▫Internet marketing ▫SaaS Products

Why A/B Testing? Web development and internet marketing ▫What metrics do you try to improve?  Conversion  Sales  ROI (on ad spend)  Engagement SaaS (cloud-based software) ▫Improve user experience (UX) ▫Increase engagement ▫Drive long-term profitability

When Should You A/B Test? Before introducing a new feature Small day-to-day improvement When you want to improve the customer exp. When you want to increase sales/subscriptions All the time!

How to A/B Test Define goal / question ▫Why are you running the test? Identify metrics ▫How do you identify a successful test? Design test experience Set up data collection ▫Google analytics (very limited), Adobe Marketing Cloud, custom, etc. Analyze data

Let’s Analyze Data Test and Control have different # of users! Google Analytics ▫ ▫Pw: ProductCamp Provo (with the space) ▫Shortcut: A/B Test on Voting Excel Data ▫  Limited data pushed into website database  User, Test Group, Post (at vote level)

Some Ideas of What to Look For Difference between Test & Control on: ▫Votes per Visitor ▫Visit duration ▫Sessions voted for (somewhat time consuming)

What Insights Have You Found? How do the test and control groups differ? ▫Number of votes per visitor  Test > Control ▫Visit duration  Test < Control ▫Different sessions voted for?

Statistical Significance Provides confidence in result ▫Insignificant: diff caused by random variation ▫Significant: most likely caused by something  Analytics and statistics only reveal correlation  Is that a bad thing? Can require more data than we can get Requires more skill to calculate ▫R▫R

Connect with linkedin.com/in/natecarrier