A/B Testing for UWP Apps: Experiment for Success Microsoft Build 2016 12/3/2018 8:16 PM A/B Testing for UWP Apps: Experiment for Success Jason Watson Sr Program Manager, Windows Dev Center © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12/3/2018 8:16 PM A/B Testing Scenarios Test variations of your app to discover optimizations that boost conversions and engagement: In-app offers Ad placements User flow Sign-up experience Game level configuration Make data-driven decisions about app changes © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Example Can I boost conversions by changing in-app offer text? 12/3/2018 8:16 PM Example Can I boost conversions by changing in-app offer text? Would “Disable Ads Now!” encourage more clicks? My conversion rate is 9% today.... © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Winning Variation 12/3/2018 8:16 PM 4514 users (Variation A) 4512 users (Variation B) 11% conversion rate 45% conversion rate Winning Variation © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Winning Variation 12/3/2018 8:16 PM 4514 users (Variation A) 4512 users (Variation B) 11% conversion rate 45% conversion rate Winning Variation © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Winning Variation 12/3/2018 8:16 PM 9026 users (Variation B) 45% conversion rate Winning Variation © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12/3/2018 8:16 PM How does it work? Update code in your app once to accommodate future experiments Add reference to the Microsoft Store Engagement and Monetization SDK Build app logic around A/B testable settings Create logging events for user views and conversions Use Windows Dev Center to run experiments Run experiments to identify winning app variations Switch your audience to winning app variations © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo #1: Update App Code for Experimentation Microsoft Build 2016 12/3/2018 8:16 PM Demo #1: Update App Code for Experimentation Jason Watson © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo #2: Act on A/B Test Results Microsoft Build 2016 12/3/2018 8:16 PM Demo #2: Act on A/B Test Results Jason Watson © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Advantages Experiment with your audience using one code base 12/3/2018 8:16 PM Advantages Experiment with your audience using one code base Identify app variations that boost conversions Switch your audience to a variation in two clicks © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Best Practices Maximize your test sample sizes 12/3/2018 8:16 PM Best Practices Maximize your test sample sizes Use split distributions for variation assignments Limit your experiment to two variations, including the control group Run experiments for at least 2 weeks Do peek at data after 1 day, but don’t peek afterwards Allow for time to collect statistically significant data Data from specific time of day can skew results (ex. Saturday mornings) © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Build 2016 12/3/2018 8:16 PM Call to Action Click on Experimentation in Windows Dev Center to get started. Download the Microsoft Store Engagement and Monetization SDK: http://aka.ms/store-em-sdk Re-visit Build on Channel 9. Continue your education at Microsoft Virtual Academy online. © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12/3/2018 8:16 PM © 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.