1. A/B Testing – can you afford not to? #13NTCab Hugh Dwyer, International Rescue Committee Jacob Colie, Stage Coach Digital Tom Williamson Jackson River

2. Today’s Agenda Introductions What is A/B Testing? Testing Goals – what should we test? Testing versus Experimenting Crafting an A/B Test Plan Selling your plan Build, Measure, Learn Testing Tools Resources & Downloads QA Slide 1 A/B Testing

3. What is A/B Testing? A/B Testing: A/B testing is an experimental approach to web design which aims to identify changes to web pages that increase or maximize an outcome of interest (e.g., donation conversion). As the name implies, two versions (A and B) are compared, which are identical except for one variation that might impact a user's behavior. Slide 2 A/B Testing

4. Standard Donation Page Slide 3 A/B Testing

5. A/B Test Donation Page Slide 4 A/B Testing

6. What is A/B Testing? Multivariate Testing Multivariate testing is similar to A/B testing, but tests more than two different versions at the same time. Slide 5 A/B Testing

7. Multivariate Slide 6 A/B Testing

8. Challenges What are some common challenges to doing A/B Testing? Resources – technology, time and people Data sampling size Is my list big enough? Are my results significant? “Do no harm” Lack of existing test data to build on Risk vs. Reward To do A/B testing you need to have an Entrepreneurial Spirit! Slide 7 A/B Testing

9. A/B Testing Channels Channel Testing Email Advertising Web Offline/Direct Mail Integrated (On and Offline) Segmentation Slide 8 A/B Testing

10. Testing Goals – what should we test? Channels and Audience Segments Slide 9 A/B Testing Snail Mail Email Search Website/Landing Environment Advertising

11. Testing Goals – what should we test? Metrics and KPIs Establish benchmark metrics: what are you trying to beat? Average time on site/page Open rates Click-through rates Form conversion (donations, email signups, etc.) Average Gift Cost to acquire Cost per lead Slide 10 A/B Testing

12. Testing Goals – what should we test? Evaluating options Slide 11 A/B Testing Alignment with strategy, context Examine conversion funnels and donor lifecycle: where are your challenges? Consider potential returns, & calculate ROI Where can you move the needle the most? Quick wins and low hanging fruit

13. Testing Goals – what should we test? Variables Slide 12 A/B Testing Design elements Persuasion assets Offer elements Benefits Default ask Products Price points Ask string CTA Multi-step forms CTA Positioning Images Copy Media mentions Premiums Security seals Testimonials Achievements Page layout Expert ratings Subject lines Video Keywords Communications strategies Communications strategies Timing Frequency Cross-channel Usability Modal windows Content

14. Testing Goals – what should we test? Considerations Evolutionary vs. revolutionary changes Frequency and pace of testing Short-term vs. long-term Easy road vs. hard road Embrace (calculated) risks Slide 13 A/B Testing

15. Testing vs. EXPERIMENTS Slide 14 A/B Testing

16. Slide 15 A/B Testing When to test? When to experiment?

17. Slide 16 A/B Testing GOALS Accelerate your REVENUE Provide VALUE for your users

18. Slide 17 A/B Testing The DIGITAL REVENUE CYCLE

19. BUILD. MEASURE. LEARN* Slide 18 A/B Testing * Read THE LEAN STARTUP

20. BUILD an MVP Slide 19 A/B Testing

21. MEASURE revenue and engagement Slide 20 A/B Testing

22. LEARN what did and didn’t work Slide 21 A/B Testing

23. Baseline IDEAL Slide 22 A/B Testing

24. Slide 23 A/B Testing vs.

25. short-term vs. LONG-TERM Slide 24 A/B Testing

26. 5% vs. 35% Slide 25 A/B Testing

27. Examples Slide 26 A/B Testing

28. MERCY CORPS Retention Slide 27 A/B Testing

29. IRC Rescue Gifts Slide 28 A/B Testing

30. Getting started with A/B Testing KISS – Keep it simple silly Define metrics to test Sustainer Acquisition Average Gift Gift String Conversion %, etc. Identify your top 3-5 items to test, and schedule them over X months! Decide how you will test each scenario Control group Minimum data required to validate Resources necessary Internal communications Implement each test Build, measure, learn – be patient Slide 29 A/B Testing

31. Individual Test Cycle Slide 30 A/B Testing Identify KPI/Metric How will you test? Coordinate Resources Implement Measure, Report, Analyze

32. Selling your plan Slide 31 A/B Testing Objectives Inputs Plan Developing a business case Analysis Recommendation

33. Resources & Downloads Presentation Materials PowerPoint - www.jacksonriver.com/nten/ab_testingwww.jacksonriver.com/nten/ab_testing Third Party Tools: Optimizely - www.optimizely.comwww.optimizely.com Visual Website Optimizer - http://visualwebsiteoptimizer.comhttp://visualwebsiteoptimizer.com Google Content Experiments 8 Seconds - http://www.8seconds.nethttp://www.8seconds.net Slide 32 A/B Testing

34. Questions and Answers Contact info Hugh Dwyer Sr. Director, Information & Operations, International Rescue Committee hugh.dwyer@rescue.org Jacob Colie Senior Strategist & Principal, Stagecoach Digital jacob@stagecoachdigital.com @jacobcolie Tom Williamson CEO, Jackson River tom.williamson@jacksonriver.com @tomwilliamson, @jacksonriver Slide 33 A/B Testing

35. Evaluate This Session! Enter for a chance to win an NTEN engraved mini iPad! Or, search by session title at www.nten.org/ntc/eval