1. A Holistic Approach to A/B Testing Jesse Hull Executive Director Technology, GSN Games

2. 2009

4. What Is A/B Testing? Randomly Assign Users We are the same Group BGroup A

5. What Is A/B Testing? Show Different Content A B

6. What Is A/B Testing? Measure Results AB

7. “ ”A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it.”

8. This Is Wrong!

9. Holistic A/B Testing You Can Do A Lot More Then Just Smooth Out The Details With…

10. Principle #1 Everyone Is Testing What Is Holistic A/B Testing?

11. Most Organizations Use These People To A/B Test

12. And They Come Up With Tests Like These AB (2.4% more revenue)

13. But You Want These People Testing

14. Test Big And Small Ideas Principle #2 What Is Holistic A/B Testing?

15. 10 Ideas That Each Improve Revenue 4% Why Small Ideas? +50%

16. 20 Ideas That Each Improve Revenue 4% Why Small Ideas? +120%

17. Why Big Ideas?

18. Example: Casino Games

19. Game Designer Tested Wildly Different Casino Game Mechanics

20. Scratch Madness

21. Roulette

22. Video Bingo

23. Blackjack

24. Video Bingo Won Time playing: +356% (over closest competition) Next day return rate: +42% Revenue: +238%

25. Engineers Tested Improving Load Time But That Was Just The Beginning

26. Time spent playing: +12.9% Next day return rate: +6.8% Revenue: +10.3%

27. Artists Tested Completely New UI

30. Revenue: +12.3% Bounce Rate: -25.8%

31. Enterprising Developer Tested Various Power-Ups

32. Revenue: +13.9% Three Extra Balls At The End

33. Revenue: +11% U Pick’em

34. Revenue: +65%! Next day return: unchanged

35. How Do You Build An Organization That Can Holistically A/B Test? In 3 Easy Steps…

36. Organization: Step 1 Build A Framework That Everyone Can Use

37. A/B Testing Framework Variations Common Testing Infrastructure Keep Variations

38. Common Testing Infrastructure Launching A New Test Description of the test Screen shots of UI Expected results (metrics of success)

39. Common Testing Infrastructure View Running Tests

40. Common Testing Infrastructure Standard Metrics And Tools For Evaluating Tests

41. Organization: Step 2 Avoid Common Mistakes

42. Common Mistakes Not Randomly Assigning Users To Group A Or B Should I ask users for their email address and send them a daily greeting?

43. Email Collection: Incorrect Group AGroup B This group is predisposed to like our application because they are willing to give us email

44. Email Collection: Correct Group AGroup B Application Unchanged

45. Common Mistakes Insufficient Data For Statistical Significance Usually enough data: Bounce rate (users that leave immediately) Percentage that finish tutorial Often not enough data: Number of payers Total revenue

46. Solution To Common Mistakes Hire A Statistician

47. Solutions To Common Mistakes Math (Statistics) Target 95% confidence Particle physics 99.999% confidence

48. Organization: Step 3 Build A/B Testing Into Your Culture

49. Set Team Goals In Terms Of Tests Perform 10 a/b tests focused on improving daily active users. The combination of these tests should improve DAU's by 20%.

50. Everyone In Organization Is Testing Game Designers

51. Everyone In Organization Is Testing Game Designers

52. Everyone In Organization Is Testing Game Designers

53. Everyone In Organization Is Testing Game Designers

54. Everyone In Organization Is Testing Game Designers

55. Artists Can Create News Feed Images

56. And Immediately See The Effect 89.3% click rate 4.3% install rate 23.5% click rate 42.0% click rate 8.9% install rate 12,039 publishes 130.2% click rate 0.4% install rate 34,084 publishes 41.2% click rate

57. Everyone In Organization Is Testing Engineers Bug fixes Performance improvements Architectural changes

58. Everyone In Organization Is Testing Marketing Test sending emails to only 50% of your user base Cross-promotion

59. Everyone In Organization Is Testing Human Resources Would love to do this (if we had enough people) Group A - Friday Beer Group B - Wednesday Lunch Which team Produces the most Highest employee retention Is most successful

60. How Do I Find Great Ideas? Hire great people Improve their intuition by testing

61. Aw, come on, baby. I wasn’t really cheating on you I was A/B testing. When Not To A/B Test

62. Technology limitations No users Long-term effects

63. Everyone in the organization should be testing all the time because...

64. the impact of your testing is limited only by the breadth and creativity of the ideas you test.

65. Our Company

66. Contact me: jhull@gsn.comjhull@gsn.com Lunch & Learn: Developer-led High Impact Game and Platform Experiments – 12:30p Today We Are Hiring! www.gsngames.com

67. "Do not believe in anything simply because you have heard it. Do not believe in anything simply because it is spoken and rumored by many.... But after observation and analysis, when you find that anything agrees with reason and is conducive to the good and benefit of one and all, then accept it and live up to it." -- Buddha

68. Questions?

69. Lady Gaga Leonardo da Vinci Lawrence (Artist GSN)

70. t test