1. Game Optimization through Large-Scale Experimentation Erik Andersen Ph.D. student Center for Game Science University of Washington Computer Science Dept.

2. Music Gameplay Sound effects Art

3. engagement retention revenue

4. A/B Testing

5. AB

6. Metrics ● Levels completed ● Time played ● Return rate

7. Aesthetics Secondary Objectives Tutorials

8. Foldit RefractionHello Worlds

9. Refraction

10. Rating: 3.8 / 5 400,000 plays

11. Hello Worlds!

12. Rating: 4.2 / 5 1,385,000 plays

13. Foldit

14. 240,000 players

15. Player Tracking ● Flash cache / login name ● New players only

16. Statistical significance 95% confidence level (p<0.05)

17. Experiment #1: Audio Sound Effects Music

18. Result: Music and sound effects did not matter

19. Experiment #2: Animations

20. Result: Animations improved engagement

21. No Animations Refraction

22. No Animations Refraction

23. No Animations Refraction

24. No Animations Hello Worlds

25. No Animations Hello Worlds

26. Experiment #3: Secondary Objectives (Super Mario Bros., Nintendo 1985)

29. Levels % of players 100 0

30. Levels % of players 100 0 With Coins Hypothesis

31. Levels % of players 100 0 With Coins Reality

34. Time % of players 100 0 What about time played?

35. Time % of players 100 0 With Coins Hypothesis

36. Time % of players 100 0 Hypothesis With Coins

37. Time % of players 100 0 With Coins Reality: Many Players Quit Sooner

38. Time % of players 100 0

41. Result: secondary objectives harmed engagement

42. (Assassin’s Creed, Ubisoft 2007)

43. Maybe easier is better? (Super Mario Bros., Nintendo 1985)

48. Time % of players 100 0 Off-path Coins

49. Time % of players 100 0 On-path Coins

52. Result: secondary objectives were good if they supported the primary objectives

53. Experiment #4: Tutorials

54. No Animations Foldit

55. No Animations Foldit

56. Refraction and Hello Worlds: no effect

57. Result: text tutorials helped only in the most complex game

58. Number of concepts 111324

59. Context-sensitivity Context-sensitiveContext-insensitive

60. Result: context-sensitive help was better (if tutorials helped at all)

61. No Animations Foldit

62. No Animations Foldit

63. Let’s gather more data ● Let us A/B test your games!

64. Coauthors Yun-En Liu Richard Snider Roy Szeto Seth Cooper Eleanor O’Rourke Jeff Lowdermilk David Truong Zoran Popović

65. Acknowledgements Kongregate: August Brown and Anthony Pecorella Logging: Kefan Xu Refraction Team: Erik Andersen, Yun-En Liu, Marianne Lee, Eric Butler, Brian Britigan, Stephen Sievers, Roy Szeto, Mai Dang, Christian Lee, Ethan Apter, Emma Lynch, Happy Dong, Zorah Lea Fung, Justin Irwen, Seth Cooper, François Boucher-Genesse, Zoran Popović Hello Worlds Team: Rich Snider, Michael Eng, Marianne Lee, Blake Thompson, Jeff Flatten Foldit Team: Seth Cooper, Adrien Treuille, Firas Khatib, Janos Barbaros, Joshua Snyder, Alex Cho Snyder, Jeff Flatten, Jeff Lowdermilk, Dun-Yu Hsiao, Jeehyung Lee, David Salesin, David Baker, Zoran Popović

66. eland@cs.washington.edu