1. Playtesting and Metrics: Getting the Most out of Your Usability Testing Jordan Lynn Player Experience Researcher, Volition Inc.

2. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

3. Brief Note on Other Systems ● Ideal System + Integrated datalogging, player feedback, and video recording + Custom user interface + Significantly reduced analysis time

4. Brief Note on Other Systems Ideal System - Requires large investment of time and money up front

5. Clarification of Terms ● Playtesting ● Bringing in new players to experience your game and observe player behavior ● Datalogging ● Using back-end software solutions to measure player and NPC behavior

6. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

7. How Our Playtesting Works Recruit outside players for a week  Participants attempt a natural playthrough  Participants stop to fill out a brief survey occasionally

8. How Our Playtesting Works ● We learn ● What are players doing ● Why players are doing things ● How players feel ● We identify areas of bad frustration

9. Playtest Setup

10. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

11. How Our Datalogging Works Significant Event  Record status variables  Upload information to database Example: Player Death

12. How Our Datalogging Works ● Reliably record ● Who is performing ● What action ● When and Where ● Designer expectations: met or no?

13. How Our Datalogging Works ● Purpose ● During Production ● Inform game design ● Post Release ● Identify successes/failures ● Inform game design for future titles ● Address glitches/bugs with patches

14. How Our Datalogging Works ● Playtesting ● Record 100% ● Post Release ● Record 10%, drop to 1%

16. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

17. Benefits of Datalogging ● Identify Combat Imbalance

18. Combat Balance Spot the Difficulty Spike!

19. Combat Balance

20. Benefits of Datalogging ● Identify Combat Imbalance ● Identify Economy Imbalance

21. Economy Balance Most “Respect” comes from weapon purchases?

22. Economy Balance Mystery Solved.

23. Benefits of Datalogging ● Identify Combat Imbalance ● Identify Economy Imbalance ● Identify areas of excessively high failure

24. Mystery Death Spike

26. Benefits of Datalogging ● Identify Combat Imbalance ● Identify Economy Imbalance ● Identify areas of excessively high failure ● Identify weapon imbalances

27. Evolution of Weapon Balance Test 1

28. Evolution of Weapon Balance Test 2

29. Evolution of Weapon Balance Test 3

30. Benefits of Datalogging ● Identify Combat Imbalance ● Identify Economy Imbalance ● Identify areas of excessively high failure ● Identify weapon imbalances ● Improve workflow in other departments

31. SR: TT is Big 2.2 Million deaths illustrated

32. QA and Datalogging ● Productivity ● Tracking each team member

33. QA and Datalogging ● Productivity ● Tracking each team member ● Execution ● Check AI behaviors

34. QA and Datalogging ● Productivity ● Tracking each team member ● Execution ● Check AI behaviors ● Planning ● Verify that all features are being tested

35. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

36. Communication with Design ● Instruction-All designers need to know: ● System capabilities ● Exactly what the data means ● How NOT to use the data

37. Communication with Design ● Instruction ● Give designers direct access

38. Communication with Design ● Instruction ● Give designers direct access ● Rapid reporting is key

39. Planning Around Constraints ● Data analysis is time consuming

40. Planning Around Constraints ● Data analysis is time consuming ● Technical concerns of post-release data ● Sampling rate ● Massive amount of incoming data ● Storage capacity ● Analysis time

41. Planning Around Constraints ● Data analysis is time consuming ● Technical concerns of post-release data ● Datalogging works best with a single programmer

42. Planning Around Constraints ● Data analysis is time consuming ● Technical concerns of post-release data ● Datalogging works best with a single programmer ● Verify QA has adequate time to test

43. Relying on Player Feedback ● Test 1 Legend Shape: Playtester Color:Cause of death

44. Relying on Player Feedback ● Test 2 Legend Shape: Playtester Color:Cause of death

45. Relying on Player Feedback ● Test 3 Legend Shape: Playtester Color:Cause of death

46. Relying on Player Feedback ● Test 4 Legend Shape: Playtester Color:Cause of death

47. Relying on Player Feedback ● Post-Release Legend Size/Color: # of deaths

48. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

49. Resolving Ambiguous Feedback ● Example: “This level is too hard.” ● This is not actionable feedback.

50. Resolving Ambiguous Feedback Location 1: Bottleneck Location 2: Elevated Rocket turret

51. Conflicting Feedback

52. Real-Time Survey Modification

55. Outcome

56. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

57. Validate Data Using Feedback ● Datalogging can’t stand alone ● Example: Death does not equal difficulty in all cases.

58. Data Overload

59. Best Datalogging ROI Across Titles ● Combat Balance ● Player Death ● NPC Death

60. Best Datalogging ROI Across Titles ● Economy Balance ● Resources Collected ● Upgrades Purchased

61. Red Faction: Best Datalogging ROI

62. ● Ammo pickup ● Out of Ammo

63. Red Faction: Best Datalogging ROI ● GPS Request ● Combine with Player feedback to improve level layout and lighting.

64. Saints Row: Best Datalogging ROI

65. ● Mission Attempts ● Activity Attempts ● Failures ● Repeated missions

66. Reporting Data Design Leads Level Designers QA Producers Audio Systems Designers

67. Readable Data ● Name each participant in database ● Date and seat ● Example: Test May 1st, player 1 named 05_01_P1

68. Readable Data ● Name each participant in database ● Translate asset names ● “Infuego” > car_2dr_exoticsports04

69. Readable Data ● Name each participant in database ● Translate asset names ● Visualization software ● We use Tableau

70. Overview ● Intro ● Playtesting ● Datalogging ● Benefits ● Pitfalls ● Successes with Mixed Methods ● Tips & Tricks ● Outro

71. Questions, Suggestions, Lessons Jordan.Lynn@Volition-Inc.com