1. Judeth Oden Choi, Jodi Forlizzi, Michael Christel, MacKenzie Bates, Rachel Moeller, Jessica Hammer Carnegie Mellon University Playtesting Workshops: Playtesting To Refine 1

2. Recon Activity Walk around the room and read other teams’ recipes. Leave 3 “That’s cool!” comments on sticky notes. Leave 3 “I’m curious or confused” questions on sticky notes. 2

3. Refine Is For You If: You are working on a prototype (lo-fi, hi-fi, a mod of an existing game, a single mechanic, or a whole experience) and You know your player experience and/or design goals 3

4. Why Playtest to Refine? What you know: You have a set of game/experience features you want to use You know the desired affect you want your experience to have. What you want to learn: How does this set of game features affect the experience? How can I improve the experience? How can I improve my goals? 4

5. Recipes & Sketches You already selected ingredients for your prototype that connect to your player experience goals. 5

6. Why Recipes? A prototype is a hypothesis. You are hypothesizing that your design (your set of ingredients) will create a certain experience. You can test this hypothesis through playtesting. Use your recipe to formulate questions. How do your ingredients affect the experience? Read the sticky notes on your recipe. Select (or write) three questions about your prototype. What are you curious about? Where do the uncertainties lie? 6

7. Methods in Refine Observe Think-Aloud Interview Survey to capture behavior to capture thoughts, process to capture motivations, feelings, attitudes to capture structured data 7

8. Observation 8

9. Observation Tips Look for ACTIONS Vocalization Gestures Reactions Look for CHANGE Shifts in posture Change of facial expression Shifts in focus (or eye contact) 9

10. Notetaking Tips Situate notes within game Use time codes, level, or event Withhold Interpretation during test Do NOT Interrupt. If you must help the player, make a note of it. 10

11. Let me repeat DO NOT INTERRUPT …don’t congratulate …don’t help …don’t explain the UI …don’t apologize …unless the player is really, really stuck. ( If they do get stuck, make a note of it. It’s really valuable information!) 11

12. Recording Tips Use video and audio to capture behavior. Use video or in-game tools to capture interface. Use photos to document moments. Consider framing. Give players space. Consider their comfort. Get permission. Review and notate immediately after playtest 12

13. Uno Playtest Player experience goal: competitive social interaction. Modified the mechanics to create 4 prototypes How do the mechanics affect the player experience? Can you observe a difference? 13

14. Think Aloud http://www.doctordisruption.com/design/design-methods-27-think-aloud-protocol/ 14

15. Think Aloud 15

16. Interviews Photo by Heinrich-Böll-StiftungHeinrich-Böll-Stiftung 16

17. Ask good questions Questions should be: 1.Important 2.Accurate 3.Answerable 17

18. Ask Important Questions Important questions relate to your player experience goal. 1.Did you like the music? 2.What feelings do you associate with the music? 3.What parts of the game made you feel _(confident)__? 4.Which feelings hadn’t the greatest impact on your ability to learn _(fractions)_? 1.Uninformative 2.Important if your goal is to iterate the music 3.Important to help assess current success status of game 4.Important … but difficult to test 18

19. Ask Accurate Questions Accurate questions address the player's relationship with the goal. 1.Is this a good game? 2.Is the game fun? 3.Why is this game fun? 4.I noticed you laughed during x? What was going through your mind? How did you feel? 5.I noticed it took several tries to get through x? How did it feel to encounter that challenge? 1.What does good mean? 2.How do you identify fun? 3.Will the player even know? 4.Measurable way of identifying player enjoyment, but only looking for positives 5.Measurable way of looking for player enjoyment and struggles 19

20. Ask Answerable Questions Answerable questions capture the aspect of your game you're focusing on. 1.How much did you like the game? 2.Did you like the game? 3.What did you think of the game? 4.What was your favorite moment in the game? 5.How would you describe this game to someone who had never played it before? 1.Assumes they liked it 2.Suggests yes/no answers 3.Open-ended, good for exploring 4.Concrete examples can get at deeper thoughts 5.Teach-back technique helps players be more precise and accurate themselves 20

21. Writing Interview Questions Refer back to the questions you generated at the beginning of the workshop. Write an interview question that you might ask a player after a playtest of your prototype. 21

22. Surveys 22

23. Surveys Benefit: easy to automate, pre-structured data Challenge: writing good questions is hard! Flaw: GIGO (Garbage In, Garbage Out) Strategies Iterate questions over the semester Try out questions on friends and advisors Include multiple-choice and open questions Keep it short 23

24. Likert Scales Degree of agreement Can also measure frequency, importance, quality, likelihood Use 5 or 7 categories Clearly and accurately labeled Low to high, arranged left to right Equal number of positive and negative 24

25. Survey Bias: Leading (1) How fast was car A going when it turned right? (2) Did you see a stop sign? 35% answered yes to (2) (x) How fast was car A going when it ran the stop sign? (y) Did you see a stop sign? 53% answered yes to (y) 25

26. Survey Bias: Anchoring 26

27. Survey Bias: Anchoring 27

28. Survey Bias: Anchoring 28

29. Survey Bias: Anchoring https://www.youtube.com/watch?v=HefjkqKCVpo 29

30. Validated Survey Questions Pre-existing surveys: Immersive Experience Questionnaire (IEQ) Player Experience of Need Satisfaction (PENS) Game Experience Questionnaire (GEQ) Use validated survey questions ONLY IF they align with your questions. 30

31. Writing Survey Questions Writing accurate and answerable questions: Writing important and relevant questions: 31

32. Writing Survey Questions Phrase one of your questions as a survey question. Will you use… ● Multiple choice ● Likert scale ● Open response Share your question! 32

33. Planning a Playtest To Refine 33

34. Step 1: Why are you testing? 1. Identify your goals. 2. Identify your questions. How does this set of game features affect the player experience? How can I improve the experience? How can I improve my goals? 34

35. Step 2: Select Methods 1. Observation Notetaking Photography Video & audio recording In-game measures 2. Think-Aloud Protocol 3. Interview Ask important, accurate & answerable questions 4. Surveys Iterate, iterate iterate Reduce bias 35

36. Step 3: Find Players 1. Do you have access to your target audience? If not, what’s the next best thing? 1. Consider your playtesters comfort What is their comfort-level with the technology? With the the type of experience? With being observed? 36

37. Step 4: Select a Setting In the wild or the lab? 37

38. Step 5: Set-up 1. How many team members do you need? Who fills what role? 2. Where will you place the experimenters in relation to the players? 3. On-Boarding Write an introduction script. Prepare written instructions. Create mock-ups of unfinished interfaces. 4. How long is each playtest? How many rounds can you schedule? 38

39. Why playtest? Each prototype is a hypothesis. Keep testing and iterating on your hypothesis. 39

40. Check Out Contact us: playtest-project@lists.andrew.cmu http://playtestingworkshops.com 40