1. Serious Play Conference Los Angeles, CA July 22, 2014 Girlie C. Delacruz Games as Assessment: Validating Patterns of Gameplay Via Cluster Analysis

2. ENGAGE Team © Regents of the University of California Eva Baker – Principal Investigator Girlie Delacruz – Project Director Greg Chung – Project Director Noelle Griffin – Associate Director Ayesha Madni – Senior Researcher Markus Iseli – Senior Researcher Alan Koenig – Senior Researcher

3. Typical Metrics

4. Determine Proficiency

5. Diagnostic Assessment

6. Games As Formative Assessment Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).

7. Games As Formative Assessment Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).

8. Qualitative Metrics

10. ENGAGE

11. Project Goals

12. ENGAGE Games

13. Go Vector Go

14. Current Study: Validating Patterns of Gameplay

15. Research Questions 1.What patterns of gameplay are identified using cluster analysis? 2.What is the association between identified clusters and performance on the related item administered the pretest?

16. Force and Motion: Constant Velocity Big Idea A change in the position and motion of an object is a result of the manipulation of forces acting on an object. Example Learning Outcome Manipulate friction, mass, force magnitude, force direction, and slope to cause an object to move at constant velocity.

17. Pretest Item

18. Diagnostic Options 1. No changes in forces. 2. Vector know what to do now. 3. A lot of forces. 4. A lot of energy. Prompt Why does Vector’s speed not change?

19. Cluster Analysis

20. Validity Evidence for Gameplay Patterns

21. Choosing Clustering Variables LevelVector is…Need to…By… Initial Force Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt)

22. Results What patterns of behavior are identified using cluster analysis? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster 178--00 Cluster 246--03 Cluster 363--33

23. Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster 178--00 Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster 246--03 Cluster 363--33 Students in this cluster were over 3 times more like to choose the incorrect answer option “Vector knows what to do now.”

24. Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster 178--00 Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster 246--03 Cluster 363--33 Students in this cluster were over 3 times more like to choose the incorrect answer option “Vector knows what to do now.”

25. Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster 1115300 Cluster 269303 Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster 178--00 Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster 246--03 Cluster 363--33 Students in this cluster were over 3 times more like to choose the incorrect answer option “Vector knows what to do now.”

26. What Does This Mean? LevelVector is…Need to…By… Initial Force Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt)

27. Implications: Validity Evidence for Gameplay Patterns

28. Next Steps

29. THANK YOU!!! © Regents of the University of California gdelacruz@cse.ucla.edu www.cresst.org

30. Gameplay Patterns: Levels 4 - 10 - 18

31. Gameplay Patterns: Levels 10 - 18

32. Gameplay Patterns: Levels 4 - 10 - 18

33. Gameplay Patterns: Levels 10 - 18

34. Association of Cluster Membership to Pretest Performance

35. Association of Cluster Membership to Incorrect Answer Options Chosen 2. Vector know what to do now.