Serious Play Conference Los Angeles, CA July 22, 2014 Girlie C. Delacruz Games as Assessment: Validating Patterns of Gameplay Via Cluster Analysis

ENGAGE Team © Regents of the University of California Eva Baker – Principal Investigator Girlie Delacruz – Project Director Greg Chung – Project Director Noelle Griffin – Associate Director Ayesha Madni – Senior Researcher Markus Iseli – Senior Researcher Alan Koenig – Senior Researcher

Typical Metrics

Determine Proficiency

Diagnostic Assessment

Games As Formative Assessment Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Formative Assessment: Use and interpretation of task performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).

Games As Formative Assessment Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967). Games as Formative Assessment: Use and interpretation of game performance information with intent to adapt learning, such as provide feedback. (Baker, 1974; Scriven, 1967).

Qualitative Metrics

ENGAGE

Project Goals

ENGAGE Games

Go Vector Go

Current Study: Validating Patterns of Gameplay

Research Questions 1.What patterns of gameplay are identified using cluster analysis? 2.What is the association between identified clusters and performance on the related item administered the pretest?

Force and Motion: Constant Velocity Big Idea A change in the position and motion of an object is a result of the manipulation of forces acting on an object. Example Learning Outcome Manipulate friction, mass, force magnitude, force direction, and slope to cause an object to move at constant velocity.

Pretest Item

Diagnostic Options 1. No changes in forces. 2. Vector know what to do now. 3. A lot of forces. 4. A lot of energy. Prompt Why does Vector’s speed not change?

Cluster Analysis

Validity Evidence for Gameplay Patterns

Choosing Clustering Variables LevelVector is…Need to…By… Initial Force Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt)

Results What patterns of behavior are identified using cluster analysis? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster Cluster Set 2: Levels 10 and 18 Cluster Cluster Cluster

Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster Cluster Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster Cluster Students in this cluster were over 3 times more like to choose the incorrect answer option “Vector knows what to do now.”

Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster Cluster Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster Cluster Students in this cluster were over 3 times more like to choose the incorrect answer option “Vector knows what to do now.”

Results What is the association between identified clusters and performance on the related item administered the pretest? Gameplay pattern of friction values chosen on first attempt (correct friction value should be ‘0’) Relation between gameplay pattern and pretest item performance Number of students in cluster Level 4 Level 10 Level 18 Set 1: Levels 4, 10, and 18 Cluster membership did not have any significant relation to student performance on the assessment item. Cluster Cluster Set 2: Levels 10 and 18 Cluster membership was a significant predictor of student performance on the assessment item. Cluster Students in this cluster were over 2 times more likely to get the item correct than students in Cluster 3. Cluster Cluster Students in this cluster were over 3 times more like to choose the incorrect answer option “Vector knows what to do now.”

What Does This Mean? LevelVector is…Need to…By… Initial Force Magnitude Initial Surface Friction 4 Not moving Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt) 10 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 2 (Grass) 18 Too slow Inc. speed Dec. friction 1 (Reverse Thruster) 1 (Dirt)

Implications: Validity Evidence for Gameplay Patterns

Next Steps

THANK YOU!!! © Regents of the University of California

Gameplay Patterns: Levels

Gameplay Patterns: Levels

Gameplay Patterns: Levels

Gameplay Patterns: Levels

Association of Cluster Membership to Pretest Performance

Association of Cluster Membership to Incorrect Answer Options Chosen 2. Vector know what to do now.