1. Evidence-Centered Game Design Kristen DiCerbo, Ph.D. Principal Research Scientist, Pearson Learning Games Scientist, GlassLab

2. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.2

3. The ability to capture data from everyday events should fundamentally change how we think about learning and assessment. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.3

4. Evidence-Centered Assessment Design What complex of knowledge, skills, or other attributes should be assessed? What behaviors or performances should reveal those constructs? What tasks or situations should elicit those behaviors? Mislevy, Steinberg, & Almond (2003)

5. Two Examples Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.5

6. Student Model - Aspire Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.6 Default Gateway Proficiency Understand the Purpose Obtain Navigate Devices Local vs. Remote Test Configuration Basic IP Addressing From Shute & Ventura, 2012

7. Task Model - Aspire Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.7

8. Evidence Model - Aspire Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.8

9. Student Model – Mars Generation One Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.9 LevelInterpretationExpression Achievement Preliminary Reason-1-I-A Identifies reasons people give to support a specific point Reason-1-E-A Generates at least one reason to support a specific point, in sentence form Foundational Reason-2-I-A Identifies supporting reasons or evidence in an argument and relates them to the point they support Reason-2-E-A Generates multiple reasons to support a point, and uses these reasons to counter others' argument in an engaging, familiar context Basic Reason-3-I-A Recognizes and explains the relationship between main and supporting points and keeps track of which evidence supports which point Reason--3-E-A Builds logical, hierarchically structured arguments by selecting and arranging reasons and evidence to support main and subsidiary points IntermediateReason-4-I-A Identifies specific points in a text that are vulnerable to objections and counterarguments Reason-4-E-A Creates simple critiques or rebuttals that critically provide summaries of or responses to other people’s arguments

10. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.10 Task Model – Mars Generation One

11. Game mechanics Matching game mechanics and learning mechanics “The nature of the construct being assessed should guide the selection of relevant task…” Sam Messick, 1994 A game is built around a core mechanic A mechanic is a set of verbs that create a core loop If the core doesn’t match the competency, the competency isn’t integrated. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.11 Find evidence Equip robot Battle robot Evaluate evidence Construct argument Critique argument

12. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.12

13. Evidence – Mars Generation One Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.13 skill milestones / tasksstandards (CCSS)game phases/actions Evidence action #1 (specific actions that provide evidence) identify 1 piece of data that relates to specific argument CCSS.ELA-Literacy.WHST.6-8.1a, CCSS.ELA-Literacy.W.6.1akeeping, sorting, discarding constructing dual-core claim/data pair for argubots (google doc: Loop 1 Processes, Stage 1.3) identify >1 piece of data that relates to a specific argument CCSS.ELA-Literacy.WHST.6-8.1a; CCSS.ELA-Literacy.WHST.6-8.1b; CCSS.ELA-Literacy.W.6.1b sorting, organizing pre- battle constructing dual-core claim/data pair for argubots identify data as pro/con CCSS.ELA-Literacy.WHST.6-8.1a, CCSS.ELA-Literacy.W.6.1a sorting, discarding, (not) using in battle accuracy of using the relevancy switch/toggle during dual-core construction identify data as valid/invalid CCSS.ELA-Literacy.RH.6-8.8 ; CCSS.ELA-Literacy.RST.6-8.8 sorting, discarding, (not) using in battle relevancy switch/toggle during dual-core construction identify common arugmentation schemes CCSS.ELA-Literacy.RH.6-8.8 ; CCSS.ELA-Literacy.RST.6-8.8 sorting, interaction w NPC/computer, ogranizing pre-battle, battle appropriateness of argubot selected for dual-core

15. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.15 Does it really get at what we think it does?

16. Cognitive Processes Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.16

17. 0:002:004:006:008:0010:0012:0014:0016:0018:0020:0022:0024:0026:00 Pair 11 Pair 10 Pair 9 Pair 8 Pair 7 Pair 5 Pair 3 Pair 6 Pair 4 Pair 2 Pair 1 Problem Identification Solution Generation/ Implementation Solution Evaluation DiCerbo, Frezzo, & Deng, 2011

18. Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.18 Thank You Kristen.DiCerbo@Pearson.com

19. References DiCerbo, K. E. & Behrens, J. T. (2014). Impact of the digital ocean on education. London: Pearson. Available: http://research.pearson.com/digitalocean DiCerbo, K. E., Frezzo, D. C., & Deng, T. (2011). Substantive validity of a simulation-based game. Research and Practice in Technology-Enabled Learning, 6(3), 161-185. Available: http://apsce.net/RPTEL/RPTEL2011NovIssue-Article2_pp161- 185.pdf Shute, V. J. & Ventura, M. (2013). Stealth assessment in digital games. White paper for MIT series, published by the MacArthur Foundation. Available: http://myweb.fsu.edu/vshute/pdf/white.pdfhttp://myweb.fsu.edu/vshute/pdf/white.pdf For the masochist: Mislevy et al. (2014). Psychometric Considerations in Game-Based Assessment. http://www.instituteofplay.org/work/projects/glasslab-research/ http://www.instituteofplay.org/work/projects/glasslab-research/ Copyright © 2011 Pearson Education, Inc. or its affiliates. All rights reserved.19