pg 1 Design Research at GlassLab March 25 th, 2014

G and g for Formative Assessment pg 14

Cognitive and Non-cognitive Constructs GlassLab is exploring the measurement of a variety of cognitive and non-cognitive constructs Systems Modeling and Thinking Argumentation Mathematical modeling Problem Solving Engagement Each GBA is required to address a constellation of competences related to CCSS and/or NGSS as well as 21 st century skills

Learning Progressions (LPs) and GBA LPs provide a framework to connect and organize Learning goals Game levels and sequencing Assessment construct and psychometric models Provide a basis for fast cycle and medium cycle within game feedback and outside of game longer cycle feedback

What’s Notable: an integrated game- based assessment development process pg 21

pg 22 Evidence Centered Game Design (ECgD)

ECgD Because the claims are about helping kids learn: o the competency model is expressed as a set of Learning Progressions o the evidence model is embedded in design docs along with game and learning components that is organized around an overall instructional pattern o the detailed evidence and task design is embedded in design docs in conjunction with the detailed game and learning designs (MicroDesign, GDD, etc) along with details in core assessment design doc for (scoring, measurement models, etc). pg 23

Production and Data Collection Cycle Most Changes to Cognitive and Scoring Models Most Changes to Game Play and Mechanics Most Changes to Bayes’ Priors

Phase Studies (N= per task) Mid Concept/early pre-pro Think- aloud N = 10 Play- testing N = 10 Mini- tryouts N = 50 Large-scale tryouts N = 200+ Late pre- production Late Production Field Test N=1000+ (Beta) Convenience sample User Studies: Why, size, outcomes, time frame, data Develop early hypotheses about cognition, performance, and behaviors Examine task performance and early evidence of Behaviors. Confirm hypotheses on cognition. Draft scoring algorithms. Careful sampling Analyze performance and observable behaviors. Evaluate scoring algorithms. Careful sampling Verbalizations Item responses Interactions Item responses Interactions Item responses Outcomes Types of Data Verbalizations Gather direct evidence for cognitions underlying performance and interactive behaviors; confirm construct validity Option: Verbal Protocol N = 20 Calibrate psychometric models. Refine feedback Beta pg 27