A M ODEL OF COLLABORATION FOR R EPORT D EVELOPMENT Tracey Hembry June 24, /24/2015 1

Developer vs. User Developer = testing company including psychometricians and content experts User = teachers, parents, administrators Commonly a disconnect Resulting reports are somewhat standard 6/24/2015 2

Goals Investigate best practices for reporting, specifically focused on: –Online reporting tools –Stakeholder input –Diagnostic classification models (DCMs) 6/24/2015 3

Overview of Activities 38 teachers participated –Meeting 1 = July 2014 –Meeting 2 = September 2014 –Meeting 3 = December 2014 Collaborated with developers –Initially, rough graphs and visualizations –Progressed to an interactive system prototype 6/24/2015 4

Meeting 1 Individual interviews – 90 minutes –Instructional decision-making process –Influence of various tools, like assessments Individual cognitive lab – 60 minutes –Review of several visualizations –Think-aloud activity (Teachers participated in both activities) 6/24/2015 5

Meeting 1 6/24/2015 6

Meeting 2 Student-level focus group – 90 minutes –Reviewed 3 student reports –Discussed instructional next steps Classroom-level focus group – 90 minutes –Reviewed 9 different data graphics –Forced group to pick “top 3” during activity (Teachers participated in 1 focus group activity) 6/24/2015 7

Meeting 2 6/24/2015 8

Meeting 3 Individual cognitive lab – 45 minutes –Navigated through the interactive system prototype –Answered questions about meaning of information and system organization System Usability Scale (SUS) 6/24/2015 9

Key Findings Results at a glance Threat of information overload Probabilities = hard 6/24/

Prototype Demonstration 6/24/

6/24/

6/24/

6/24/

6/24/

6/24/

Thank You 6/24/

6/24/

6/24/