1. A M ODEL OF COLLABORATION FOR R EPORT D EVELOPMENT Tracey Hembry June 24, 2015 6/24/2015 1

2. Developer vs. User Developer = testing company including psychometricians and content experts User = teachers, parents, administrators Commonly a disconnect Resulting reports are somewhat standard 6/24/2015 2

3. Goals Investigate best practices for reporting, specifically focused on: –Online reporting tools –Stakeholder input –Diagnostic classification models (DCMs) 6/24/2015 3

4. Overview of Activities 38 teachers participated –Meeting 1 = July 2014 –Meeting 2 = September 2014 –Meeting 3 = December 2014 Collaborated with developers –Initially, rough graphs and visualizations –Progressed to an interactive system prototype 6/24/2015 4

5. Meeting 1 Individual interviews – 90 minutes –Instructional decision-making process –Influence of various tools, like assessments Individual cognitive lab – 60 minutes –Review of several visualizations –Think-aloud activity (Teachers participated in both activities) 6/24/2015 5

6. Meeting 1 6/24/2015 6

7. Meeting 2 Student-level focus group – 90 minutes –Reviewed 3 student reports –Discussed instructional next steps Classroom-level focus group – 90 minutes –Reviewed 9 different data graphics –Forced group to pick “top 3” during activity (Teachers participated in 1 focus group activity) 6/24/2015 7

8. Meeting 2 6/24/2015 8

9. Meeting 3 Individual cognitive lab – 45 minutes –Navigated through the interactive system prototype –Answered questions about meaning of information and system organization System Usability Scale (SUS) 6/24/2015 9

10. Key Findings Results at a glance Threat of information overload Probabilities = hard 6/24/2015 10

11. Prototype Demonstration 6/24/2015 11

12. 6/24/2015 12

13. 6/24/2015 13

14. 6/24/2015 14

15. 6/24/2015 15

16. 6/24/2015 16

17. Thank You tracey.hembry@alpinetesting.com 6/24/2015 17

18. 6/24/2015 18

19. 6/24/2015 19