1. C OPYRIGHT W EST E D, 2011 Integrating Simulation-Based Science Assessments into Balanced State Science Assessment Systems: Findings and Implications Richard Vineyard Nevada Department of Education Edys Quellmalz and Matt Silberglitt WestEd Presented to the Council of State Science Supervisors, San Francisco, CA, March 9, 2011 Enhanced Assessment Grant

2. C OPYRIGHT W EST E D, 2011 Enhanced Assessment Grant State Partners Nevada (Lead State) Utah North Carolina Massachusetts Connecticut Vermont

3. C OPYRIGHT W EST E D, 2011 Advisors Rebecca Kopriva, Wisconsin Center for Education Research – Assessments for English Learners Martha Thurlow, National Center on Education Outcomes – Assessments for Students with Disabilities

4. C OPYRIGHT W EST E D, 2011 Subcontractors WestEd STEM (SimScientists) – Project Management – Content Development Redhill Studios – Interactive (Flash) Programming UCLA/CRESST – External Evaluation

5. C OPYRIGHT W EST E D, 2011 EAG Goals Establish the technical quality of simulation- based assessments – for two topics, ecosystems and force and motion Study the feasibility and utility of simulation- based assessments in classrooms from 3 states

6. C OPYRIGHT W EST E D, 2011 EAG Goals Study the measurement of science knowledge and inquiry by simulation-based curriculum-embedded assessments and unit benchmark assessments in comparison to traditional items for: – All students – ELLs – Students with disabilities Propose alternative models for integrating simulation-based assessments into state assessment systems

7. C OPYRIGHT W EST E D, 2011 Relevance to Current Assessment Programs Need innovative, technology-enhanced assessments that align with new frameworks that focus on – fewer, deeper, more integrated core knowledge targets E.g., the new Framework for Science Education and next generation national science standards Models as structures for understanding and studying science systems (model-based learning) Science practices for using knowledge and inquiry in significant, recurring authentic tasks

8. C OPYRIGHT W EST E D, 2011 Relevance to Current Assessment Programs Need innovative, technology-enhanced assessments that Target 21 st century skills within STEM Use technology to engage students in use of 21st century “tools of the trade” Provide evidence supporting one type of technology- enhanced performance assessment – For summative and formative purposes – Called for in the RTT Consortia

9. C OPYRIGHT W EST E D, 2011 EAG Design Align to national science standards and to science standards of the pilot states – Ecosystems and force & motion – At middle school level – Nevada, North Carolina, Utah Accommodations – Extended time – Audio – Zoom

10. C OPYRIGHT W EST E D, 2011 Ecosystem Target Model System Model Levels Model Level Descriptions Content Targets by Model Level Inquiry Targets Components What are the components of the system and their rules of behavior? Every ecosystem has a similar pattern of organization with respect to the roles (producers, consumers, and decomposers) that organisms play in the movement of energy and matter through the system. Use principles to identify role of organisms. Interactions How do the the individual components interact? Matter and energy flow through the ecosystem as individual organisms participate in feeding relationships within an ecosystem. Observe interactions among organisms. Emergent Behaviors What is the overall behavior or property of the system that results from many interactions following specific rules? Interactions among organisms and among organisms and the ecosystem’s nonliving features cause the populations of the different organisms to change over time. predict observe explain investigate

11. C OPYRIGHT W EST E D, 2011 Multiple Modes of Representation Active Inquiry Organism box shows icons that appear and disappear Graph shows size of population over time. Data inspector shows population for one point on graph. Table shows population at start, end, and point selected by data inspector.

12. C OPYRIGHT W EST E D, 2011 E MBEDDED IN C LASSROOM I NSTRUCTION Assessment Module Components for Each Topic/System Online assessment without feedback Teacher scores constructed responses Bayes Net Proficiency report Benchmark Summative Unit Assessments Online assessment with feedback and coaching Follow up Classroom Reflection Activity Progress report Embedded Assessments for Formative Purposes (2 to 4)

13. C OPYRIGHT W EST E D, 2011 Ecosystem Embedded Assessment: Feedback, Coaching, and Self-Assessment Demo

14. C OPYRIGHT W EST E D, 2011 Progress Reports to Students

15. C OPYRIGHT W EST E D, 2011 Progress Reports to Teachers NH = needs helpP = making progressOT = on track

16. C OPYRIGHT W EST E D, 2011 Grouping Recommendations for Classroom Reflection Activity

17. C OPYRIGHT W EST E D, 2011 Classroom Reflection Activity Formative use of assessment results – Students assigned to teams based on embedded results Transfer to different, more complex system Jigsaw structure – Allows differentiated instruction via tasks of varying difficulty – Promotes small and large group discourse and collaboration Guidance for teacher – Teacher review of key points in simulation – What to look for during group work and questions to pose in response – Presentations – Evaluation of presentations

18. C OPYRIGHT W EST E D, 2011 Benchmark Assessment: Assess Transfer to New Context Demo

19. C OPYRIGHT W EST E D, 2011 Benchmark Report: Class Summary

20. C OPYRIGHT W EST E D, 2011 Benchmark Report: Student Details

21. C OPYRIGHT W EST E D, 2011 Samples 28 think aloud sessions Pilot testing in 4 states 60 participating teachers ~5,800 students 40 schools 28 districts – Response data on the assessments, post tests and classroom observations

22. C OPYRIGHT W EST E D, 2011 Research Evidence Quality of the assessments (external review) – Alignment with standards and curriculum – Coherence – Science content and inquiry – Item design Implementation – Feasibility in schools and classrooms – Utility for students and teachers

23. C OPYRIGHT W EST E D, 2011 Research Evidence Technical Quality (Validity & Reliability) – Alignment with science standards – Cognitive labs – Analyses of responses Comparison of student responses on simulation-based and traditional items – • Cognitive labs – • IRT model fits

24. C OPYRIGHT W EST E D, 2011 Reliability of Benchmark and Post Tests Ecosystems – Benchmark (n=3844) Mean score was 28.14 (max 39) Standard Deviation 6.64 Coefficient Alpha: 0.85 – Post Test ( n=3422 ) Mean score: 16.52 (max 30) Standard deviation: 6.68 Coefficient Alpha (KR 20): 0.88

25. C OPYRIGHT W EST E D, 2011 Reliability of Benchmark and Post Tests Force and Motion – Benchmark (n= 1504 ) Mean score: 28.15 (max 40) Standard Deviation: 5.51 Coefficient Alpha: 0.79 – Post Test ( n=2166) Mean score: 14.87 (max 28) Standard Deviation: 5.73 Coefficient Alpha (KR 20): 0.82

26. C OPYRIGHT W EST E D, 2011 Correlation of Benchmark to Posttest ** Correlation is significant at the 0.01 level (2-tailed)

27. C OPYRIGHT W EST E D, 2011 CRESST Evaluation Questions How effective was SimScientists implementation? Was SimScientists feasible and useful in middle school classrooms? How effective were SimScientists’ accommodations?

28. C OPYRIGHT W EST E D, 2011 CRESST Evaluation Methods Convenience sample of teachers from all three pilot states, both topics – Eight teachers with 3-4 classes – Five schools Observation and interview protocols – Assessments – Reflection Activities Teacher survey

29. C OPYRIGHT W EST E D, 2011 Teacher’s Role (Assessment)

30. C OPYRIGHT W EST E D, 2011 Class Organization (RA)

31. C OPYRIGHT W EST E D, 2011 CRESST Interviews “I think that [students] are way more engaged … using this kind of interactive assessment … it was more engaging them simply, “Here is a pencil, here is a paper, answer questions”.”

32. C OPYRIGHT W EST E D, 2011 CRESST Interviews “ …I like that kind of feedback when it doesn’t just go to the next page and they don’t know whether they did it right or not.” “…Yes, the science content is really being tested. Students are asked to conduct experiments, investigate, and draw conclusions and to use scientific skills. Students are not able to guess on the multiple choice questions because it probes them until they choose the right answer. Students are also taught about food webs in one biome and they are tested on another biome.”

33. C OPYRIGHT W EST E D, 2011 CRESST Evaluation Conclusions Observations showed that students were actively engaged most of the time during assessments. Both teachers and students generally believed that the SimScientists program was beneficial to learning. Teachers found the automatically scored, immediate feedback—especially the reports generated by the questions—helpful to students. The instant reports allowed teachers to easily see which questions students had the most difficulty with so that they could tailor their lessons accordingly.

34. C OPYRIGHT W EST E D, 2011 CRESST Evaluation Conclusions Teachers collectively agreed that the simulation assessments had greater benefits than traditional paper-and-pencil tests because of the simulation’s instant feedback, interaction, and visuals. Teachers agreed that the assessments would be useful in measuring their individual state standards.

35. C OPYRIGHT W EST E D, 2011 Multilevel Balanced Assessment System Models Reporting benchmark results alongside district and state data Matrix sampling of short “signature” tasks from different topics

36. C OPYRIGHT W EST E D, 2011

37. Matrix Sampling Short “Signature Tasks” Across Topics

38. C OPYRIGHT W EST E D, 2011 Policy Brief Summary of alternative models for integrating simulation-based assessments into balanced state assessment reporting systems Summary of study findings on feasibility, utility, and technical quality Implications for further study

39. C OPYRIGHT W EST E D, 2011 Related Activities Increased use of technology in classroom instruction and assessment Continued (and growing) frustration around measuring student’s knowledge and skills above conceptual level Growth in costs associated with hand scoring of CR items on science assessments at grades 5 and 8 Development of electronic and on-line resources for instruction, assessment, and professional development for math and science – MINES, Science Builder, Pathways to Nevada’s Future

40. C OPYRIGHT W EST E D, 2011 Other Projects Race to the Top Assessment consortium related to CCSS in math and reading Multilevel Assessments of Science Standards Science Assessments with UDL EAG proposal for 2011? – Study series of assessment suites within a grade

41. C OPYRIGHT W EST E D, 2011 Contact information Email: rvineyard@doe.nv.gov equellm@wested.org msilber@wested.org Website: www.simscientists.org