1. Lessons Learned from the Evaluation of an Inquiry Science Project
Presented by Sarah Brasiel, Leslie Grunden, and Eric Rolfhus

2. Outline of Presentation
Project and Evaluation Overview
Ensuring Quality Data for Informative Analysis
Problem-solving Data Collection Issues
Presenting Findings that are of Utility for Program Improvement
Documenting implementation issues
Project management

3. Project and Evaluation Overview

4. Inquiry-Based Science Kit Instruction Project
Based on the National Science Resources Center (NSRC) inquiry-centered science education reform model
High quality K–8 curriculum
Sustained PD for teachers
Material support
Community and administrative support
Assessment and evaluation.
National Science Resources Center was formed by the Smithsonian Institution and the National Academies in 1985.

5. Participants 4 Southwest school districts 10 schools
Elementary students (n=1,514) and teachers (n=77) in grades 3–6
Science Coaches

6. Inquiry- based Science Kits
Theory of Change
Teacher Professional Development and Coaching
Increased teacher content and pedagogical knowledge and skills in science
Inquiry- based Science Kits
Effective delivery of classroom instruction
Improved student achieve- ment

7. Evaluation Components
Teacher Knowledge and Skills
Curriculum assessment performance
Delivery of Classroom Instruction
Fidelity of implementation
Student Achievement
State standardized Science assessment performance

8. Inquiry-based Science Kits
Full Option Science System (FOSS), developed by the Lawrence Hall of Science, Berkeley
Science and Technology for Children (STC), developed by the National Science Resource Center (NSRC)

9. Ensuring Quality Data for Informative Analysis

10. Student Data Issue Resolution
Unusual patterns in over student kit assessment performance data
Resolution
Time spent to document issues and discuss with project leader to correct
Requested item level data
Re-ran analyses and problem-solved data issues

11. Problem Solving Data Collection Issues

12. Classroom Observations
Issue
If observers did not see behavior, they were to leave form items blank, resulting in skewed data
Recommend Resolution for Future
Require a rating per Item to document expected teacher and student behaviors
Determine acceptable benchmark of low versus high implementation

13. Availability of Data Issue Resolution
Not granted access to state data to perform propensity score matching
Resolution
Requested and granted access to data within participating (plus one additional) district to perform student level propensity score matching

14. Presenting Findings that are of Utility for Program Improvement

15. Issue: Presentation of Findings not Useful
Main users of report did not find typical statistical displays useful
Example: Distribution of Student Performance at Pretest and Posttest

16. Resolution: Item Analysis with Stars for Areas of Low Effect

17. Resolution: Reflection for Areas of Low Effect
Take a look at this item. This item had a small effect for both students and teachers.
What makes this item difficult?
What instructional strategies might be used to improve understanding related to this item?

18. Classroom Observations

19. HLM Analysis findings

20. Documenting Implementation Issues

21. Issues Resolved through Communication
Biweekly meetings with client
Interim Reports
Discuss potential implementation biases
Future recommendations for improvement of program in future as well as study design

22. Project Management

23. Deliverables Issue: Late delivery of data to evaluators
Resolution: Change in statement of work to remove hard deadlines of exact dates for deliverables to # of days following receipt of data.