1. CaMSP Science Assessment Webinar Public Works, Inc. Sharing Lessons Learned in the Development and Use of Science Assessments for CaMSP Teachers and Students October 6, 2010 Patty O’Driscoll Scott Phelps Public Works, Inc.

2. Public Works, Inc. serves as the statewide evaluator for California Mathematics and Science Partnership (CaMSP) program. In addition to supporting the completion of the APR and conducting the statewide evaluation and reports, PW also provides technical assistance to partnerships through the CaMSP Learning Network, Webinars, and orientations.

3. Purpose of Today’s Webinar Provide Technical Assistance to sites for local evaluation questions Respond to requests for sharing of information and instruments among partnerships Overview and sharing of Public Works, Inc. Website tools

4. Why the Focus on Science Partnerships? Few “off-the-shelf” assessments available for teachers OR students in science Limited grade levels for CST Share the work of partnership evaluators, project directors, and districts to meet the difficult challenges of this work Provide lessons learned and forum for sharing among science partnerships

5. Sharing Best Practices in the Field How we collected information: –Site visit interviews –Responses to email and phone calls from project directors and evaluators What do you use for student assessment? What do you use for teacher assessment? What are some of the lessons learned? What are some of your successes? What are some best practices you would recommend for others?

6. CaMSP Science Partnerships –3 partnerships in Cohort 5 –3 partnerships in Cohort 6 –2 partnerships in the Research Cohort –5 partnerships in Cohort 7 –Total of 13 CaMSP science partnerships currently operating (out of 45 partnerships)

7. Dimensions of Assessment in CaMSP Science State evaluation measures student outcomes using CST’s: –Limited to 5th & 8th grades –Cannot control for prior achievement in science –Not longitudinal for students Local evaluations for CaMSP measure: –Student outcomes in other grade levels –CST’s and local benchmark or project assessment data –Outcomes for treatment and control students –Teacher content knowledge –Classroom Implementation –Research questions of interest to partners

8. Teacher Content Knowledge Existing: –University of Louisville’s DTAMS Life and Physical science tests –Horizons Research ATLAST Assessments –Added links to these on the PW Website

9. Teacher Content Knowledge Partnership Developed –Pre-test and Post-test essay questions related to PD rotated through physical science, earth science, and life science Topic choices and a randomly assigned topic and how they would explain to students Scored by team of raters (blind) with a 6 element scaled rubric –Use of standards aligned students’ pre- and post-tests –Pre- and post-tests specifically aligned to content of intensive activities

10. Other Teacher Instruments Science Teaching Efficacy Belief Instrument (STEBI) Oregon Teacher Observation Protocol (O- TOP) PLC Survey Pedagogical Practices Questionnaire of Teacher Strategies Classroom observations

11. Student Assessment Locally developed standards-aligned student assessment system housed in county office using content-validated item bank for multi-district partnership Assessments for 7th and 8th graders measuring knowledge and science misconceptions in physical and life science Partnership-developed grade 3 to 5 pre and post- test addressing physical, earth, and life sciences –Different pre- and post-test items –Multiple choice--75% test and 25% incorporating graphic images with context, problems or challenges for students –Common scale constructed across grades, validation, and reliability tests conducted

12. Student Assessment (continued) Evaluator developed pre- and post-tests for participating grades aligned to standards Student notebooks and rubric for analysis of both student work and teacher practice Unit Assessments for earth, life, and physical science aligned to FOSS adopted instructional materials developed by CaMSP district staff District-developed benchmark assessment system provides good example for a partner district to adapt and implement

13. Lessons Learned Standardized tests are important but not sole measure of student achievement because: –Not all grade levels, not comparable across grade levels, and not always aligned with content in PD Logistics are difficult especially across districts, evaluators must weigh administration against “test- weariness” of teachers and students Don’t overwhelm but don’t let teacher resistance to additional assessments hinder collection Combination of instruments/measures allows for a review of data and connections to local context for professional development model and practice

14. Successes and Best Practices Development of system over time resulted in a useful assessment that can detect small program impacts Because of few existing options, measurement development, pilot testing, and other strategies are important to be able to measure program impacts Working with program staff is key to explaining the importance of local evaluation and the data being collected--dialogue with teachers and feedback of results embedded in PD is key to buy-in Multiple measures have helped to capture impact of partnership over time and how contributing to CST improvements –Example of student notebooks contributing to retention of information

15. CaMSP Website Demo www.publicworksinc.org

16. Public Works, Inc. Team Questions about the evaluation and the CaMSP Web site can be addressed to: - Patty O’Driscoll, patty@publicworksinc.orgpatty@publicworksinc.org - Albert Chen, achen@publicworksinc.orgachen@publicworksinc.org - Velette Bozeman, velette@publicworksinc.orgvelette@publicworksinc.org Phone number: 626-564-9890