1. CaMSP Cohort 8 Orientation Cohort 8 State and Local Evaluation Overview, Reporting Requirements, and Attendance Database February 23, 2011 California Department of Education Mikala L. Rahn, Ph.D Public Works, Inc.

2. Who Are We? Public Works, Inc. a nonprofit in Pasadena dedicated to working with communities, government, schools and parents by providing services and resources to educate and inform children, youth and families. Our work is in three areas: Education Reform Workforce Development Intervention/Prevention

3. Our Team Project Director: Mikala L. Rahn, PhD. Team: Patty O’Driscoll, Velette Bozeman, Albert Chen, Scott Phelps, James Maloney

4. Multiple Evaluation Requirements 1)Statewide Evaluation 2)Local Evaluation 3)Federal Reporting

5. Levels of Evaluation Central versus local CDE/Public Works, Inc. CDE/Federal Partnership/ Local evaluators

6. Statewide Evaluation Research Questions 1.How have the Partnerships ensured that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses? 2.How have the Partnerships enhanced the quality of the mathematics and science teacher workforce? 3.What evidence-based outcomes from the Partnerships contribute to our understanding of how students effectively learn mathematics and science?

7. Key Features Features of the programs reviewed: Partnership driven Teacher quality Challenging courses and curricula Evidence-based design and outcomes Institutional change and sustainability

8. Balanced Evaluation Approach in the Statewide Evaluation Process measures focus on how a program is implemented and the outcome measures focus on the results of the program or intervention. Public Works, Inc. uses both quantitative and qualitative data collection methods.

9. Qualitative Analysis Qualitative research including site visits to Cohort 8 partnerships and selected additional partnerships from previous cohorts in Spring 2012. Site visits include interviews, focus groups and observations to gain general patterns across partnerships for a statewide picture, and best practices discovered throughout the state. Telephone interviews in years you are not visited (Spring 2013).

10. Quantitative Data Collected Teacher Database: Incorporates PAIF data on all teachers in participating districts and attendance in professional development for all participating teachers (ongoing) Partner Survey (Winter) Participating Teacher Survey (Spring) Student rosters of comparison and treatment teachers for statewide study/CST Data (first in Summer-Fall 2012)

11. Teacher outcomes examined in state evaluation Teachers Number and characteristics of teachers who participate in professional development; satisfaction with training Demographics/Years teaching Hours of Training Qualifications/Assignments Treatment vs. Comparison Student Performance on CSTs

12. Student outcomes examined in state evaluation Students Improved student academic achievement on state mathematics and science assessment across the state and at the partnership level

13. Local Evaluation Two Goals: Fulfill state evaluation requirements Fulfill commitments made in Local Evaluation Plan Internal or External evaluator (not a training provider) qualified to conduct a research study

14. Local Evaluation Fulfill commitments in CaMSP proposal: Pre- and post- assessments of teacher content knowledge Matched control/comparison group of teachers Local assessment data/CSTs Teacher observation, portfolio, lesson study, video Surveys Qualitative/case studies Value added/comparison/treatment methodologies Most important for local: Teacher knowledge and instructional strategies & measuring student knowledge with local assessments. Also, the completion of an evaluation report to attache to the March CDE YTD report and another for the fall Federal Report.

15. Measuring the Unmeasurable Capturing your story…. Narrative/journaling professional growth from the teacher’s perspective Demonstration lessons/video Teacher portfolios Performance tasks from students How do we document our work in math and science?

16. Being Clear on Terms What is your model for… Coaching Lesson study Hands-on Classroom follow-up Mentoring

17. Bottom Line on Local Evaluation Documentation is crucial High profile at the state-level: outcomes matter! Impact: How does professional development translate into improved classroom practice? Demonstrate best practices in content delivery/pedagogy within state content

18. Federal Reporting Cohort 8 Completion of APR is partnership and local evaluator responsibility PW provides the Student GPRA data and technical assistance to partnerships Cohort 8 First APR: PW Data Collection for APR Begins July 15, 2012 First Annual Performance Report (APR) due to CDE October 15, 2012 Block it now! Narrative, Project Profile, & Local Evaluation Report are Required Due for Cohort 8 from CDE to ED by November 30, 2012

19. PW State Evaluation Work Plan: Cohort 8 Contacts/Partnership Profiles on CaMSP site (March 2011) Conduct site visits, observations, and phone interviews (Spring 2012) Administer Partner Survey (Nov. 2011) Web-based Database (ongoing) Toolbox of instruments (ongoing) Administer teacher survey (Feb-March 2012) Determine list of treatment/control teachers based on attendance through June 30, 2012 by July 15, 2012 Create matched comparison list of teachers by Aug 15, 2012 Collect rosters of students of treatment and comparison teachers from partnership districts by September 15, 2012 Match to 2012 CSTs and provide student GPRA # to partnerships by October 15, 2012

20. Next Steps After Lunch: Demonstration of CaMSP Website and PW Attendance and Teacher Participation Database CDE provides updated PAIF (working from 08-09); CDE provides statewide CST data Determine who should get access to the database Input attendance data into Web system frequently (used by CDE for YTD’s!!) Closeout attendance data every June 30th by July 15th All professional development activities are expected to be logged as soon as possible following the activity.

21. Web-based Teacher Database Reminders Demographic teacher data is updated once per year from statewide PAIF (October). Input training data as you go. Database information will be reviewed by CDE as part of the YTD process.

22. To Do List Provide Updated contact information to CDE and PW (including database contacts) Review and approve PW partnership profile for CaMSP Website Work with Public Works, Inc. to begin data entry of attendance for periodic reports

23. Web-based Communication www.publicworksinc.org Contacts (check) Profiles (check) Administrative (use) Database (use)

24. Contact Information Mikala Rahn, mikala@publicworksinc.org mikala@publicworksinc.org Patty O’Driscoll, patty@publicworksinc.org patty@publicworksinc.org Albert Chen, achen@publicworksinc.org achen@publicworksinc.org Velette Bozeman, velette@publicworksinc.org velette@publicworksinc.org Phone: (626) 564-9890