1. Statewide Evaluation Cohort 7 Overview of Evaluation March 23, 2010 Mikala L. Rahn, Ph.D

2. Who Are We? Public Works, Inc. a nonprofit in Pasadena dedicated to working with communities, government, schools and parents by providing services and resources to educate and inform children, youth and families. Our work is in three areas: Education Reform Workforce Development Intervention/Prevention

3. Our Team Mikala L. Rahn: Project Director Team: Velette Bozeman, Albert Chen, Scott Phelps, Patricia O’Driscoll, James Maloney

4. Multiple Evaluation Parts 1)Local Evaluation 2)Statewide Evaluation 3)Federal Reporting

5. Statewide Evaluation Research Questions 1.How have the Partnerships ensured that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses? 2.How have the Partnerships enhanced the quality of the mathematics and science teacher workforce? 3.What evidence-based outcomes from the Partnerships contribute to our understanding of how students effectively learn mathematics and science?

6. Key Features Features of the programs reviewed: Partnership driven Teacher quality Challenging courses and curricula Evidence-based design and outcomes Institutional change and sustainability

7. Balanced Evaluation Approach Process measures focus on how a program is implemented and the outcome measures focus on the results of the program or intervention. Public Works, Inc. will use both quantitative and qualitative data collection methods.

8. Qualitative Analysis Qualitative research including site visits to Partnerships every year with interviews, focus groups and observations to gain general patterns across the Partnerships for a statewide picture, and best practices discovered throughout the state.

9. Quantitative Data Teacher Database: PAIF data on all teachers in participating districts and professional development hours on all participating teachers Partner Survey (Winter) Participating Teacher Survey (Spring) Student Outcomes (CST data) from control/treatment group of teachers on statewide measures (Fall)

10. Example Quantitative Indicators Increased number of teachers who participate in professional development More elementary school teachers with academic background in mathematics or science More secondary classes taught by qualified teachers Changes in teachers’ characteristics

11. Example Quantitative Indicators Improved achievement on state mathematics and science assessments CSTs Increased participation and success in advanced mathematics and science courses

12. Teacher Outcomes Teachers Increased number of teachers who participate in professional development and are highly qualified Demographics/Years teaching Hours of Training Qualifications/Assignments Treatment vs. Control (Local Evaluation: Pre-/post-/post content knowledge and comparison-control group. In addition to pre-post assessments timed with summer training, local evaluators are encouraged to conduct additional post-assessments e.g. at the conclusion of training at the end of the school year)

13. Student Outcomes Students Improved student academic achievement on state mathematics and science assessment and increased participation by students in advanced courses in mathematics and science

14. Steps CDE will Provide 2009-10 PAIF’s Determine who should get access to the database Input attendance data into Web system at least quarterly Closeout every June 30th In Fall 2011…… PW provides treatment and control group student data request Data to PW before October 15

15. State Work Plan: Cohort 7 Year 1 Contacts/Profile Partnerships (March 2010) Observations (Spring-Summer 2011) Conduct site visits and Phone Interviews (Spring-Summer 2011) Partner Survey (Nov. 2010) Quarterly Web-based Database (ongoing) Toolbox of instruments (ongoing) Administer teacher survey (April 2011) Complete control/treatment lists of teachers (Sept. 30, 2011) Student data for control/treatment (Before Oct. 15, 2011) Federal APR completed and ready for CDE review (Oct. 15, 2011) CDE approved Federal Reports by Partnership (Nov. 30, 2011)

16. State Work Plan: Cohort 7 Year 2 Year Two is the start of the outcome analysis phase. Each year will add new grantees (baseline) and increase our ability to measure change over time for existing grantees. Cohort Study Longitudinal Study

17. Levels of Evaluation Central versus local CDE/Public Works, Inc. CDE/Federal Partnership/ Local evaluators

18. Federal Reporting September 1-October 15, 2011 Block it now! Narrative, Project Profile, & Local Evaluation Report Due for Cohort 7 from CDE to ED by November 30, 2011

19. Local Evaluation Two Goals: Fulfill state evaluation requirements Fulfill commitments made in Local Evaluation Plan Internal or External evaluator (not a training provider) qualified to conduct a research study

20. Local Evaluation Various commitments: Teacher observation, portfolio, lesson study, video Surveys Pre- and post- assessments Qualitative/case studies Value added/control/treatment Most important for local: Teacher knowledge and instructional strategies & measuring student knowledge with local assessments. Also, the completion of the spring CDE YTD report as well as the fall Federal Report.

21. Measuring the Unmeasurable Capturing your story…. Narrative/journaling professional growth from the teacher’s perspective Demonstration lessons/video Teacher portfolios Performance tasks from students How do we document our work in math and science?

22. Being Clear on Terms What is your model for… Coaching Lesson study Hands-on Follow-up Mentoring

23. Bottom Line on Evaluation Documentation is crucial High profile at the state-level: outcomes matter! Impact: How does professional development translate into improved classroom practice? Demonstrate best practices in content delivery/pedagogy within state content

24. Web-based Communication www.publicworksinc.org Contacts (check) Profiles (check) Administrative (use) Database (use)

25. Web-based Teacher Database Input Demographic teacher data once per year (October) Input training data as you go Database information will be reviewed by CDE as part of the YTD process.

26. To Do List Update contact information (including database contacts) Check profile information Work with Public Works, Inc. to begin data entry for quarterly reports