1. District Response to Program Improvement Accountability Sanctions and Technical Assistance in California December 1, 2011 – CERA Presentation Presented by: Theresa Westover and Mary Stump 1

2. Evaluation Context Federal: ESEA mandates increasing sanctions for districts not making AYP. – In addition, ESEA requires states provide provide technical assistance to all LEAs identified for improvement (NCLB, PL 107- 110, TI, Sec 1116(c) California: Sanction F/Corrective Action 6 – Curriculum AB 519 – Chaptered in 2008 – Required independent evaluation 2009-12 Evaluation focus: DAIT Process and Student Achievement Changes – Mixed Methods – Cohort 1: 43 Districts in PI Year 3 assigned corrective action 6 in 2008-09 & their DAITs – Cohort 2: 30 Districts in PI3, assigned Corrective Action 6 in 2009-10 & their DAITs 2

3. Program Improvement – The Top-Down Accountability Era 3

4. Today’s focus: 1.Brief look: what districts are in program improvement and focus of evaluation 2.How the DAITs engaged with their districts 3.What actions the district and DAIT took - the focus of the improvement efforts 4.The barriers and facilitators encountered along the way 4

5. Cohort 1: Districts in PI (N=43) Broad geographic and size range 5

6. PI3 Districts: High Need Students, Less Experienced Teachers PI3 districts have more disadvantaged students than do districts that are not in PI Within PI3 districts, those with DAIT treatment appear to have the most disadvantaged students in California Districts with assigned DAITs have a significantly different teacher workforce in terms of average and median experience levels and certification rates. (Details in Year 1 Report, available at cees.ucdavis.edu) 6

7. There are significant disparities in poverty and ELL designations between PI districts Non-PI Districts PI 1 Districts PI 2 Districts All PI 3 Districts PI 3 Districts – No Aid PI 3 Districts – TA PI 3 Districts – Contracted DAIT PI 3 Districts – Assigned DAIT

8. PI3 districts service more Hispanic and fewer white students Non-PI Districts PI 1 Districts PI 2 Districts All PI 3 Districts PI 3 Districts – No Aid PI 3 Districts – TA PI 3 Districts – Contracted DAIT PI 3 Districts – Assigned DAIT

9. Methods and Data (Year 2 Report) For each of the 43 Cohort I districts we: – Analyzed weaknesses and recommendations in the capacity studies – Issued surveys to both the DAIT providers and the Districts – Interviewed both the DAIT provider and the District leadership team Response rates for surveys and interviews: 100% for DAIT providers, close to 90% for districts 9

10. Qualitative data CEES selected areas of evaluation inquiry based on: legislation requirements the guidelines provided by CDE to DAIT providers (the DAIT “strands”) California’s 9 Essential Program Components (EPCs) http://www.cde.ca.gov/ta/lp/vl/essentialcomp.asp 10

11. DAIT Process in California 11

12. DAIT Recommendations in Capacity Studies 12

13. DAIT engagement 97% of providers remained engaged in their districts for 2 or more years Over 90% agreed on surveys that the DAIT: – Effectively diagnosed district needs & priorities – Provided support for the LEA plan/addendum – Was provided access and information needed to understand district needs – Was able to effectively engage the district leadership to address needed changes Most providers met at least monthly w/ district and reported having successfully established open & cooperative relations 13

14. Focus of improvement efforts Open ended interview responses indicated that high priority areas for improvement included: – Supports for English learners (50% mentioned) – Focus on math curriculum/instruction (33%) – Data based decision making/data systems (40%) These areas were reflected in survey ratings as showing improvement over the 2 years 14

15. Changes in implementation of EL/ELD supports Practice/Policy Percent Full/Substantial Implementation 2008-09 Percent Full/Substantial Implementation 2009-10 Change Administrative procedures to implement and monitor district ELD programs 45.280.5+35.3% All teachers participate in ELD/SDAIE professional development 53.673.8+20.2 District ensures that teachers use SBE adopted/approved materials for ELD 58.581.0+22.5 District ensures that ELs have access to grade level core instruction in Math 75.697.0+21.4 15

16. Changes in data systems/data-based decision making Practice/Policy Percent Full/Substantial Implementation200 8-09 Percent Full/Substantial Implementation20 09-10 Change District administrators regularly use data to monitor student progress 47.683.3+35.7% District has a system of regular data collection to determine the effectiveness of its academic program 69.092.9+23.9 All teachers are provided collaboration time specifically for examining student data to inform instruction 61.988.1+26.2 District staff adhere to established criteria for student entry and exit into interventions 35.764.3+28.6 16

17. Overall Changes in Implementation Ratings (4 point scale, with 4=full implementation) 17

18. Barriers and Facilitators Research in organizational change and district/school improvement demonstrate that organizational change takes time and needs to be responsive to contextual factors. Interviews confirmed that the district improvement efforts are highly contextual – not only the content of work but the way it was undertaken and the success, or lack thereof, of the efforts 18

19. District leadership factors Tenure of district leadership (Supt & cabinet) Leadership’s willingness to engage in reform Leadership style Existing relationships among district staff, between district staff and school board, teacher unions and other stakeholders Change in district leadership over the course of the engagement 19

20. District history, setting, and culture Tradition of school site accountability to district History of student achievement, mobility & demographics Location & size of district Responsiveness to external pressure/mandates Culture around expectations for student achievement and student subgroups 20

21. Local stakeholders Local political climate and relations among stakeholder groups School Board stability, level of involvement in district policies and practice, political positions Teacher union contractual language, relationship w/ Board and district, history of negotiations w/ district Legal actions 21

22. District resources, management and structures Fiscal resources Human resources – e.g. expertise, staffing levels Existing structures and practices – Data & assessment systems – Practices around monitoring classroom instruction – Communication & accountability structures district:schools – Allocation of responsibilities among district staff 22

23. Most significant changes Supports for under-performing students, esp. ELs Improved instructional materials in ELA & Math, including teacher & administrator PD Teacher support – coaches, professional learning communities (PLCs) Data systems and use of data for decision making 23

24. Sustainability Unclear at this point Districts/DAITs identified a number of threats: – Fiscal crisis is a major threat to sustainability – Shifting priorities back from district to school level – Lack of on-going support and accountability for change once DAIT is gone 24

25. Preliminary Recommendations Continue to support district level capacity building and technical assistance Increase or maintain accountability structures for both districts & DAITs Assess district readiness for change and act more quickly to intervene when necessary Educate stakeholders – especially local boards Simplify and consolidate federal and state mandated reporting requirements Provide additional support and resources to assist district capacity building 25

26. Resources and Information About Program Improvement in California 1. California Department of Education CDE Website for PI: http://www.cde.ca.gov/ta/ac/ti/programimprov.asp and specifically for PI3 resources: http://www.cde.ca.gov/ta/ac/ti/pilearesources.asphttp://www.cde.ca.gov/ta/ac/ti/programimprov.asp CDE/SBE criteria for determining level of LEA need for technical assistance: EC Section 52055.57(d) Criteria identified in Item 16 on the SBE Agenda--November 18-19, 2009 Web page at http://www.cde.ca.gov/be/ag/ag/yr09/agenda200911.asp 2. California Comprehensive Center at WestEd Webinars, history and assessment instruments: http://www.cacompcenter.org/cs/cacc/print/htdocs/cacc/esea-requirements.htm http://www.cacompcenter.org/cs/cacc/print/htdocs/cacc/esea-requirements.htm 3. Center for Education and Evaluation Services, UC Davis - Theresa Westover & USC Rossier School of Education, Katharine Strunk. Interim Evaluation Reports on AB519 – Enactment of Corrective Action 6 in California: http://education.ucdavis.edu/select-publications-and-reports http://education.ucdavis.edu/select-publications-and-reports 4. The Use and Efficacy of Capacity-Building Assistance for Low-Performing Districts: The Case of California’s District Assistance and Intervention Teams Paper prepared for the Annual Research Conference of the American Education Finance and Policy Association, March 23-25, 2011. www.aefpweb.org/.../Strunk_Westover_and_McEachin_AEFP_2011 26

27. Contact Information: Theresa Westover tnwestover@ucdavis.edu 530-754-9523 Mary Stump mlstump@ucdavis.edu 530-752-2809 For more information, please visit the Center for Education and Evaluation Services website at: cees.ucdavis.edu 27