November 18 th, 2010 San Diego, California CERA Conference The California Statewide ASES Evaluation

2 / 27 Purpose of Study 1.Prop 49 becomes effective in Sept Afterschool funding increases to 550 million annually 3.One of the stipulation is that the CDE shall contract for an Independent Statewide evaluation on the effectiveness of programs receiving funds. 4.CRESST will conduct 2 separate evaluations- ASES and ASSET 5.Evaluation will cover analyzing STAR and other CDE data from through

3 / 27 Deliverables Annual reports for ASES & ASSETs including achievement and behavioral outcomes Annual reports on afterschool profiles and changes In year 3 and 4 of the study, youth development outcomes will also be included in the annual reports. Final report to the Governor and the Legislature on December 1 st 2011.

Study Theory Best practices evidenced in the literature on out-of-school time suggest that several critical components such as: goal-oriented programs program structure and program process These components contribute to the effectiveness and success of programs.

5 / 27 Instructional Features Program Environment Program Orientation Staff Efficacy Setting Features Expectation Aspirations Positive Youth Development Academic Relationships Safety Goals Management External Connections Evaluation system Alignment Program Climate Satisfaction STAR CASHEE School Attendance Monitoring Resources Professional Development Theoretical model Student Engagement

6 / 27 Data Sources CDE will provide: STAR CAHSEE CELDT SSID maintenance system Graduation rate for High School students Annual accountability reports with student level data, including attendance in school and afterschool

7 / 27 Data Collected by CRESST Profiling survey distributed to all grantees program goals, structures, and process In depth site visits program satisfaction, program process, community partnerships, program impacts

8 / 27 Study Sample Definition Study Sample I All programs participating in the ASES and ASSET programs Study Sample II 100 ASES and 30 ASSET districts Study Sample III All programs returning the profiling questions Study sample IV Small sample for in depth analyses

9 / 27 The 6 evaluation questions for Study Sample IV 1. What is the impact of after school programs on the academic performance of participating students? Does participation in after school programs appear to contribute to improved academic achievement? 2. Does participation in after school programs affect other behaviors such as: school day attendance, homework completion, positive behavior, skill development, and healthy youth development? 3. Examine the similarities and differences in program structure and implementation. Describe how and why implementation has varied across programs and schools, and what impact these variations have had on program participation, student achievement, and behavior change. 4. What is the level of student, parent, staff, and administration satisfaction concerning the implementation and impact of after school programs? 5. What is the nature and impact of organizations involved in local partnerships? 6. What unintended consequences have resulted from the implementation of the after school programs?

10 / 27 Preliminary Afterschool Attendance Findings Elementary students tend to attend more regularly About 50% attended over 100 days Middle school students less regularly 26% less than 10 days, about 20% attended 100 or more days

11 / 27 Preliminary School Attendance Outcomes Statistically significant effect on School attendance Effect higher for frequent attendees No difference on school suspension-Sample II

12 / 27 Program Characteristics Most Programs are free for all First come first serve Majority of the programs stated that they have an academic goal Top three programming activities offered are: homework assistance, academic enrichment and tutoring Top three programming activities offered are: physical fitness/sports, arts/music, and recreation Top developmental activities offered are: ASES school safety, youth development

13 / 27 Example: Goal Setting & Progress GoalSet goalMet goal Progressed towards goal Set goal, but not evaluated Failed to progress Academic improvement (n = 298) 93.6%15.4%49.1%35.5%0.0% Improved day school attendance (n = 275) 68.0%27.8%36.9%34.8%0.5% Improved homework completion (n = 279) 80.6%29.3%40.9%29.3%0.4% Positive behavior change (n = 270) 69.3%19.8%50.8%28.3%1.1% Improved program attendance (n = 274) 86.9%43.7%35.3%19.3%1.7% Increased skill development (n = 260) 62.3%23.5%45.1%30.2%1.2%

14 / 27 Professional Development Site coordinators are the most likely to receive professional development The three top format are: training/workshop, program level meetings, and site level meetings The top three topics are: classroom management, behavioral management, and federal mandated trainings

15 / 27 Program Orientation Most programs use a collaborative management approach Having prior work experienced is important Most site claimed to have at least some credentialed staff on site at any day—especially in the urban areas Support of educational goals is the most likely recruitment strategy for ASES programs Staff recognition and is the most mentioned strategies for retaining staff, followed by promotion opportunities

16 / 27 Community Partnership Majority of the programs maintained community partnership ASES programs partnered with parents, school and district staff and high school students The entity most school sites partnered with is the local LEAs of school districts or colleges LEA helps in providing operational assistance (data collection), site management, and professional developments Parents and other community members generally were involved in setting and revising goals, providing goods and helping out with the implementation and fundraising of the programs. Other community members also played a role in fundraising for the programs.

17 / 27 Additional Insights from Preliminary Findings of Sample IV Students perceived the program climate to be safe physically and emotionally. They felt cared for and supported by the staff, while the staff reported to encourage students’ efficacy and aspiration. In general, staff reported to have sufficient writing materials, space to work, and enough time to work with the students. Both site staff and site coordinators were likely to report that they did not have time to prepare for activities. staff efficacy was enhanced by professional development. Both site coordinators and site staff agreed that the professional development offered was aligned with program goals and that the training was provided annually. Parent surveys reviewed a low degree of involvement, though they also appeared to be satisfied with the way programs connected with them.

18 / 27 Evaluation Systems Most site coordinators reported that they tracked the different stakeholders’ levels of satisfaction They also responded that they had an internal method of evaluating the program activities for program improvement

19 / 27 Perceived Outcomes Overall, students, parents, and after school staff all have positive perceptions on the effects of program participation. Both students and their parents felt that attending a program led to students feeling more academically efficacious. They also believed participation increased the students’ cognitive competence, & socio- emotional competence. Students reported that they felt that they had the skills, knowledge, and desire to become successful adults as a result of attending an after school program. High school students further reported that attending a program helped them resist the lure of alcohol, drugs, and gangs.

20 / 27 Obstacles for Parent Involvement ASES 60% visited the program One in three attended any event One in five volunteered or given feedback Conflicts with job Care for other children Language barrier Lack of transportation

21 / 27 Obstacles to Student Recruitment and Retention ASES-NONE Staff Language barrierLack of transportation Student Other activities