Evaluation of After School Programs Denise Huang CRESST Conference September 8th, 2005

What is Evaluation Research?  Evaluation research is the systematic application of social research methods to the assessment of social intervention programs.  It draws upon the concepts and techniques of several disciplines, and is useful at every stage of the programs.

Three Major Focus of Evaluation Research:  Analysis related to the conceptualization and design of interventions  Monitoring of program implementation  Assessment of program effectiveness and efficiency

For Efficient Evaluation Different evaluation strategies are called for at different stages in the life of programs:  In the planning stages of social intervention programs, evaluation focuses on the assessing the extent and severity of the problems requiring intervention, and design programs to alleviate the problems.  In the conduct of new and on-going programs, evaluations help to determine the degree to which the programs are effective.  After the program has been implemented for sufficient time, impact assessments estimate the effects of intervention. For accountability purposes, evaluations consider costs in relation to benefits and compare an intervention’s cost effectiveness.

Strategies for Impact Assessment  Impact assessments are undertaken to determine whether a program has its intended effects  Underlying all impact assessment is the experimental model of comparison  Assessments may make use of quantitative or qualitative data.

Evaluating after school programs: An LA’s BEST Example The ever evolving role of after school programs: Safe haven Enrichment activities Healthy life style Improve academic achievement

LA’BEST: Better Educated students for Tomorrow Serving 23,000 students over 133 school sites (147) Focus on developing the “whole” child Education, enrichment,& recreation Research-based curricula Community resources

Indicators for program effectiveness Student attendance Performance measures (achievement tests, homework completion, classroom grades, language re-designation, school retention, future aspirations, etc.) Non-cognitive measures (safety, attitudes towards school, relationships with adults, social competence, conflict resolution skills, self-esteem, and self-efficacy etc.) Parent involvement Long-term effects (drop out rate, life satisfaction, etc)

Challenges in evaluating after school programs Some of the main concerns: Self-selected sample Consent forms Comparison groups High transient rate Archive data-record keeping

Encouraging notes Dedicated staff Happy kids Satisfied parents Satisfied teachers Builds resiliency

Summary of Findings: Academic Performance  Overall LA’s BEST students either maintained or improved their SAT-9 /CAT- 6 or CST performances  Overall there is not a difference in academic performance between LA’s BEST and non-LA’s BEST participants  Pre-post analyses indicates that students improved their school attendance after participating in LA’s BEST

Summary of Findings: Social Development  concerns about safety  conflict resolution  academic self-efficacy  study habits  family involvement  future aspirations

Who Benefited Most :  LEP students  Female students  Students who attended more regularly  Students who scored low initially on attitudinal surveys (self-efficacy, work habits, conflict resolution skills)  Students with low performance in test scores

Summary of Findings: Long-term effects Student and Parent surveys responses between the two groups were very similar regarding:  academic competence  social competence  behavioral indices  future orientations LA’s BEST parents exhibit a higher degree of parental involvement in school and had higher academic expectations for their children than non-LA’s BEST parents

Former Students’ Perceptions of Positive Effects Maintain friendships Former LA’s BEST students Learn about importance of school Develop social and academic skills Receive opportunities for mentoring

Parents’ Perceptions of Positive Effects on Child Parents of former LA’s BEST students Improved grades Improved social skills Think about college

Implications of the Effects of LA’s BEST After School Program Impact of LA’s BEST not limited to academic achievement Contributes to improvement in school attendance and social skills, and impacts educational aspirations Provides a safety net for students who might otherwise fall through cracks Initial findings on drop out rates

Next steps  DOJ long-term effect study  National Partnership study to provide web-based resources to after school programs