American Educational Research Association Annual Meeting AERA San Diego, CA - April 13-17, 2009 Denise Huang Identification of Key Indicators of Quality in Afterschool Programs

2 / 19 The study questions What are the basic core benchmarks for quality afterschool programs? What are the key indicators that help define the core benchmarks in afterschool programs? How can afterschool programs use a data-based system to identify their own strengths and weaknesses and strive for continuous improvement?

3 / 19 Study design Synthesis of literature on quality indicators was conducted to extract commonly referenced indicators and benchmarks for quality programs A theoretical model was constructed Surveys, interviews and observation protocols were developed by CRESST to validate these benchmarks and indicators Three local well established and high functioning afterschool programs were identified through a strategic recommendation procedure for validation A statistical weighing system was utilized to develop the preliminary tool

4 / 19 Synthesis of literature A wide search for relevant literature The abstracts of the 1,269 citations were obtained then carefully reviewed and discussed among the research team members When abstracts met the established criteria, the full articles (216) were reviewed 54 studies met the criteria for inclusion These studies were coded for benchmarks/indicators extracted from the synthesis

5 / Benchmark across categories Program OrganizationProgram EnvironmentInstructional Features SafetyProgramming/ActivitiesAcademic Support HealthFamily/Community Involvement Social Development Physical space/resourcesStaff/PDEnrichment Human relationshipsManagement/AdministrationPositive Youth Development Evaluation

6 / 19 Participating programs LA’s BEST (Los Angeles Better Educated Students for Tomorrow ) Lawndale RAP (Lawndale Realizing Amazing Potential ) Pasadena Learns (Pasadena Leading Educational Achievement – Revitalizing Neighborhoods) Each has been designated as a California After-School Partnership (CASP) Regional Learning Center and LA’s BEST was selected by World Hunger Year (WHY) as one of the top afterschool programs in the State of California.

7 / 19 Study Participants by Role and Afterschool Program ParticipantResponses Program Coordinators (Total)17 LA’s BEST5 Lawndale RAP5 Pasadena LEARNs7 Site Staff (Total)102 LA’s BEST39 Lawndale RAP26 Pasadena LEARNs37

8 / 19 Data analysis-Weighing system Step 1 – Establishing the two-thirds rule To make determinations as to whether the benchmarks/indicators were prevalent at the validation afterschool sites. Data were analyzed at the site level. Step 2 – Establishing the weighting system A checklist or numerical scoring tablet was created for each benchmark with all the indicators for that benchmark listed underneath When all the literature under the benchmark was re-examined, a numerical score was calculated for each indicator Statistical weighting was applied to give weight to all extracted indicators according to their importance as referenced in the literature Weekly research team discussions on the appropriateness of assigned weight to the rating score were conducted until consensus had been reached on all items

9 / 19 Establishing the core benchmarks Benchmarks that received a mean score of 7 out of 10 were considered as “core” components of quality afterschool programs Benchmarks that received a mean score lower than 7 were considered as “additional” or “exemplary” benchmarks that afterschool programs could use to enhance their program quality staff survey and observation was weighed 2:1

10 / 19 Formulating the “Quality Indicator System” Clear mission statement Staff/student input Collaboration with day school Clear mission statement Staff/student input Collaboration with day school Program policies Budgeting Sustaining Program policies Budgeting Sustaining Staff salary Staff feedback Staff orientation Staff salary Staff feedback Staff orientation Partnership with communities Service projects Plan for community involvement Partnership with communities Service projects Plan for community involvement Communication with parents Parent involvement Parent feedback Communication with parents Parent involvement Parent feedback Staff-student ratio Staff competency Professional development Cultural diversity Staff-student ratio Staff competency Professional development Cultural diversity Evaluation of staff performance Program activities Student engagement Student outcomes Continuous improvement Evaluation of staff performance Program activities Student engagement Student outcomes Continuous improvement Program Organization Management Staff Support Family Involvement Community Partnership Evaluation Administration Staff Training

11 / 19 Program Organization – Mean Scores, Core and High Quality Benchmarks BenchmarkMean score Core benchmark Exemplary benchmark 1. Program Management8.50  2. Program Administration8.00  3. Staff Support8.00  4. Staff Experience and Training8.40  5. Family Involvement7.00  6. Community Partnerships5.73  7. Evaluation8.27 

12 / 19 Program Environment Clean & secure Prevention strategies Proper supervision Clean & secure Prevention strategies Proper supervision Promote healthy habits Nutritious snacks Minimize health risks Safe equipments Promote healthy habits Nutritious snacks Minimize health risks Safe equipments Staff: student relationship Student: student relationship Staff: staff relationship Staff: student relationship Student: student relationship Staff: staff relationship Sufficient space Proper layout of space Provision for multiple activities Sufficient space Proper layout of space Provision for multiple activities Program Environment Safety Health Physical Space Positive Relationships

13 / 19 Program Environment – Mean Scores, Core, and High Quality Benchmarks BenchmarksMean score Core benchmarks High quality benchmarks 8.Safe Environment9.73  9.Student Health and Well-Being6.60  10.Well-Equipped and Suitable Physical Space 9.33  11.Positive Relationships9.37 

14 / 19 Instructional Features Appropriate activities Student engagement Cultural diversity Meets students’ needs Teaching & learning Opportunities for practice Appropriate activities Student engagement Cultural diversity Meets students’ needs Teaching & learning Opportunities for practice Academics Enrichment Socialization Academics Enrichment Socialization Youth development Personal responsibility Self-direction Leadership Youth development Personal responsibility Self-direction Leadership Instructional Features Quality of Implementation Variety of Activities Support Youth Development

15 / 19 Instructional Features – Mean Scores, Core and High Quality Benchmarks BenchmarksMean score Core benchmarks High quality benchmarks 12. Quality of implementation9.90  13. Variety of activities8.60  14. Activities support youth development 5.86 

16 / 19 Characteristics of the quality benchmark rating system 1) applicable to all programs serving students of different races, gender, and age groups. 2) applicable to programs with different program goals and approaches, such as academic achievement, enrichment, etc. 3) applicable to programs run by different organizations such as school districts, and community-based and religious based institutions, etc.

17 / 19 Using the rating system Each major program component (organization, environment, instructional feature) has its own checklist for quality If the indicator can be checked off– points will be allocated In order for a program to a meet the benchmark, it required a minimum score of 7 out of 10 points For programs that desired further improvement, they could examine the indicators that they did not check off and make plans on improving those weak areas In order to reduce subjectivity, it was best to have at least 3 raters completing the same instrument On benchmark ratings that did not reach consensus among the raters, discussions on those particular benchmarks and indicators would reveal insights and pinpoint areas of strength and weakness for program improvements

18 / 19 The score sheet BenchmarkScoreCore Quality (Y/N) High Quality (Y/N) Program Organization 1.Program management 1.Program administration 1.Staff support 1.Staff experience and training 1.Family involvement 1.Community partnerships 1.Evaluation Program Environment 1.Safe environment 1.Student health and well-being 1.Well-equipped and suitable physical space 1.Positive relationships Instructional Features 1.Quality of implementation 1.Variety of activities 1.Activities support youth development