1. California Afterschool Outcome Measures Project California Afterschool Outcome Measures Project University of California, Davis and University of California, Irvine In collaboration with the California Department of Education and with support from the David and Lucile Packard Foundation

2. Project Overview: Three Phases Phase 1—INSTRUMENT DEVELOPMENT (2008-2009) Phase 1—INSTRUMENT DEVELOPMENT (2008-2009) Literature review Literature review Survey of the field (expected outcomes and current practices) Survey of the field (expected outcomes and current practices) Selection of measures with input from the CDE and Accountability and Outcomes Subcommittee Selection of measures with input from the CDE and Accountability and Outcomes Subcommittee Phase 2—PILOT STUDY (currently underway) Phase 2—PILOT STUDY (currently underway) Pre-piloting to test procedures and instructions to students, staff, and teachers; and to determine feasibility of online administration of surveys Pre-piloting to test procedures and instructions to students, staff, and teachers; and to determine feasibility of online administration of surveys Piloting to test NEAR-FINAL protocols under varying conditions Piloting to test NEAR-FINAL protocols under varying conditions Phase 3—FIELD STUDY (2010-2011) Phase 3—FIELD STUDY (2010-2011)

3. Outcome Measures Surveys based on existing measures with demonstrated reliability and validity: Surveys based on existing measures with demonstrated reliability and validity:  BEHAVIOR CHANGE (social efficacy, social skills with peers, child behavior, misconduct, and substance use)  SKILL DEVELOPMENT (work habits and task persistence) Three sets of instruments: Three sets of instruments:  Surveys of students who attend ASES afterschool programs  Surveys of ASES program staff  Surveys of classroom teachers

4. Profile of Pre-Pilot Sites Nine pre-pilot sites—5 Southern CA, 4 Northern CA Nine pre-pilot sites—5 Southern CA, 4 Northern CA 5 elementary school sites 5 elementary school sites 3 intermediate/middle school sites 3 intermediate/middle school sites 1 center-based site serving K-8 1 center-based site serving K-8 Seven school districts Seven school districts 3 urban 3 urban 3 suburban 3 suburban 1 rural 1 rural Range of school/program sizes Range of school/program sizes School enrollments 478 - 1,519 School enrollments 478 - 1,519 ASES enrollments 81 - 475 ASES enrollments 81 - 475

5. Pre-Pilot Activities Staff directed the administration of the online surveys to groups of students (2 to 6 groups per site) Staff directed the administration of the online surveys to groups of students (2 to 6 groups per site) Staff were provided a script and guidelines about the survey administration Staff were provided a script and guidelines about the survey administration Students completed online surveys about their skill development and behavior and about their experience taking the survey Students completed online surveys about their skill development and behavior and about their experience taking the survey Researchers observed surveying process and noted student feedback Researchers observed surveying process and noted student feedback ASES staff and classroom teachers completed the online surveys about individual students’ behavior and were interviewed about their experiences completing the surveys ASES staff and classroom teachers completed the online surveys about individual students’ behavior and were interviewed about their experiences completing the surveys Instructions and surveys were tweaked (mid pre-pilot) based on issues identified in the observations and interviews Instructions and surveys were tweaked (mid pre-pilot) based on issues identified in the observations and interviews

6. Pre-Pilot Phase November 2009—February 2010 Completed the test of initial versions of measures Completed the test of initial versions of measures Completed the test of initial protocols and procedures Completed the test of initial protocols and procedures Tested feasibility of online administration Tested feasibility of online administration Minor modifications of procedures and protocols during the pre-pilot Minor modifications of procedures and protocols during the pre-pilot

7. P re-Pilot Data Collected SurveyRespondents Total # Reports Completed Total # Reports Completed Average Time to Complete Survey Students34716 min Elementary (4 subscales) 19215 min Middle (5 subscales) 15518 min 39 staff (5 subscales) 1765 min per student 23 teachers (5 subscales) 1144 min per student

8. Survey Data Summary Tables Elementary School Students UCD and UCI Pre-Pilot DataPromising Programs Study nMeanSDRangeAlphanMeanSDRangeAlpha Social Efficacy1892.84.671.00 - 4.00.67 Misconduct1920.50.530.00 - 2.11.8112600.53.580.00 - 3.70.82 Work Habits1913.14.691.17 - 4.00.8212553.26.571.17 - 4.00.76 Middle School Students UCD and UCI Pre-Pilot DataPromising Programs Study a nMeanSDRangeAlphanMeanSDRangeAlpha Social Efficacy a 1532.91.651.00 - 4.00.811933.09.581.00 - 4.00.85 Misconduct1550.60.480.00 - 2.22.847530.63.560.00 - 4.00.83 Substance Use1510.17.400.00 - 3.00.837540.13.400.00 - 4.00.79 Work Habits1552.83.671.17 - 4.00.837543.15.511.33 - 4.00.75 Task Persistence1533.00.511.14 - 4.00.687543.16.491.14 - 4.00.68 a Comparison data for Social Efficacy are from the Tiger Woods Learning Center Study.

9. Survey Data Summary Tables Teachers Program Staff UCD and UCI Pre-Pilot DataPromising Programs Study nMeanSDRangeAlphanMeanSDRangeAlpha Social Skills1763.62.891.00 - 5.00.926373.66.841.14 - 5.00.95 Behavior w/Peers Aggressive Prosocial 176 0.38 1.39.52.46 0.00 - 2.00 0.25 - 2.00.93.88 636 0.31 1.41.43.47 0.00 - 2.00.92.91 Work Habits1763.581.041.00 - 5.00.956363.75.981.00 - 5.00.97 Task Persistence1762.98.711.00 - 4.00.896313.06.591.00 - 4.00.86 UCD and UCI Pre-Pilot DataPromising Programs Study nMeanSDRangeAlphanMeanSDRangeAlpha Social Skills1143.431.011.00 - 5.00.9612203.55.991.00 - 5.00.96 Behavior w/Peers Aggressive Prosocial 114 0.27 1.45.40.44 0.00 - 2.00 0.25 - 2.00.90.88 1219 1220 0.35 1.52.49.47 0.00 - 2.00.94.92 Work Habits1143.411.251.00 - 5.00.9712233.491.131.00 - 5.00.97 Task Persistence1142.96.711.00 - 4.00.9012212.95.721.00 - 4.00.93

10. Next Phase: Pilot March—June 2010 Pilot measures at a broad spectrum of ASES programs Pilot measures at a broad spectrum of ASES programs  12 Northern CA and 12 Southern C A Size of program (small, medium, large) Size of program (small, medium, large) Grade level (8 elementary, 4 middle/intermediate) Grade level (8 elementary, 4 middle/intermediate) Program status (challenged to well functioning) Program status (challenged to well functioning) Location (urban, suburban, rural) Location (urban, suburban, rural) Diverse student characteristics (ethnicity, % ELL and FRL) Diverse student characteristics (ethnicity, % ELL and FRL) Purpose of pilot: Purpose of pilot:  Identify issues at a range of programs with & without our physical presence 12 sites with UC researchers present 12 sites with UC researchers present 12 sites to administer independently [sent packet; support by email and phone only] 12 sites to administer independently [sent packet; support by email and phone only]  Test administration of teacher survey at 8 elementary sites 4 with teacher compensation 4 with teacher compensation 4 without teacher compensation 4 without teacher compensation  Test hard copy administration of student survey at sites without computer and internet access  Determine reliability of the measures under these different conditions