1. Presentation to the Before and Afterschool Advisory Committee California Department of Education January 11, 2011 January 11, 2011 Pilot Findings and Field Test: An Update

2. California Afterschool Outcome Measures Project  Last report to the Advisory Committee — February 24, 2010  Pre-Pilot activities and Key Findings:  Completed pre-pilot test of initial versions of measures and online administration  Measures performed well in online format; preliminary outcomes comparable to previous studies  Minor modifications made to measures and procedures based on pre-pilot findings Changes to wording in items, scales removed and added Changes to wording in items, scales removed and added Survey administration instructions revised—clear step-by- step Survey administration instructions revised—clear step-by- step Special instructions created to support ELL students and slow readers Special instructions created to support ELL students and slow readers

3. Highlights of Today’s Report  Phase 2—Report on the Pilot Test  Creation of project website  Results of the spring 2010 pilot work  Phase 3—Update on the Field Test  Fall survey administration experience  Dissemination plans

4. Phase 2: PILOT PHASE  Project website created: afterschooloutcomes.org  Piloted online measures of positive behavior change and skill development:  27 sites (13 in Southern and 14 in Northern CA)  5 urban; 10 rural; 12 suburban  18 elementary; 6 middle school; 3 center-based serving K-8  10 regions across the state  Range of school/program sizes in Pilot sample: School enrollments 225 to 1,963 School enrollments 225 to 1,963 ASES enrollments 43 to 574 ASES enrollments 43 to 574  Demographically diverse students (ethnicity, ELL and FRL)

5. Pilot Data Number of Respondents & Time to complete Survey  1,656 Students completed surveys 1,175 Elementary; 481 Middle 1,175 Elementary; 481 Middle On average less than 10 minutes to complete survey (observed) On average less than 10 minutes to complete survey (observed) 51% elementary students report they took less than 10 minutes 51% elementary students report they took less than 10 minutes 61% middle school students report taking less than 10 minutes 61% middle school students report taking less than 10 minutes  105 Staff completed 769 student reports 54% report taking less than 5 minutes per student report 54% report taking less than 5 minutes per student report 46% report taking 5 to 9 minutes 46% report taking 5 to 9 minutes  70 Classroom Teachers completed 535 student reports 81 % report completing each student report in less than 5 minutes 81 % report completing each student report in less than 5 minutes 18% report taking 5-10 minutes per report 18% report taking 5-10 minutes per report 1% took more than 10 minutes 1% took more than 10 minutes

6. FIELD TEST (2010-2011) Overview of key elements:  Participation open to all programs in the state  Outreach effort to all regions : Presented on Field Test at Regional Lead Meeting (October 13, 2010) Presented on Field Test at Regional Lead Meeting (October 13, 2010) Sent Field Test materials electronically to RLs for dissemination to all grantees Sent Field Test materials electronically to RLs for dissemination to all grantees  Web-based surveys for students, program staff, and classroom teachers completed in fall 2010 and spring 2011 Programs select which surveys to administer Programs select which surveys to administer  Confidential summary report of survey results students’ scores of positive behavior change and skill development provided to participating programs students’ scores of positive behavior change and skill development provided to participating programs  All materials and instructions available on Project website  Technical assistance provided to programs Support provided by email and telephone during pre and post survey administrations & in interpreting their scores at the end of Field Test Support provided by email and telephone during pre and post survey administrations & in interpreting their scores at the end of Field Test  Measures modified based on Pilot results and feedback from field

7. FIELD TEST (2010-2011) Outcome Measures  Student —Elementary & Middle School— (5 subscales) Work Habits Work Habits Misconduct Misconduct Reading Efficacy Reading Efficacy Math Efficacy Math Efficacy Social Efficacy Social Efficacy  Staff (5 subscales) Work Habits Work Habits Aggressive Behavior with Peers Aggressive Behavior with Peers Prosocial Behavior with Peers Prosocial Behavior with Peers Task Persistence Task Persistence Social Skills Social Skills  Teacher (6 subscales) Same as Staff plus Mock Report Card Same as Staff plus Mock Report Card

8. FIELD TEST FINDINGS (Fall 2010 – Spring 2011)  Diverse programs across the state participate in fall 2010 field test 156 Sites (elementary & middle school) 156 Sites (elementary & middle school) 23 Programs (10 rural, 8 urban, 5 suburban) 23 Programs (10 rural, 8 urban, 5 suburban) 8 Regions represented 8 Regions represented  Pre-Surveys Completed (November 1 through December 17) 3934 elementary students 3934 elementary students 1847 middle school students 1847 middle school students 360 program staff completed 4611 student reports 360 program staff completed 4611 student reports 29 classroom teachers completed 185 student reports 29 classroom teachers completed 185 student reports  Initial feedback from Field Test sites positive Instructions clear Instructions clear Online administration easy Online administration easy Happy with the quality of technical support Happy with the quality of technical support  Field Test psychometrics consistent with Pilot results and previous studies

9. FIELD TEST FINDINGS Time to Complete Survey Elementary and Middle School Students Reported Time to Complete Survey PilotField Test Elementary Students N=1,175N=3934 < 5 minutes 23%30% 5-9 minutes 28%27% 10-15 minutes 16%10% More than 15 minutes 2%3% Don’t Know 31%30% Middle School Students N=481N=1847 < 5 minutes 24%37% 5-9 minutes 38%33% 10-15 minutes 8%5% More than 15 minutes 3%2% Don’t Know 27%23%

10. FIELD TEST FINDINGS Time to Complete Survey PilotField Test Teachers N = 70N = 29 < 5 minutes 81%50% 5-9 minutes 18%46% 10 minutes or more 1%4% Staff N = 105N = 360 < 5 minutes 54%61% 5-9 minutes 46%31% 10 minutes or more —4% Classroom Teacher and Staff Reported Time to Complete Each Student Report

11. NEXT STEPS ign up now?  Can programs sign up now? Yes, but they will not get a report of pre/post change scores, only the results of the spring survey administration. Yes, but they will not get a report of pre/post change scores, only the results of the spring survey administration.  Provide support to new participants enlisting in Field Test for spring 2011  Contact current Field Test participants to prepare for spring survey administration (April - May 2011)  Dissemination of findings in summer 2011 in collaboration with California Afterschool Network Regional Leads and CDE