Neil Naftzger Principal Researcher Washington 21st CCLC Evaluation February 2015 Copyright © 20XX American Institutes for Research. All rights reserved Data Collection Activities

 Evaluation update  Leading indicator surveys  Youth surveys Topics 2

 Evaluation Activities in 2015 Leading indicator surveys (February to March 2015) Youth surveys (April to May 2015) Data for federal reporting (timeline uncertain) Case study site selection and recruitment (Summer 2015) Leading indicator reports and planning with data (Fall 2015) Case study site visits begin (Fall 2015) Evaluation Update 3

 Used to populate the leading indicator reports Data obtained from federal reporting systems Surveys PQA data Youth outcome data  Purpose is to inform quality improvement efforts  Leading indicator surveys collect data from two groups: Site coordinators Staff working directly with youth in the delivery of programming Leading Indicator Surveys 4

 s to project directors containing a link to the survey management system and a username and password will be sent during the week of February 16th  Complete surveys from site coordinators and afterschool program staff are due March 31st Leading Indicator Survey Timeline 5

Survey Administration Process Once logged into the Project Director Survey Management page: Ensure the centers listed for a given grant are correct and send any corrections needed to Under the name of each center, a survey Web link unique to each center is provided This link needs to be forwarded to the site coordinator for that center When the link is clicked on, the respondent will be asked to create a survey account and then be prompted to take the survey in question Site coordinators and staff will utilize the same link to create a survey account and take the survey for their role in the program You can return to the Project Director Survey Management page to view the completion status of surveys and send reminders

Staff Selection In addition to the site coordinator, ideally, a minimum of 8 staff per site should be identified per center Staff selected to the survey:  Need to spend the majority of their time working directly with youth by leading activities  Can be paid or a volunteer  Can work for the grantee directly or for a partner organization  Can be working in the program or recently ended their participation in the program (e.g., led activities in a previous session that was active at some point in )  Should work (or did work) in the program at least once a week If there is less than 8 staff at a given center that meet these criteria, then all staff at the center meeting these criteria should be asked to take the survey

Project Director Survey Management Page

Center Coordinator and Staff Login Page Staff asked to complete student reports will need to first create an account

Creating a New Survey Account Completion of this field will determine if staff take the site coordinator or staff survey Clicking the create user button will take the user to either the staff survey or site coordinator management page

Monitoring Survey Completion Project directors can see both: Who has completed a survey account Whether or not they have completed their survey from the Project Director Survey Management page Automated reminders can be sent directly from the Project Director Survey Management Page Project directors are encouraged to monitor the survey process frequently

Youth Survey 12

 Piloted a revised version of the Student Engagement, Motivation, & Beliefs Survey developed by the Youth Development Executives of King County  Pilot involved 38 centers funded by 21st CCLC  A total of 1,199 surveys were collected from 21st CCLC participants enrolled in grades 3-10 Youth Survey Pilot 13

 Based on pilot results, the survey was revised to improve its functioning  Current version comprised of the following scales (54 items): Academic identity Mindsets Self-management School belonging Interpersonal skills Retrospective program impact on academic behaviors Retrospective program impact on self-management Program belonging and engagement Youth Survey Pilot 14

Revised Self-Management Scale 15

Revised Self-Management Scale 16

 Assessing if youth survey scores are correlated with school-related outcomes State assessment results in reading and mathematics School-day attendance Disciplinary incidents  Examining how these relationships may vary by grade level  Additional revisions may be made to the survey based on results from these analyses Youth Survey Next Steps 17

 All centers serving youth in grades 4-12 will be asked to collect youth survey data in the spring of 2015  This data collection replaces the teacher survey (programs can continue to collect these data if they find this information useful for local evaluation purposes, but OSPI will not collect these data)  Preference to collect data online Let us know if this is going to be an issue by sending an to Youth Survey Next Steps 18

 Survey will involve a sample of participating youth in targeted grade levels that meet certain program participation levels  The statewide student identifier will be collected in relation to youth taking the survey or some similar method will be employed to connect survey data with the state data warehouses  A draft of the survey will be available by March 1st, along with additional information on data collection procedures  Online collection of youth survey data will begin on April 1 and continue through May 29 Youth Survey Next Steps 19

 A separate Webinar will be scheduled to review the survey interface  For this cycle, we are interested in collecting responses from youth on all survey scales  Providing programs with an option of selecting scales relevant to their programs in the future is under consideration  Results from the youth survey will be made available in the leading indicator reports Youth Survey Next Steps 20

Neil Naftzger East Diehl Road, Suite 200 Naperville, IL General Information: