MASN, Evaluation, and PQAs MASN Overview – Systems building Quality Framework Training and Technical Assistance – Site Visits – Quality Action Plans Evaluation – Statewide Evaluation – Local/External Evaluation Data Collection – Survey data collection – Kids Care Center data – PQA data collection

MASN builds systems across the state that improve, support, and sustain high quality afterschool programs. MASN is one of 40+ Statewide Afterschool Networks (SANs) funded by the Mott Foundation.

MASN Goals Goal 1 Foster partnerships at the local, state, and national level to support afterschool professionals and programs Goal 2 Develop public support and garner resources to advance and sustain afterschool programs Goal 3 Ensure quality by providing supports to afterschool programs that are necessary to meet and exceed the Missouri Quality Standards

Two Avenues to Meet our Goals MASN Steering Committee Funded Projects MOARCKCC Project LIFTOFF MOSAC2 Coordination Integrated PD System Funding & Sustainability Public Awareness Public PolicyQualitySTEM

MASN Steering Committee Meets quarterly (December 10 th, February?, May ?) Sub-committee meetings are open to everyone – 10 a.m.-noon, people participating Networking lunch – Noon-1p.m., with sub-committee reports Steering Committee Leadership Meeting – 1- 3 p.m., just the Steering Committee members – Note: Training option available during this time.

Sample MASN Quarterly Meeting Schedule Salon AColumbiaMeadowsPinesLewis & ClarkTimberlands 10:00-10:15Full group welcome 10:15-12:00 Committee Meetings QualityPublic Awareness Funding & Sustainability STEMPublic Policy/ Advocacy 12:00-12:45Full group networking lunch 12:45-1:00Reports from the morning meetings of the sub- committees 1:00-3:00 Business Meetings MASN Steering Committee Training Session MO Girls CSI

Who are the Steering Committee members? – 4-H – ARCHS – Boys & Girls Clubs – Charter Schools – Child Care Aware® of Missouri – DESE – DHSS – DSS – Francis Institute – LINC – Missouri Chamber of Commerce – MOSAC 2 – OPEN Initiative – United for Children – United Way of Greater KC – YMCA – Independent program representatives that are serving as committee chairs

Quality Framework

Afterschool Logic Model Afterschool Program Quality Afterschool Program Quality Youth Outcomes Youth Outcomes College and Career Readiness /Success AS Staff Skills AS Program Structure Training Training Coaching Coaching Self- assessment Self- assessment

Weikart Center for Youth Program Quality Youth Program Quality Improvement (YPQI)

Assess – KCC, Surveys, PQAs Youth Survey Family Survey Coordinator/Director Survey Site Staff Survey School Administrator Survey Community Partner Survey

Plan – TA Visits & APWD Grantee Level Action Plans – Coaching visits Site Level Action Plans – Advanced Planning with Data (Y2, Y4) – Coaching visits

Do – Coaching and Training Coaching visits Methods Trainings Summits Webinars Statewide Conferences

Missouri Afterschool Resource Center (MOARC) Provides training and technical assistance to 21 st CCLC and SAC grantees. Supports successful implementation of grants and quality improvement process.

Technical Assistance 2 Program Visits per grantee – Program Visit Discussion Checklist – Program Level Action Plan 2 Site Visits per site (many grants have multiple sites) – Site Visit Discussion Checklist, Focused Observation, Feedback – Site Level Action Plan

Training Advanced Planning with Data – Fall after PQA (Y2, Y4) Four clock-hour webinars this Spring – Parent and Family Involvement – Community Partnerships – Marketing Your Program – Afterschool Enrichment Methods Trainings – Based on action plans developed at the Advanced Planning with Data session – This year, one training per grant (Cohort 6 and 7) Cohort 8 and SAC may attend if space available

Evaluation Statewide and Local (Outside) Evaluation

Evaluation Work MASN is coordinating the new Statewide and Local Evaluation and processes – Sub contracting with Weikart Center and OSEDA for survey development, data analysis, and report writing Quality Committee - looking at the resources being available for all programs – Providing training/certification of external evaluators

Coordinated Evaluation Local evaluation parallels statewide evaluation – Same data and framework for all grantees Same data used for quality improvement efforts and evaluation efforts

PQA/Survey Report Fee $1500 per site to 4-H Center for Youth Development – Data: PQA(s) Surveys – Reports: PQA(s) Weikart Center Leading Indicators (Goal 2) Missouri College & Career Readiness (Goals 1 & 3)

Local Evaluation Fee $2000 per grantee, includes one site, $200 per additional site – Consulting: Local context Data analysis Guided Reflection document (report) Total cost to grantees for data collection and certified evaluator averages out to almost the same per site as the average spent per site in FY13 (prior to this new system).

Logic Model Afterschool Program Quality Afterschool Program Quality Youth Outcomes Youth Outcomes College and Career Readiness /Success AS Staff Skills AS Program Structure Training Training Coaching Coaching Self- assessment Self- assessment

Goals Goal 1: Support or increase student achievement and sense of competence in the areas of reading/communication arts, mathematics, and science. Goal 2: Develop and maintain a quality program that includes a safe and supportive environment, positive interactions, and meaningful opportunities for engagement. Goal 3: Enhance youth’s college and career readiness skills and behaviors, including positive school behaviors, personal and social skills, and commitment to learning.

Data Sources used for the Statewide Evaluation Program Quality Assessments (PQAs) Surveys Kids Care Center (KCC) Data DESE Core Data

Program Quality Assessments (PQA) Developed by the Weikart Center for Youth Program Quality – School Age Program Quality Assessment (SAPQA) Walk-Through Method – Youth Program Quality Assessment (YPQA), with STEM Add-on On-site assessment by trained, reliable external assessors

Surveys Survey links sent by March 2015 to grant contact – One per site – coded with Org ID so it is important to make sure to use the correct links Paper survey available for parents and Spanish parent/youth version Youth survey requires a student identifier and “opt out” consent Weekly with number of completed surveys

l&deptcode=11558

Kids Care Center (KCC) Statewide database used to collect youth enrollment and activity participation KCC training and technical assistance Data used for Statewide and Local Evaluation – KCC ID used for the surveys – MOSIS ID - 9 digit number from DESE Data is entered into federal system for reporting

KCC Variables Afterschool attendance Grades Missouri Assessment Program (MAP) test scores Teacher surveys

DESE Core Data State student-level education database Core Data variables – School day attendance – Behavior (suspension)

Statewide Evaluation Sneak Peak at FY14 Report

Statewide Results Data received from 185 sites 9015 youth 2880 parents 242 site coordinators 1280 site staff 232 school administrators 330 community partners

Preliminary Statewide Results: Instructional Quality and Overall Quality

Preliminary Statewide Results: Youth Outcomes

Preliminary Statewide Results: Parent Outcomes

Predicting Grade Maintenance/Increase Calculated multiple regressions predicting maintenance/increase of grades in three subject areas. All youth scales and demographic variables were entered. Significant predictors shown below.

Statewide Evaluation Required by Federal Funding Used to: – Refine the grant requirements – Determine training and technical assistance needs – Plan for systems level changes or new development that will support afterschool sites – Use aggregate data to communicate the importance and impacts of afterschool

Local (Outside) Evaluation 21 st CCLC Cohort 8

Ensuring Consistency New model – Same goals, objectives, and data as Statewide Evaluation Outside Evaluators – must attend a training to become “certified” – all paid the same rate, receive the same data, and submit the same report

Local Evaluation Process Local Evaluators are trained on the data and process Local Evaluator and Program Administration meet to complete Guided Reflection documentation – Deep conversation about local context, analysis of the data, and recommendations based on this information

Local Evaluation Timeline January – Local (Outside) Evaluator training February – Certified Evaluator list available March-May – Grantees select and contract with Certified Evaluators By June 30 – Local Evaluator and Program Administration meet to discuss local context August 15 - Data available to Local Evaluator and Program Administrator Prior to October 15 – Local Evaluator and Program Administrator meet to complete Guided Reflection document October 15 – Guided Reflection document due to DESE

Local (External) Evaluation Required by 21 st CCLC Grant Funding Used to: – Document need/success of grants – Set program and site goals with coaches – Target training resources

Tying it all Together Statewide and Local Evaluation Afterschool Program Quality Afterschool Program Quality Youth Outcomes Youth Outcomes College and Career Readiness /Success AS Staff Skills AS Program Structure Training Training Coaching Coaching Self- assessment Self- assessment Goal 2 Goal 3 Goal 1

Youth Program Quality Improvement (YPQI) Surveys KCC Core Data PQAs Program Level Action Plan Site Level Action Plan (Advanced Planning with Data) Coaching Visits Methods Trainings Summits Conferences Other Trainings

Instrument Selection Consent Scheduling Reports Coaching PQA Specifics

Program Quality Assessments Developed by the Weikart Center for Youth Development Research based, reliable, and valid tool for self-assessment and external evaluation Used in many states and localities

l Light Green- full-state implementation Dark Green- place-based implementation Gold- full-state + place-based YPQI Quality Improvement Systems

Program Quality Assessments Four domains – Multiple scales in a domain Multiple items in a scale

Ones, threes, fives, Oh My!!! 1 – Not observed on the day of the observation 3 – Was observed at least once on the day of the observation – Examples: once, not all youth, some youth 5 – Was observed multiple times on the day of the observation – Examples: all youth had opportunities

PQA Protocols School Age Walk Through (SAPQA) – K – 6 th Grades – From start of programming, through departure – Limit of 3 hours – Welcome, multiple activities, educational offering, snack, transitions, departure STEM Program Quality Assessment (STEM PQA) – 4 th – 12 th Grades – Two offerings – one STEM, other on same day

PQA Process Schedule the assessment Confirm with program and assessor Assessment takes place Data Entry (2-4 weeks) Quality Control Report – available in Weikart Center Scores Reporter Discuss PQA results with Coach at Program and Site Visits Advanced Planning with Data – set site level action plans (Sept/Oct) Methods Trainings

During the Visit Please tell your staff that someone is coming to observe Assessors will dress casually – 4-H Extension nametag/shirt They will be a “fly on the wall” – Once the program starts, they will be observing and taking notes so please do not try to hold a conversation with them. They will confirm with the site director and arrive early to hear about the program/schedule for the day. They will go from area to area – may not see everything, but that is ok… it is a “snapshot” – 2-3 hours is usually sufficient to have a good feel for the program even if they do not observe every specific interaction Assessors will say goodbye to a staff member, but it may not be the site director, if not readily available Assessor may ask on the confirmation call or before the assessment begins which activities you would prefer to have observed if there are multiple offerings (except if older youth STEM)

Participation Agreement Key important policies Sign and return to Anne Reeder – Today or by December 15 th

Questions?