An Evaluation Primer: A Plan for Measuring Program Quality and Outcomes in After School and Summer Programs by Brooke Culclasure, Ph.D. May 3, 2016

Introductions Presenter is Dr. Brooke Culclasure, research director in the Riley Institute at Furman University and principal investigator of numerous studies across S.C. Introductions of session attendees/program affiliation/description of current evaluation activities

Format of Presentation Discuss why evaluation is important Discuss why quality is important, related to evaluation Discus past outcomes evaluations and their link to quality Discuss available tools to measure quality Discuss QSA and how it has been used Breakout groups

Why is evaluation important? Provides us with useful information to help us know our strengths and weaknesses, how to improve our program, and better serve our students Helps us highlight successes and be able to tell our success stories Provides us with solid data to help us compete for resources

Why is quality important? In after school and summer learning programs, research has shown that only high-quality programs produce positive outcomes for youth Many studies have looked at test scores, engagement, and other factors…BUT programs rated as low quality do not show impact Studies by scholars such as Vandell, Noam and others all focus on quality as key in determining outcomes

How do we assess program quality? Becomes obvious, then, that we must focus on quality Numerous tools available, some are self-assessment and some are conducted by external evaluators There are 3-4 widely used tools, we will go into detail about one today Talk first about why self-assessment is a good choice

How do we assess program quality? Why Self-Assessment? It helps program providers to understand overall program quality and identify steps needed for improvement It is a low-stakes process It can focus on program needs and priorities while keeping areas outside the focus from slipping Low cost

How do we measure quality? Four widely used tools Youth Program Quality Assessment (PQA) A Program Assessment System Training (APAS): Only assessment that helps afterschool programs link quality to youth outcomes! Dimensions of Success Observation Tool (DOS): STEM focused Quality Self Assessment (QSA)

Quality Tools What is the QSA Tool? (NY State Network) The QSA Tool is organized around 10 elements of quality It uses a set of indicators to help program providers determine their strengths and areas in need of improvement It is structured to lead to an action plan for increasing program quality The tool was developed by experienced program and TA providers to be relevant and reasonable

What Makes a Quality Program? Ten Essential Elements of a Quality Afterschool Program Environment and Climate Administration and Organization Relationships Staffing and Professional Development Programming and Activities Linkages Between Day and Afterschool Youth Participation and Engagement Parent, Family, and Community Partnerships Program Sustainability and Growth Measuring Outcomes and Evaluation Source: NY State Network for Youth Success

QSA Tool vs. Evaluation QSA Tool: provides structure for dialogues around outcomes and quality. strengthens communication between stakeholders. helps build a collective vision. uses data gathered to inform practice and build capacity. is often facilitated by program staff and/or other stakeholders. Formal Evaluation: takes a more formal approach to assessing programs to determine what their outcomes are. often uses data gathered for accountability and compliance. has higher stakes and less gray areas. is often facilitated by an external observer.

Cycle of Improvement

Breakout Groups Divide into breakout groups Reconvene for discussion/reporting back Questions: Please contact Brooke Culclasure, ,