© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Program Quality & Youth Outcomes A Ready by 21® Quality Counts Webinar March 13, 2008 The Forum for Youth Investment

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Submit a Question at Any Time! To ask a question, go to the “Question and Answer” box in the upper right- hand corner of your screen. Click on the small arrow next to “Question and Answer,” type in your question, and click send.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Agenda Welcome, Review Goals, Speakers, Logistics - Nicole Yohalem Making the Case for Quality - Karen Pittman Population Level Outcomes & Indicators - Karen Finn Program/System Level Quality & Outcomes -Charles Smith Summary Reflections - Karen Pittman Question and Answer Period Closing, Review Key Resources – Nicole Yohalem

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Why is Program Quality the Focus of this Initiative? Making the Case for Quality Karen Pittman The Forum for Youth Investment

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Improving Community Supports is the intermediate step toward improving Youth Outcomes. While improving youth outcomes is the goal we are all working toward, we believe that to get there at scale, our focus needs to be on helping leaders improve the quality and reach of programs and services available in the community.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Increasing program availability isn’t enough. There is basic agreement that young people need programs and services – in particular that they need more structured, voluntary activities to fill their out-of-school hours. There is a basic belief, and evidence, that participation in these programs is associated with improved youth outcomes – academic, social, emotional, civic, health, management of risky behaviors. There is, consequently, growing public and political will, to increase youth participation in programs and to ensure that the programs are of good quality. There is a growing base of research that suggests that improvement in youth outcomes is strongly related to level of participation and level of program quality – more refined measures of both are needed. Measuring quality at the system level has always been challenging.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. What the research says. “Dosage and duration” matter. Young people who participate regularly in programs over a sustained period of time show gains in targeted outcomes. Those who attend sporadically don’t. Program quality also matters. Controlling for participation, young people who participate in high quality programs achieve greater gains than those who do not. Recent meta-analysis by Durlak and Weissberg suggests that low- quality programs may have no effect on adolescent outcomes.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Durlak and Weissberg study Recent meta-analysis by Durlak and Weissberg suggests that low-quality programs may have no effect on adolescent outcomes. 73 after-school programs reviewed grouped into 2 clusters based on SAFE criteria: Sequenced set of activities to achieve goals Active learning techniques for skill acquisition Focus, to some extent, on personal/social development Explicit objectives for personal and/or social skill development Programs that had the SAFE features showed positive effects on almost every outcome – school performance, social behavior, attitudes and beliefs. Programs that did not have the SAFE features showed no effect on any outcome.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Implications for Programs, Public and Policy Makers Quality counts as much as participation. Two youth, spending equal time in two programs of different quality will reap significantly different benefits. Overtime, the cumulative effect of participation in low quality programs, especially when coupled with participation in low quality schools and harried families, can be huge. Improving program quality at the point of service – where youth and adults interact – has the biggest impact on youth outcomes. But program quality is either not measured, not measured well or not measured consistently across programs. This means that increasing funds for program expansion and participation, without increasing funding and monitoring for quality, may net mediocre results.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. POS Quality Meets System Performance Measurement Results-based accountability helps program/system directors answer 3 questions: How much do we do? How well do we do it? Is anyone better off? These 3 questions can be easily linked to the concepts of participation, POS quality, and youth outcomes. Program/System Level Performance PARTICIPATION How much do we do? QUALITY How well do we do it? PROGRAM OUTCOMES Is anyone better off? Point of Service Quality Improved skills, knowledge & behaviors As measured by indicators e.g., % with improved decision-making skills Frequency, duration, intensity of participation

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Defining Terms POPULATION LEVEL Outcomes or Results: A condition of well-being for children, adults, families or communities (e.g. youth ready for college, work, life). Indicators: Measures that help quantify achievement of a result or outcome (e.g. rate of high school graduation, teen birth rates, employment rates). PROGRAM/SYSTEM LEVEL Performance Measurement: Measuring how well a program, organization or system is working: Participation. Measuring frequency, duration and intensity of program attendance. Program Outcomes: Impacts or changes for participants during or after a program, often expressed in terms of knowledge, skills or behaviors. Program Quality: Youth access to developmentally significant experiences in a setting. Although program quality can be defined more broadly, point of service quality refers primarily to staff practices during specific activities with children and youth. Based on Mark Friedman’s Results-Based Accountability

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Point of Service Quality: Youth access to developmentally significant experiences in a setting. Although program quality can be defined more broadly, point of service quality refers primarily to staff practices during specific activities with children and youth. Defining Terms

Connecting Thriving Leading Working Learning As measured by indicators e.g., Teen birth rate (common resources: Kids Count; Search internal assets) Program/System Level Performance PARTICIPATION How much do we do? QUALITY How well do we do it? PROGRAM OUTCOMES Is anyone better off? Point of Service Quality Improved skills, knowledge & behaviors As measured by indicators e.g., % with improved decision-making skills Frequency, duration, intensity of participation CHILD & YOUTH FAMILY & COMMUNITY OUTCOMES or RESULTS Caring Adults Safe Places Effective Ed Healthy Starts Opportunities to Contribute As measured by indicators e.g., % with caring adult (common resources: Search external assets) SUPPORTS or INPUTS POPULATIONPOPULATION LEVEL GOALSLEVEL GOALS e.g. How is our community doing?How are our young people doing?

The Forum for Youth Investment The Maryland Ready by 21 Action Agenda Karen Finn

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Maryland’s Child Well-Being Results Babies born healthy Healthy Children Children enter school ready to learn Children are successful in school Children complete school Children are safe in their families and communities Stable and economically self-sufficient families Communities that support family life

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Youth Told Us….. “They need to craft programs around older youths’ interests….bring in adults in the same field of interest and partner them with youth…” “….foster parents and youth workers…..show them how to connect with kids….”

Ready for College, Work & Life As measured by indicators e.g., educational attainment, employment, health care coverage, HIV/AIDS rate… Program/System Level Performance PARTICIPATION How much do we do? QUALITY How well do we do it? PROGRAM OUTCOMES Is anyone better off? Frequency, duration, intensity of participation Point of Service Quality Improved skills, knowledge & behaviors Maryland’s Ready by 21 Action Agenda DESIRED RESULTS Competent, Caring Adults Accessible, Affordable Housing Access to Health Care Pathways to Education and Employment Equal Treatment Under the Law ACTION AREAS POPULATIONPOPULATION LEVEL GOALSLEVEL GOALS How is our community doing?How are our young people doing?

The Forum for Youth Investment Quality & Outcomes at the Program/ System Level Charles Smith

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. POS Quality and Program Outcomes: OST Settings Durlak & Weissberg, 2007 SAFE NRC Blue Book, 2002 “research specifically linking…[setting] features to outcomes is rare…”, that “we are only beginning to understand how the different combinations of features in organized activities interact to promote positive development…”, and finally that “precisely which features are involved and how they co-act to produce specific developmental change has not yet been evaluated” (Mahoney et al., 2005, p ). Robert Halpern

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. POS Quality and Program Outcomes: Classroom Settings Research on Motivation: Relatedness, autonomy, and competence Research on Learning: Positive affect, active learning and meta-cognition

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. What is actually happening in 599 OST settings? Does prior research make sense to you? 5 = occurred for everyone 3 = occurred for some 1 = did not occur Reflect Plan Choose Engagement Reframing conflict Experience belonging Welcoming atmosphere Encouragement Skill building Active engagement Be in small groups Supportive Environment Interaction Safe Environment

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Findings from Several Samples POS quality findings: Supportive environment related to: Attendance Interaction related to: Interest in program Engagement related to: Sense of challenge, sense of growth, school-day reading, school-day suspension Note: No offerings get to high engagement without high support and high interaction Quality Improvement (YPQI) Findings

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Are the Youth PQA and YPQI Improvement Model RESEARCH BASED? YES – The Youth PQA has been the subject of lots of research can be considered a reliable and valid measurement instrument by widely shared standards of evaluation YES, conditional – The YPQI sequence has been tested in at least one high fidelity implementation and use of the YPQI sequence is related to POS quality change IN PROCESS – The YPQI sequence is currently the subject of a randomized field trial, results available in fall 2008 Fidelity to broad developmentally focused intervention models

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. How to Talk about Outcomes OST programs can produced positive change in social-emotional skills and academic achievement for both at-risk and lower-risk populations of children and youth. Not all OST programs produce positive change so we should be careful to protect investment by doing quality assurance. It’s cost effective to ensure that staff are actually doing good stuff. Programs can raise child outcomes by striving to improve the quality of their offerings. Broad developmentally-focused programming is a research-based strategy.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Question & Answer Period To ask a question, go to the “Question and Answer” box in the upper right- hand corner of your screen. Click on the small arrow next to “Question and Answer,” type in your question, and click send.

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Program Quality & Youth Outcomes A Ready by 21 Quality Counts Webinar March 13, 2008 The Forum for Youth Investment

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Key Readings Population Level Trying Hard is Not Good Enough (Mark Freidman) Finding Out What Matters for Youth (Gambone, Klem & Connell, 2002) Maryland’s Ready by 21 Five-Year Action Agenda Forum Focus: What Gets Measured Gets Done Steering a Course Toward Effective Youth Policies: Dashboards for Youth Program/System Level YPQA validation study (Smith & Hohmann, 2005) Afterschool Quality and School-Day Outcomes (Blazevski & Smith, 2007) Improving Afterschool Program Quality (Granger, Durlak, Yohalem, Reisner, 2007) The Impact of After-school Programs…(Durlak & Weissberg, 2007) Participation During Out-of-School Time (Forum for Youth Investment, 2004)

© The Forum for Youth Investment Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment. Program Quality & Youth Outcomes A Ready by 21 Quality Counts Webinar March 13, 2008 The Forum for Youth Investment