1. ASSESSING AND IMPROVING YOUTH PROGRAM QUALITY Nicole Yohalem October 2008

2. © The Forum for Youth Investment 2008 Outline Intro discussion Making the case for investing in quality Where is the field? Defining Quality Assessing Quality Improving Quality

3. © The Forum for Youth Investment 2008 Features of Positive Developmental Settings National Research Council, 2002 Physical and Psychological Safety Appropriate Structure Supportive Relationships Opportunities to Belong Positive Social Norms Support for Efficacy and Mattering Opportunities for Skill-Building Integration of Family, School and Community efforts

4. © The Forum for Youth Investment 2008 “Communities should put in place some locally appropriate mechanism for monitoring the availability, accessibility and quality of programs…” - Community Programs to Promote Youth Development, 2002 National Research Council Recommendation

5. © The Forum for Youth Investment 2008 Discussion In your organizational context, which of these features do you consider strengths? Which represent areas of growth? Are you trying to measure any of these in your settings? Any common themes at your table?

6. © The Forum for Youth Investment 2008 Making the Case: Why focus on quality? 1. Because too few young people get the supports they need to thrive. According to the America’s Promise Alliance National Promises Survey, only 31% of 6-17 year olds have at least 4 of the 5 promises. 21% have 1 or none. 1. Safe Places 2. Caring Adults 3. Effective Education 4. Opportunities to Help Others 5. Healthy Start

7. © The Forum for Youth Investment 2008 Why focus on quality? 2. Because participating in OST programs can make a difference. Several empirical reviews of the effects of programs conducted over the past decade show that on average, programs have positive effects on social, emotional and academic outcomes. (Durlak & Weissberg, 2007; Bodilly & Beckett, 2005; Lauer et al., 2006).

8. © The Forum for Youth Investment 2008 Why focus on quality? 3. Because the quality of OST program matters. Durlak and Weissberg, 2007 73 programs reviewed grouped into 2 clusters based on SAFE criteria: (Sequenced, Active, Focused, Explicit) Programs that had the SAFE features showed positive effects on almost every outcome – school performance, social behavior, attitudes and beliefs. Programs that did not have the SAFE features showed no effect on any outcome. Other studies: Vandell, MARS, High/Scope’s YPQA Validation Study

9. © The Forum for Youth Investment 2008 Why focus on quality? 4. Because lots of children and youth spend time in these settings. Afterschool Alliance estimates 6.5 million children are in after-school programs. The parents of another 15.3 million say their child would participate were programs available. State of California moving toward universal access – elementary and middle.

10. © The Forum for Youth Investment 2008 Why focus on quality? 5. Because significant public and private dollars flow into these settings. The Finance Project estimated a $3.6 billion federal investment in after-school 2002. State and local funds increasingly important. California’s recent commitment of $550 million. Major private investments by national foundations (Wallace, Mott Atlantic, Clark, Robert Wood Johnson, W.T. Grant) and regional foundations (Nellie Mae, Skillman, William Penn, Colorado Trust).

11. © The Forum for Youth Investment 2008 Why focus on quality? 6. Because we know quality varies significantly within and across settings. All programs are not created equal (Durlak & Weissberg). Despite the variation, there are some patterns. Overall, programs do better on traditionally regulated things like safety than they do on higher- order things like engagement, interactions. Not all OST programs produce positive change. Funders can protect their investments by doing quality assurance.

12. © The Forum for Youth Investment 2008 Why focus on quality? 7. Because quality is measurable, and even more important – it is malleable. Palm Beach County, FL. Programs conduct assessments, sites develop improvement plans targeting specific areas, receive quality coaching, gains made in every area targeted by staff. American Camp Association. Based on youth surveys, leadership sets initial targets in partnership with youth and staff, develop an action plan, re-do survey. 17 out of the 23 camps had positive change in at least one area.

13. © The Forum for Youth Investment 2008 Why focus on quality? 8. Because measuring program quality is an efficient, affordable, and productive alternative (or complement) to measuring youth outcomes. There is more agreement in the field about the components of quality than about appropriate outcomes. Demonstrating impact is important, but doing it in such a way that individual programs can take credit for the change is extremely difficult, expensive and not necessarily instructive.

14. © The Forum for Youth Investment 2008 Why focus on quality? 1. Too few young people are getting the supports they need to thrive. 2. Participating in community programs can make a difference. 3. The quality of community programs matters. 4. Lots of children and youth spend time in these settings. 5. Significant public and private dollars flow into these settings. 6. We know quality varies significantly with and across settings. 7. Quality is measurable and malleable. 8. Measuring program quality can is an efficient, affordable, and productive alternative (or complement) to measuring youth outcomes.

15. © The Forum for Youth Investment 2008 Why focus on quality? Quality matters. Quality is measurable. Quality is malleable.

16. © The Forum for Youth Investment 2008 Ages Times of Day Outcome Areas (Cognitive, Physical, Social, Civic, Vocational…) ? ? ? Quality and Reach Count Morning... Night 21. 0 School After School At best, school only fills a portion of developmental space

17. © The Forum for Youth Investment 2008 Where is the field? Attention to and interest in quality has increased significantly over the past several years among key stakeholder groups in OST. There is consensus about what matters. Increasingly, evidence suggests our focus should be on the point of service. A lot of work is underway to refine assessment tools and develop systemic approaches to quality improvement. Washington is a site in the Ready by 21 Quality Counts initiative.

18. © The Forum for Youth Investment 2008 Measuring Youth Program Quality: A Guide to Assessment Tools Assessing Afterschool Program Practices Tool (APT) National Institute on Out-of-School Time and the MA Department of Education Out-of-School Time Observation Instrument (OST) Policy Studies Associates Program Observation Tool (POT) National Afterschool Association Program Quality Observation (PQO) Deborah Vandell and Kim Pierce Promising Practices Rating Scale (PPRS) WI Center for Education Research and Policy Studies Associates, Inc. Quality Assurance System (QAS) Foundations Inc. Program Quality Self-Assessment Tool (QSA) New York State Afterschool Network School-Age Care Environment Rating Scale (SACERS) Frank Porter Graham Child Development Center, UNC Youth Program Quality Assessment (YPQA) High/Scope Educational Research Foundation

19. © The Forum for Youth Investment 2008 Measuring Youth Program Quality: A Guide to Assessment Tools Purpose and History Content Structure and Methodology Technical Properties User Considerations Application in the Field

20. © The Forum for Youth Investment 2008 How is quality defined? There is a lot of similarity across definitions. Common elements include: Relationships Environment Engagement Social/Behavioral Norms Skill Building Opportunities Routine/Structure

21. © The Forum for Youth Investment 2008

22. How is quality measured? There are more differences in how quality is measured than in how it is defined. Why the differences in emphasis and approach? Tool purposes (regulatory, self-assessment) Program purposes (achievement, recreation) Developers’ perspectives & backgrounds (ECE, YD)

23. © The Forum for Youth Investment 2008 Differences in emphasis SACERSYPQA Social Processes/ Interactions 9 items14 items Resources (financial, human, material) 35 items16 items

24. © The Forum for Youth Investment 2008 Differences in approach Data collection methods Types of measures Technical properties

25. © The Forum for Youth Investment 2008 Data collection methods Type (observation, interview, questionnaire, document review) Target users (line staff, program leaders, youth, external observers) Intensity of data collection

26. © The Forum for Youth Investment 2008 Types of measures: high vs. low inference Program Observation Tool Staff are engaged with children Youth Program Quality Assessment During activities, staff generally smile, use friendly gestures, and make eye contact. Staff encourage all youth to try out new skills or attempt higher levels of performance. During activities, staff are almost always actively involved with youth (e.g. they provide directions, answer questions, work as partners or team members, check in with individuals or groups). Staff make use of frequent open-ended questions.

27. © The Forum for Youth Investment 2008 Types of measures: diagnostic vs. prescriptive Diagnostic – NY QSA A quality program provides participants with a variety of engagement strategies. Diagnostic and prescriptive - APT Youth are busy and engaged in conversation or activities. Youth appear relaxed and in control of themselves. Youth independently gather resources, materials or get information. Youth help select, lead or contribute to the running of the activity. Youth solve problems alone or in groups. When trying to solve a problem, youth try to identify the source, nature of the problem and/or try out potential solutions.

28. © The Forum for Youth Investment 2008 Technical properties

29. © The Forum for Youth Investment 2008 The YPQA is a good example of new class of tools that: Produce data that leads to real change in staff performance Provide continuity that is place-based (not silo-based) Link accountability policy with workforce development Offer a more efficient and effective use of resources than a sole focus on measuring child outcomes The YPQA: Assessing point of service quality

30. © The Forum for Youth Investment 2008 The YPQA: Assessing point of service q uality Plan Make choices Reflect Partner with adults Lead and mentor Be in small groups Experience belonging Engagement Reframing conflictEncouragement Skill building Active engagement Session flow Welcoming atmosphere Supportive Environment Interaction Safe Environment Healthy food and drinks Program space and furnitureEmergency procedures Psychological and emotional safety Physically safe environment

31. © The Forum for Youth Investment 2008 Safety Support Interaction Engage SAE System Accountability Environment PLC Professional Learning Community POS Point Of Service Quality in context: Multi-level systems

32. © The Forum for Youth Investment 2008 Youth PQA scores

33. © The Forum for Youth Investment 2008 What quality looks like on the ground “Positive Youth Development” 28% “Staff Centered” 39% “Low quality” 33% Sample of nearly 600 different youth workers

34. © The Forum for Youth Investment 2008 Approaches to Improving Quality Approaches differ along several dimensions: Nature of Agency Involvement (mandatory/voluntary) Level of Accountability (high stakes/low stakes) Reach (universal/targeted) Source of Expertise (internal/external capacity) Focus of Change (organizational issues/staff practice) Staff Level Targeted (targets leadership/line staff) Type of Data Collected (high/low inference; diagnostic/prescriptive) Support Strategy (one-on-one/group support)

35. © The Forum for Youth Investment 2008 Common lessons Quality assessment can advance multiple goals Data about their own practice is a powerful motivator for staff Common language helps pave the way for change Important to couple standards and assessment with tangible supports Building Quality Improvement Systems, 2007

36. © The Forum for Youth Investment 2008 mbus etroit Minneapolis Kentucky Iowa Oklahoma New York Rhode Island Austin Georgetown Divide Columbus Indianapolis Grand Rapids Nashville St. Louis Washington* West Palm Beach County Rochester Chicago Quality improvement systems using the YPQA YPQA is part of state and local quality improvement efforts: –Washington, New York, Iowa, Kentucky, Oklahoma (part of Ready by 21 Quality Counts) –Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico –Cities and Counties: Rochester, Syracuse, Detroit, Grand Rapids, Palm Beach, St. Louis, Nashville, Austin, Georgetown Divide CA, Indianapolis, Columbus IN, Chicago

37. © The Forum for Youth Investment 2008 Core components of a quality improvement system Self-assessment External assessment Action planning (with data) Training Coaching/advising

38. © The Forum for Youth Investment 2008 STEP 1 Decide to build system STEP 2a Program Self- assessment STEP 2b External assessment STEP 3 Plan for improve- ment STEP 4 Carry out plan IMPROVEMENT MONITORING ACCOUNTABILITY STEP 6 Re-assess and move forward! STEP 5 External assessment w/ criterion Components Self-assessment External assessment Action planning with data Training Coaching Example: Palm Beach, 60 sites county-wide; multi-year model

39. © The Forum for Youth Investment 2008 STEP 1 Decide to build system STEP 2a Program Self- assessment STEP 2b External assessment STEP 3 Plan for improve- ment STEP 4 Carry out plan IMPROVEMENT MONITORING ACCOUNTABILITY STEP 6 Re-assess and move forward! STEP 5 External assessment with criterion Components Self-assessment External Data Collection Action Planning (with Data) Training Coaching Example: Minnesota 115 sites statewide; multi-year model

40. © The Forum for Youth Investment 2008 What does the future hold? A QIS will gain more traction and more likely be effective and sustainable if the community/system also has solid information about: The overall program landscape (the full range of providers that could be engaged in quality improvement) The youth-serving workforce (who youth workers are, backgrounds and ambitions, turnover rates, full vs. part-time, etc.) Program participation Child and youth outcomes And if this information is part of an integrated information system

41. Measuring Youth Program Quality: A Guide to Assessment Tools www.forumfyi.org/node/297 Building Quality Improvement Systems www.forumfyi.org/node/299 www.forumfyi.org/node/297 www.forumfyi.org/node/299 Nicole Yohalem nicole@forumfyi.org