1. Accountability for Quality: Policy Innovation in Out-of-school Time Programs November 2, 2007

2. Introduction Why focus on quality? –Quality matters –Quality is measurable –Quality is malleable

3. Introduction A New Guide to Quality Metrics Assessing Afterschool Program Practices Tool (APT) National Institute on Out-of-School Time and the MA Department of Education Out-of-School Time Observation Instrument (OST) Policy Studies Associates Program Quality Observation (PQO) Deborah Vandell and Kim Pierce Promising Practices Rating Scale (PPRS) WI Center for Education Research and Policy Studies Associates, Inc. Program Observation Tool (POT) National Afterschool Association Quality Assurance System (QAS) Foundations Inc. Program Quality Self-Assessment Tool (QSA) New York State Afterschool Network School-Age Care Environment Rating Scale (SACERS) Frank Porter Graham Child Development Center, UNC Youth Program Quality Assessment (YPQA) High/Scope Educational Research Foundation

4. Introduction Exemplar: The Youth Program Quality Assessment (YPQA) YPQA is a good example of new class of tools that: –Produce data that leads to real change in staff/teacher performance –Provide continuity that is place-based, not silo-based –Link accountability policy with workforce development policy –Offer a more efficient and effective use of resources than a sole focus on measuring child outcomes

5. Part I. About the YPQA –Characteristics Observation at point of service Takes 1-2hours, training available but not required Applies to across content areas –Purposes: Metric for most important part of education and human service programs – point of service Increase access to experiences that motivate clients to attend and engage Staff learning and performance change Foundation for more effective accountability

6. Part II: How YPQA Works in Programs High/Scope YPQA Pilot Site Safe Environment Supportive Environment Peer Interaction Youth Engagement Psychological & emotional safety Physically safe environment Emergency proceduresProgram space and furniture Healthy food and drinks Welcoming atmosphere Session flowActive engagement Skill building Encouragement Reframing conflict Experience belonging Be in small groups Lead and mentor Partner with adults Make choices PlanReflect 1.33 2.17 4.01 4.92 Winter 2007 Youth Program Quality Assessment 5 2.53 2.96 3.78 4.41 North Branch Norm Scores* * The norm column represents mean scores for approximately 635 offerings within 170 programs collected during the years 2001-2006.

7. Part II. YPQA in Programs North Branch Area Community Education Provide a comprehensive learning program for all ages; Seek out human resources in our community; Develop citizen advisory councils; Strive toward minimum use of facilities

8. Part II: YPQA in Programs Youth Classes 2006-2007 We offered 178 youth classes in our Fall, Winter, and Summer sessions, having a total of 3482 participants.

9. Part II: YPQA in Programs North Branch Area SAC Program Goals School Age Care (SAC) program will provide appropriate activities which will positively contribute to the total development of each youth in areas of: personal discipline social development physical skills health and safety skills

10. Part II: YPQA in Programs Program, Plan and Staff Procedures Handbook Program Curriculum Examples provided by State Department Behavior Guidance and Procedure Procedures for Tornado/Fire/Lock Down Drills Accidents/First Aid Staff Guidelines for Working with Children

11. Part II: YPQA in Programs So what? Now what? If you always do what you always did, then you will always get what you have always got. Our youth need us to do better than we currently do. Program Quality Defined in North Branch: Atmosphere + Activities = Quality - But how do we get more specific?

12. Part II: YPQA in Programs Quality Scores in North Branch Safe Environment Supportive Environment Peer Interaction Youth Engagement Psychological & emotional safety Physically safe environment Emergency proceduresProgram space and furniture Healthy food and drinks Welcoming atmosphere Session flowActive engagement Skill building Encouragement Reframing conflict Experience belonging Be in small groups Lead and mentor Partner with adults Make choices PlanReflect 1.33 2.17 4.01 4.92 Winter 2007 YPQA Scores

13. Part II: YPQA in Programs Data-Driven Improvement Goals Activities support active engagement. (YPQA quality scale II-H) - Creative new science lessons to supplement current themes.

14. Part II: YPQA in Programs Data-Driven Improvement Goals Youth have an opportunity to develop a sense of belonging. (YPQA quality scale III-L) Youth have an opportunity to participate in small groups. (YPQA quality scale III-M)

15. Part II: YPQA in Programs Data-Driven Improvement Goals Youth have opportunities to act as group facilitators and mentors. (YPQA quality scale III-N)

16. Part II: YPQA in Programs Challenges and Opportunities Staff and student habits are hard to change. Time and funds for staff training on new strategies. New lens through which to see our program. Creative, purposeful planning. Engaged kids!

17. Part III. YPQA in State Accountability Policy Context Child Care Services School Age Child Care County Youth Bureaus NYC Out of School Time State Education Department 21 st Century Community Learning Centers TANF - Advantage After School Program

18. Part III. YPQA in State Accountability Policy Three similar quality trains are moving in NY state, all focused on raising quality and building more effective forms of accountability: –NYSAN Quality Self Assessment –Accreditation for School Age Child Care –YPQA

19. Part III. YPQA in State Accountability Policy Key roles for state policy –Set clear priorities about quality –Fund provider networks to support implementation of quality policy –Meet provider-level demand for more effective forms of accountability

20. Part III. YPQA in State Accountability Policy How YPQA is used in Advantage After School Programs

21. Part IV. Scope of Use YPQA is part of state and county accountability policies: –Cross sector (DHS& DOE) snapshots: IA, WA –Statewide 21st Century: MI, ME, MN, RI –Cities and Counties: Rochester, Detroit, Grand Rapids, Palm Beach

22. STEP 1 Decide to build system STEP 2a Program Self- assessment STEP 2b External assessment STEP 3 Plan for improve- ment STEP 4 Targeted T&TA STEP 5 External assessment with criterion IMPROVEMENT Sequence Accountability Policy Elements: External Data Collection, Reporting & Data Analysis, Training, Coaching MONITORING & ACCOUNTABILITY Sequence Part IV. Scope of Use Generic Acct Policy Elements Using Quality Metric STEP 6 Re-assess and move forward!

23. Part IV. Scope of Use Leveraging Impact from Existing Change Resources POS Quality Access to key developmental and learning experiences Professional Development Local Evaluators Accountability Measurement

24. Summary The field is looking for more effective ways to spend resources that are already dedicated to accountability, evaluation, and professional development. Observational quality assessment metrics are increasingly the answer Youth PQA (and tools like it) allow us to measure the most important elements of programs – and raises the possibility of increased returns on existing investments With the right kind of supports, this form of accountability can be effective and sustainable part of programs that serve children and youth Constituencies of state government want to move in this direction – want to be accountable in ways that actually and demonstrably change the quality of services available to youth State and local governments are already experimenting with new accountability policies based on the youth PQA – and they are affordable