Laenne Thompson Webinars September 5 & 10, 2012 Join us on the phone for the audio portion: ReadyTalk Dial-In Number(s): 866.740.1260 Access Code: 2073334 2012 NYS-OCFS Quality Accountability and Improvement System kickoff Laenne Thompson Webinars September 5 & 10, 2012

Your Presenter Laenne Thompson

Logistics This webinar will be recorded. Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Logistics This webinar will be recorded. This webinar is being recorded. If you come late, leave early or just want to refresh your memory. You can access the recording afterwards at www.cypq.org.

Logistics This webinar will be recorded Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Logistics This webinar will be recorded Dial *7 to mute yourself. Please stay muted when you are not speaking. To reduce background noise, during most of the webinar, your line will be muted.

Logistics This webinar will be recorded Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Logistics This webinar will be recorded Dial *7 to mute yourself. Please stay muted when you are not speaking. Dial *6 to unmute yourself.

Logistics This webinar will be recorded Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Logistics This webinar will be recorded Dial *7 to mute yourself. Please stay muted when you are not speaking. Dial *6 to unmute yourself. Enter a message in the chat box if you need assistance. The chat only goes to the meeting’s chairperson.

Logistics This webinar will be recorded. Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Logistics This webinar will be recorded. Dial *7 to mute yourself. Please stay muted when you are not speaking. Dial *6 to unmute yourself. Enter a message in the chat box if you need assistance. The chat only goes to the meeting’s chairperson (in this case, Ravi). Click the “Raise Hand” button if you wish to speak. Please raise your hand if you have a question or would like to make a comment

Agenda Introductions What is the NYS-OCFS PQA? ASSESS PLAN IMPROVE PQA Assessment Next Steps Agenda Introductions What is the NYS-OCFS PQA? What is the NYS-OCFS Quality Accountability and Improvement System (QAIS)? PQA Assessment Next Steps Questions, Comments & Clarifications

Introductions Please share with us: Your name PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Introductions Please share with us: Your name Your organization (State, County, Agency or other) Type(s) of programs you work with One thing that you are hoping this assessment process can help you with

K-W-L: What do you know about Program Quality or Assessment? Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps K-W-L: What do you know about Program Quality or Assessment? Know Want to know Learned

What is the NYS-OCFS PQA?

Quality Construct: What is the NYS-OCFS PQA? Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Quality Construct: What is the NYS-OCFS PQA? A revised, shorter version of the standard Youth PQA, which is a validated instrument designed to assess the quality of youth programs and identify staff training needs. A set of items that measures youth access to key developmental experiences. A tool which produces scores that can be used for comparison and assessment of progress over time.

NYS-OCFS PQA Quick Facts Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps NYS-OCFS PQA Quick Facts NYS-OCFS worked with the Weikart Center to tailor an instrument based on the validated Youth Program Quality Assessment (PQA) and the best practices it measures. The NYS-OCFS PQA reflects the 8 features identified by the National Research Council that positive developmental settings reflect. From January to May 2012, the NYS-OCFS PQA was piloted by your peers and then revised based on feedback. (report available)

Researchers Agree on What It Takes to Support Development Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Researchers Agree on What It Takes to Support Development The National Research Council & Institute for Medicine list the following key features of positive youth development settings: Physical and psychological safety Appropriate structure Supportive relationships Opportunities to belong Positive social norms Support for efficacy and mattering Opportunities for skill-building Integration of family, school and community efforts Eccles, J. S., & Gootman, J. A. (2002). Community programs to promote youth development. Washington, DC: National Academy Press. A 2002 report by the National Research Council reviewed the combined results of many research studies and synthesized those findings into this list of key features of programming that work best to support youth development. I’ll give you a minute to read the list. {wait 1 minute} There aren’t many surprises in here, but these are the things that research has shown will positively impact youth development….

The Pyramid of Program Quality Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps The Pyramid of Program Quality … and this research is in support of what is measured in the Youth PQA. The Youth PQA is designed to measure key developmental experiences within an afterschool setting. It is based around Maslow’s hierarchy of needs, where the bottom of the pyramid needs to be in place before the top can be reached. At the bottom of the Pyramid of Program Quality is safe environment. This includes both physical and emotional safety. Above that we have a supportive environment, which is mostly provided by the staff – do they plan sessions so that they’re at the right pace for youth? Do youth have an opportunity to build skills, and are they encouraged to do so despite mistakes? Are adults providing support for learning?

Discussion preview – keep these in mind as we look at the 8 features through the lens of the PQA How do you think the PQA will be useful in your setting? What is your experience using the full Youth PQA or NYS-OCFS PQA? Which of the 8 features have you had success improving in the last 6 months? And how did you do it? Which of the 8 features is most challenging to address?

Physical and Psychological Safety emotional climate emergency procedures health and safety supervised access

Appropriate Structure staff explain expectations staff appropriately share control staff clearly explain activities appropriate time for activities behavior guidelines developed together rules and expectations available

Supportive Relationships staff warm, friendly, and respectful staff actively involved with youth opportunities for youth to talk staff orientation on youth development staff training provided regularly

Opportunities to Belong opportunities to get to know each other youth work cooperatively together mutual respect cultural diversity

Positive Social Norms inclusive relationships strong identity with program publicly acknowledge achievement explicit conflict resolution policy

Support for Efficacy and Mattering plan choose reflect opportunities to lead a group tangible products or performances share administrative responsibilities

Opportunities for Skill Building engage with materials or ideas balance concrete and abstract encouraged to try new skills mistakes allowed

Integration of Families, Communities, Schools communication with families, schools & organizations links to community strength-based family welcomed

Discussion How do you think the PQA will be useful in your setting? What is your experience using the full Youth PQA or NYS-OCFS PQA? Which of the 8 features have you had success improving in the last 6 months? And how did you do it? Which of the 8 features is most challenging to address?

What is the NYS-OCFS Quality Accountability and Improvement System?

Theory of Change: Multiple Levels of Settings Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Theory of Change: Multiple Levels of Settings Point of Service Setting Organizational Policy Continuous Improvement Practices for Site Teams Quality Instruction & Proximal Child Outcomes Low Stakes Accountability and Supports / T&TA Policy – low stakes vs High stakes Org setting – Research based and validated process POS – assessed by NY-OCFS PQA Form

QAIS Continuous Improvement Cycle Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps QAIS Continuous Improvement Cycle The NY-OCFS PQA is just the first step in a larger cycle of improvement, which will be the Quality Accountability and Improvement System. Today we’ll focus on the ASSESS & PLAN phases as we help you get comfortable with your role in this cycle. So after sites conduct their self-assessments, the next step is to develop a plan for improvement based on the data, and then to carry out that plan to improve the quality of the program for the youth participants.

Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Assess The PQA allows you to observe what actually goes on in your youth programs. How often do you have the time or energy to simply observe a program led by another staff? How often do you rely on what you think happens in your programs?

Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Plan Looking at the results of your assessment gives you a chance to make concrete plans. What are your program strengths? What do you want to improve?

Improve During the improve phase you will implement your plans. Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Improve During the improve phase you will implement your plans. What kind of professional development do you need? What additional resources? What changes to the curriculum?

Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps What does this all mean? Teams (or individuals) observe programming and create PQA scores based on those observations Look at the data collected and identify areas for improvement. Then create an improvement plan. Implement the improvement plan through professional development, curriculum changes, etc. THEN REPEAT

NYS-OCFS QAIS Goals Positively affect child development and learning Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps NYS-OCFS QAIS Goals Positively affect child development and learning Instill a culture and practice of continuous quality improvement that is driven by data -Positively affect child development and learning in OCFS-funded settings by building a QAIS for all 3,500 sites. -Retool current OCFS-at-scale infrastructure by updating professional skill sets for OFCS state and county level staff, retooling existing OCFS online technology, and incentivize supplemental investment at the site level -Instill a culture and practice of continuous quality improvement that is driven by data. -Test the newly developed NY-OCFS PQA for a larger-scale implementation.

PQA assessment

PQA Assessment Overview Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps PQA Assessment Overview Training (and Team Selection) Data Collection Preparation Observation and Note-Taking (Team Based) Scoring Meeting Entering Scores

1. Training and Team Selection Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps 1. Training and Team Selection Assessment Leader County Youth Bureau Staff, Agency Manager, Program Staff, etc. Assessment Team Members at least two program staff parents volunteers board member youth DO Include program staff DON’T Include only administrative staff “ One person is an evaluation. Two people is a negotiation. Three people is a conversation. Ten people is too many!”

NY-OCFS PQA Assessment Terms Youth Bureau Assessment  (one or more county members observing) Youth Bureau-Agency Assessment  (county + agency members observing) Agency Observation/Assessment (agency members, either administrative or frontline staff observing) Agency Self-report/Assessment- (agency members completing Administrator Self-Interview questions, through interview or use of documentation) Youth Bureau-Youth Assessment  (one or more county members + youth observing) Youth Bureau-Agency-Youth Assessment  (county + agency + youth members observing) Agency-Youth Observation/Assessment  (administrative/frontline agency members + youth observing) External Assessment (one trained reliable assessor observing and completing one PQA alone)

1. Training and Team Selection Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps 1. Training and Team Selection Assessment Leader PQA Basics (live or online) Assessment Team Members PQA Basics online PQA Introduction online Training by the Assessment Leader Visit www.cypq.org/takeitback After the site team members have been selected, the Self Assessment Leader can prepare the other team members to participate in the program self assessment. The Youth PQA Online Free Intro and the Youth PQA Crash Course Power Point presentation are available to help the Self Assessment Leader to prepare the team.

2. Data Collection Preparation Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps 2. Data Collection Preparation Have team members take turns observing programs in action. Try to schedule a total of at least 1 hour of observation, divided among team members in 15-20 minute chunks. Ideally, you will observe the elements of an entire program offering. Schedule observations of Enrichment activities. Do what works for your program. The more you can observe, the richer your conversations will be! Have team members take turns observing programs in action Rearrange schedules to provide coverage to free staff up to observe Project Director or Site Coordinator could provide coverage Schedule a total of at least 1 hour of observation divided among team members. For example, if you have 3 team members, have them each observe for 20 minutes. If you have only 2 team members, be sure each observes at least 30 minutes each. Be sure that you vary observing the beginning, middle, and end of sessions. Schedule observations of: Academic Enrichment Activities: science club, journalism , tutoring, etc. Social/Cultural Enrichment Activities: Arts & Crafts, Heath/wellness, etc. Avoid: homework help, open gym, unstructured computer lab time, etc. Notify frontline staff of scheduled observation times, and have them notify the youth Plan time as soon as possible following the observations for discussion and scoring Notify frontline staff of observation. Plan time as soon as possible following the observations for discussion and scoring.

3. Observation and Note-Taking Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps 3. Observation and Note-Taking Bring the back page of the NYS-OCFS PQA Take notes throughout the offering on factual information Bring the back page of the Youth PQA as a reference for the things you should be looking for. Do not take notes directly in a Youth PQA form. Take notes throughout the offering on factual information (include quotes, actions, etc.). Use the skills you practiced at the Youth PQA Basics training.

4. Team-based Scoring Meeting (or individual scoring) Intro PQA ASSESS PLAN IMPROVE Program Self Assessment Next Steps 4. Team-based Scoring Meeting (or individual scoring) Discuss each item and row as a team. Have each team member present evidence from their observation notes. Together, select the best score for each item. Note contentious items as potential areas to focus on in your Improvement Plan. Plan at least 3 hours for this process (remember, the conversation is the most important part!) DO Involve program staff Plan more time than you think you’ll need!

Program Self Assessment Intro PQA ASSESS PLAN IMPROVE Program Self Assessment Next Steps 5. Entering Scores Excel Scores Reporter available at cypq.org/nyocfs

Next Steps

NY-OCFS QAIS Dates (suggested) Intro PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps NY-OCFS QAIS Dates (suggested) DATE ACTIVITY WHO Sept 5 11am Sept 10 2pm Kick-Off Webinars NY State Staff (NYSS) Youth Bureau County Staff (YBCS) Agency Managers (AM) Sept 11 Sept 12 Sept. 21 LIVE Basics Training Batavia Rensselaer & Syracuse Hempstead (Long Island) NYSS YBCS AM Program Staff Sept -Oct Assessments conducted – use online courses & local supports NYSS, YBCS, & AM Sept. 28 11am Planning with Data Webinar NYSS & YBCS Oct - Dec Enter assessment data Use the data to design improvement strategies Implement improvement strategies NYSS, YBCS, AM & Program Staff

Lingering Questions?

PQA ASSESS PLAN IMPROVE PQA Assessment Next Steps Thank you! When the webinar closes, you will be prompted to leave feedback. Please take a minute to tell us about your experience with this webinar. Continuous improvement is our policy! Contact us with any additional questions: Laenne Thompson laenne@cypq.org 734.714.2516