1. Laenne Thompson Webinars September 5 & 10, 2012
Join us on the phone for the audio portion:
ReadyTalk Dial-In Number(s): Access Code:
2012 NYS-OCFS Quality Accountability and Improvement System
kickoff
Laenne Thompson
Webinars September 5 & 10, 2012

2. Your Presenter
Laenne Thompson

3. Logistics This webinar will be recorded.
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Logistics
This webinar will be recorded.
This webinar is being recorded. If you come late, leave early or just want to refresh your memory. You can access the recording afterwards at

4. Logistics This webinar will be recorded
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Logistics
This webinar will be recorded
Dial *7 to mute yourself. Please stay muted when you are not speaking.
To reduce background noise, during most of the webinar, your line will be muted.

5. Logistics This webinar will be recorded
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Logistics
This webinar will be recorded
Dial *7 to mute yourself. Please stay muted when you are not speaking.
Dial *6 to unmute yourself.

6. Logistics This webinar will be recorded
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Logistics
This webinar will be recorded
Dial *7 to mute yourself. Please stay muted when you are not speaking.
Dial *6 to unmute yourself.
Enter a message in the chat box if you need assistance. The chat only goes to the meeting’s chairperson.

7. Logistics This webinar will be recorded.
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Logistics
This webinar will be recorded.
Dial *7 to mute yourself. Please stay muted when you are not speaking.
Dial *6 to unmute yourself.
Enter a message in the chat box if you need assistance. The chat only goes to the meeting’s chairperson (in this case, Ravi).
Click the “Raise Hand” button if you wish to speak.
Please raise your hand if you have a question or would like to make a comment

8. Agenda Introductions What is the NYS-OCFS PQA?
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Agenda
Introductions
What is the NYS-OCFS PQA?
What is the NYS-OCFS Quality Accountability and Improvement System (QAIS)?
PQA Assessment
Next Steps
Questions, Comments & Clarifications

9. Introductions Please share with us: Your name
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Introductions
Please share with us:
Your name
Your organization (State, County, Agency or other)
Type(s) of programs you work with
One thing that you are hoping this assessment process can help you with

10. K-W-L: What do you know about Program Quality or Assessment?
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
K-W-L: What do you know about Program Quality or Assessment?
Know
Want to know
Learned

11. What is the NYS-OCFS PQA?

12. Quality Construct: What is the NYS-OCFS PQA?
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Quality Construct: What is the NYS-OCFS PQA?
A revised, shorter version of the standard Youth PQA, which is a validated instrument designed to assess the quality of youth programs and identify staff training needs.
A set of items that measures youth access to key developmental experiences.
A tool which produces scores that can be used for comparison and assessment of progress over time.

13. NYS-OCFS PQA Quick Facts
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
NYS-OCFS PQA Quick Facts
NYS-OCFS worked with the Weikart Center to tailor an instrument based on the validated Youth Program Quality Assessment (PQA) and the best practices it measures.
The NYS-OCFS PQA reflects the 8 features identified by the National Research Council that positive developmental settings reflect.
From January to May 2012, the NYS-OCFS PQA was piloted by your peers and then revised based on feedback. (report available)

14. Researchers Agree on What It Takes to Support Development
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Researchers Agree on What It Takes to Support Development
The National Research Council & Institute for Medicine list the following key features of positive youth development settings:
Physical and psychological safety
Appropriate structure
Supportive relationships
Opportunities to belong
Positive social norms
Support for efficacy and mattering
Opportunities for skill-building
Integration of family, school and community efforts
Eccles, J. S., & Gootman, J. A. (2002). Community programs to promote youth development. Washington, DC: National Academy Press.
A 2002 report by the National Research Council reviewed the combined results of many research studies and synthesized those findings into this list of key features of programming that work best to support youth development. I’ll give you a minute to read the list.
{wait 1 minute}
There aren’t many surprises in here, but these are the things that research has shown will positively impact youth development….

15. The Pyramid of Program Quality
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
The Pyramid of Program Quality
… and this research is in support of what is measured in the Youth PQA.
The Youth PQA is designed to measure key developmental experiences within an afterschool setting.
It is based around Maslow’s hierarchy of needs, where the bottom of the pyramid needs to be in place before the top can be reached.
At the bottom of the Pyramid of Program Quality is safe environment. This includes both physical and emotional safety.
Above that we have a supportive environment, which is mostly provided by the staff – do they plan sessions so that they’re at the right pace for youth? Do youth have an opportunity to build skills, and are they encouraged to do so despite mistakes? Are adults providing support for learning?

17. Discussion preview – keep these in mind as we look at the 8 features through the lens of the PQA
How do you think the PQA will be useful in your setting?
What is your experience using the full Youth PQA or NYS-OCFS PQA?
Which of the 8 features have you had success improving in the last 6 months? And how did you do it?
Which of the 8 features is most challenging to address?

18. Physical and Psychological Safety
emotional climate
emergency procedures
health and safety
supervised access

19. Appropriate Structure
staff explain expectations
staff appropriately share control
staff clearly explain activities
appropriate time for activities
behavior guidelines developed together
rules and expectations available

20. Supportive Relationships
staff warm, friendly, and respectful
staff actively involved with youth
opportunities for youth to talk
staff orientation on youth development
staff training provided regularly

21. Opportunities to Belong
opportunities to get to know each other
youth work cooperatively together
mutual respect
cultural diversity

22. Positive Social Norms inclusive relationships
strong identity with program
publicly acknowledge achievement
explicit conflict resolution policy

23. Support for Efficacy and Mattering
plan
choose
reflect
opportunities to lead a group
tangible products or performances
share administrative responsibilities

24. Opportunities for Skill Building
engage with materials or ideas
balance concrete and abstract
encouraged to try new skills
mistakes allowed

25. Integration of Families, Communities, Schools
communication with families, schools & organizations
links to community
strength-based
family welcomed

26. Discussion How do you think the PQA will be useful in your setting?
What is your experience using the full Youth PQA or NYS-OCFS PQA?
Which of the 8 features have you had success improving in the last 6 months? And how did you do it?
Which of the 8 features is most challenging to address?

27. What is the NYS-OCFS Quality Accountability and Improvement System?

28. Theory of Change: Multiple Levels of Settings
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Theory of Change: Multiple Levels of Settings
Point of Service
Setting
Organizational
Policy
Continuous Improvement Practices for Site Teams
Quality Instruction
& Proximal Child Outcomes
Low Stakes Accountability
and Supports / T&TA
Policy – low stakes vs High stakes
Org setting – Research based and validated process
POS – assessed by NY-OCFS PQA Form

29. QAIS Continuous Improvement Cycle
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
QAIS Continuous Improvement Cycle
The NY-OCFS PQA is just the first step in a larger cycle of improvement, which will be the Quality Accountability and Improvement System. Today we’ll focus on the ASSESS & PLAN phases as we help you get comfortable with your role in this cycle. So after sites conduct their self-assessments, the next step is to develop a plan for improvement based on the data, and then to carry out that plan to improve the quality of the program for the youth participants.

30. Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Assess
The PQA allows you to observe what actually goes on in your youth programs.
How often do you have the time or energy to simply observe a program led by another staff?
How often do you rely on what you think happens in your programs?

31. Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Plan
Looking at the results of your assessment gives you a chance to make concrete plans.
What are your program strengths?
What do you want to improve?

32. Improve During the improve phase you will implement your plans.
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Improve
During the improve phase you will implement your plans.
What kind of professional development do you need?
What additional resources?
What changes to the curriculum?

33. Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
What does this all mean?
Teams (or individuals) observe programming and create PQA scores based on those observations
Look at the data collected and identify areas for improvement. Then create an improvement plan.
Implement the improvement plan through professional development, curriculum changes, etc.
THEN REPEAT

34. NYS-OCFS QAIS Goals Positively affect child development and learning
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
NYS-OCFS QAIS Goals
Positively affect child development and learning
Instill a culture and practice of continuous quality improvement that is driven by data
-Positively affect child development and learning in OCFS-funded settings by building a QAIS for all 3,500 sites.
-Retool current OCFS-at-scale infrastructure by updating professional skill sets for OFCS state and county level staff, retooling existing OCFS online technology, and incentivize supplemental investment at the site level
-Instill a culture and practice of continuous quality improvement that is driven by data.
-Test the newly developed NY-OCFS PQA for a larger-scale implementation.

35. PQA assessment

36. PQA Assessment Overview
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
PQA Assessment Overview
Training (and Team Selection)
Data Collection Preparation
Observation and Note-Taking
(Team Based) Scoring Meeting
Entering Scores

37. 1. Training and Team Selection
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
1. Training and Team Selection
Assessment Leader
County Youth Bureau Staff, Agency Manager, Program Staff, etc.
Assessment Team Members
at least two program staff
parents
volunteers
board member
youth DO
Include program staff
DON’T
Include only administrative staff
“ One person is an evaluation. Two people is a negotiation. Three people is a conversation. Ten people is too many!”

38. NY-OCFS PQA Assessment Terms
Youth Bureau Assessment  (one or more county members observing)
Youth Bureau-Agency Assessment  (county + agency members observing)
Agency Observation/Assessment (agency members, either administrative or frontline staff observing)
Agency Self-report/Assessment- (agency members completing Administrator Self-Interview questions, through interview or use of documentation)
Youth Bureau-Youth Assessment  (one or more county members + youth observing)
Youth Bureau-Agency-Youth Assessment  (county + agency + youth members observing)
Agency-Youth Observation/Assessment  (administrative/frontline agency members + youth observing)
External Assessment (one trained reliable assessor observing and completing one PQA alone)

39. 1. Training and Team Selection
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
1. Training and Team Selection
Assessment Leader
PQA Basics (live or online)
Assessment Team Members
PQA Basics online
PQA Introduction online
Training by the Assessment Leader
Visit
After the site team members have been selected, the Self Assessment Leader can prepare the other team members to participate in the program self assessment. The Youth PQA Online Free Intro and the Youth PQA Crash Course Power Point presentation are available to help the Self Assessment Leader to prepare the team.

40. 2. Data Collection Preparation
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
2. Data Collection Preparation
Have team members take turns observing programs in action.
Try to schedule a total of at least 1 hour of observation, divided among team members in minute chunks. Ideally, you will observe the elements of an entire program offering.
Schedule observations of Enrichment
activities.
Do what works for your program.
The more you can observe, the richer your conversations will be!
Have team members take turns observing programs in action
Rearrange schedules to provide coverage to free staff up to observe
Project Director or Site Coordinator could provide coverage
Schedule a total of at least 1 hour of observation divided among team members.
For example, if you have 3 team members, have them each observe for 20 minutes. If you have only 2 team members, be sure each observes at least 30 minutes each.
Be sure that you vary observing the beginning, middle, and end of sessions.
Schedule observations of:
Academic Enrichment Activities: science club, journalism , tutoring, etc.
Social/Cultural Enrichment Activities: Arts & Crafts, Heath/wellness, etc.
Avoid: homework help, open gym, unstructured computer lab time, etc.
Notify frontline staff of scheduled observation times, and have them notify the youth
Plan time as soon as possible following the observations for discussion and scoring
Notify frontline staff of observation.
Plan time as soon as possible following the observations for discussion and scoring.

41. 3. Observation and Note-Taking
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
3. Observation and Note-Taking
Bring the back page of the NYS-OCFS PQA
Take notes throughout the offering on factual information
Bring the back page of the Youth PQA as a reference for the things you should be looking for. Do not take notes directly in a Youth PQA form.
Take notes throughout the offering on factual information (include quotes, actions, etc.). Use the skills you practiced at the Youth PQA Basics training.

42. 4. Team-based Scoring Meeting (or individual scoring)
Intro
PQA
ASSESS
PLAN
IMPROVE
Program Self Assessment
Next Steps
4. Team-based Scoring Meeting (or individual scoring)
Discuss each item and row as a team.
Have each team member present evidence from their observation notes.
Together, select the best score for each item.
Note contentious items as potential areas to focus on in your Improvement Plan.
Plan at least 3 hours for this process (remember, the conversation is the most important part!) DO
Involve program staff
Plan more time than you think you’ll need!

43. Program Self Assessment
Intro
PQA
ASSESS
PLAN
IMPROVE
Program Self Assessment
Next Steps
5. Entering Scores
Excel Scores Reporter available at cypq.org/nyocfs

44. Next Steps

45. NY-OCFS QAIS Dates (suggested)
Intro
PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
NY-OCFS QAIS Dates (suggested)
DATE
ACTIVITY
WHO
Sept 5 11am
Sept 10 2pm
Kick-Off Webinars
NY State Staff (NYSS)
Youth Bureau County Staff (YBCS)
Agency Managers (AM)
Sept 11
Sept 12
Sept. 21
LIVE Basics Training
Batavia
Rensselaer & Syracuse
Hempstead (Long Island)
NYSS
YBCS AM
Program Staff
Sept -Oct
Assessments conducted – use online courses & local supports
NYSS, YBCS, & AM
Sept am
Planning with Data Webinar
NYSS & YBCS
Oct - Dec
Enter assessment data
Use the data to design improvement strategies
Implement improvement strategies
NYSS, YBCS, AM & Program Staff

46. Lingering Questions?

47. PQA
ASSESS
PLAN
IMPROVE
PQA Assessment
Next Steps
Thank you!
When the webinar closes, you will be prompted to leave feedback. Please take a minute to tell us about your experience with this webinar. Continuous improvement is our policy!
Contact us with any additional questions:
Laenne Thompson