1. 521: QSR Training:
“Refresher Session”

2. Learning Objectives
Identify the core components of PA’s Child Welfare Practice Model
Describe the roles and responsibilities of being a QSR reviewer.
Explain the core components of the QSR Protocol, scoring, rating rationale, and providing feedback.
Introductions – be sure to confirm participants who are in attendance, as individuals will only be certified as a QSR Reviewer if they sign in at the beginning and end of the session.
Session participants need to have one of the following in order to participate:
Their copy of the QSR Protocol.
NOTE: Link to CWRC website where they can find the PA Protocol Version 4.0.
Introduction
Distribute or refer to Handout #1 (PowerPoint Presentation: QSR Training: “Refresher Session”) noting that this is a hardcopy of the PowerPoint presentation.
Distribute or refer to Handout #2 (Learning Objectives and Agenda) and display PowerPoint Slides #2 (Learning Objectives). Discuss the Learning Objectives. 2

3. Welcome and Introductions
Agenda
Welcome and Introductions
Core Components of Pennsylvania’s Child Welfare Practice Model
Roles and Responsibilities of a QSR Reviewer
Interviewing Refresher
Indicators, Scoring, and Rationale
Feedback
Questions, Next Steps, and Closing
Display PowerPoint Slides #3 (Agenda). Discuss the Agenda. Ask if there are any additional agenda requests. 3

4. At its most basic level, a child welfare practice model is a conceptual map and organizational ideology of how child welfare professionals, families, and stakeholders should unite in creating a physical and emotional environment that focuses on the safety, permanency, and well-being of children, youth and their families. The practice model should make an explicit link connecting the child welfare system’s policy, practice, training, supervision, and quality improvement with its mission, vision, agency values, and strategic plan. It is the (system’s) guide to the daily interactions among employees, children, youth, families, stakeholders, and community partners working together to achieve defined outcomes. (APHSA, 2009).
The Pennsylvania Child Welfare Practice Model HO #3 (PA's Child Welfare Practice Model) was developed by child welfare professionals and their partners, including youth and families, court personnel, private providers, OCYF, and technical assistance partners. The practice model defines standards of practice and identifies how implemented practices can support the strategies to improve outcomes. PowerPoint Slide #4 depicts the key components of PA’s Child Welfare Practice Model – the Values, Principles and Skills that are necessary to achieve positive outcomes for children, youth and families. These concepts are further defined in HO #3 (PA's Child Welfare Practice Model). It should be noted that these concepts are evaluated through the QSR Process and the 23 indicators outlined in the QSR Protocol.
QSR Reviewers – Will evaluate practice by gathering information about a child, youth and family team. QSR Reviewers will take the information gathered through the interview process and evaluate the implementation of the Practice Model by scoring what was learned about the case in conjunction with the QSR Protocol, the tool that looks at the child/youth/family status and practice performance through the evaluation of 23 indicators.
Share HO #3 (PA's Child Welfare Practice Model) and discuss PA’s Child Welfare Practice Model
Ensure that the participants see the alignment of the PA Child Welfare Practice Model with the 23 Indicators of the QSR Protocol.

5. What is Your Role? Case file review
Each pair of reviewers will conduct case reviews:
Case file review
Interviews
Roll-Up Sheet – data entry of child/youth/family information, case information, scores, rationale, and recommendations
Provide feedback to the caseworker (and supervisor)
Present scoring rationale and recommendations at the Case-Specific Team Debriefing
Reviewers may also be invited to attend the Exit Conference.
Review the QSR Reviewers role, by discussing key components of the case review process outlined in PowerPoint Slide #5 (What is Your Role?)
The Pennsylvania Child Welfare Resource Center

6. Responsibilities of a QSR Reviewer
Rate results/outcomes, not intentions (except
Practice Performance Indicators 1a and 8).
Remember that this is a review of the system, not
individual caseworker.
Rate the current child/youth status, not the past or future (unless required by the parameters of the indicator).
Follow the principles of the Practice Model and give feedback in a constructive way, including improvement needs, as well as strengths.
Review the responsibilities of a QSR Reviewer outlined in PowerPoint Slide #6 (Responsibilities of a QSR Reviewer):
Rate results/outcomes, not intentions (unless rating indicators focused on effort);
This is a review of the system, not of the Caseworker or Supervisor;
Rate the current child/youth status, not the past or future (unless rating indicators require other timeframes to be considered);
Follow the principles of the Practice Model and give feedback in a constructive way, including improvement needs as well as strengths;
Other key responsibilities:
It is important that QSR Reviewers create a QSR experience that ensure a SAFE learning environment.
It is important that QSR Reviewers focus on collecting information to aid in the evaluation of the case and Practice Model implementation and that QSR Reviewers NOT do case work.
It is important that QSR Reviewers notify Site Leads of any safety concerns as soon as possible.
Refer to Handout #4: QSR Reviewer Role and Demeanor to support this discussion. Feel free to have QSR Reviewers share their successes of how they have been able to fulfill these roles and responsibilities successfully. 6 6

7. Interview Refresher: Prior to the Interview
Tune into self- Get in touch with feelings/concerns and beliefs/values.
Learn something about the interests of the child/youth to better engage him/her.
Review the interview schedule and identify potential questions.
Identify the lead interviewer and note taker for each interview.
Remember that you are a reviewer who is responsible for evaluating the system, not a caseworker responsible for case management.
Review the content of PowerPoint Slide #7 (Interview Refresher: Prior to the Interview). Have discussion about interview preparations.
The Pennsylvania Child Welfare Resource Center

8. Interview Refresher: During Each Interview
Clarify purpose and roles
Recognize family strengths
Be patient with family stories
Listen more than talk
Use QSR questions and indicators to keep you on track
Always ask about the child’s safety
Gather information, do not identify solutions
Avoid leading questions
Review the content of PowerPoint Slide #8 (Interview Refresher: During Each Interview). Have discussion about tips and tricks for successful interviews.
The Pennsylvania Child Welfare Resource Center

9. Interview Refresher: After the Interview
Do not share information between interviewees
Resist reaching conclusions/scoring before all interviews
Note areas needing confirmation or follow-up
Contact your Site Lead and/or next interviewee if you are running behind
Contact your Site Lead to arrange any additional interviews you.
Review the content of PowerPoint Slide #9 (Interview Refresher: After the Interview). Have discussion about post-interview activities.
The Pennsylvania Child Welfare Resource Center

10. Child/Youth and Family Status Indicators
Safety: Exposure to Threats of Harm
Safety: Risk to Self/Others
Stability
Living Arrangement
Permanency
Physical Health
Emotional Well-Being
Early Learning and Development
Academic Status
Pathway to Independence
Parent or Caregiver Functioning
Discuss PowerPoint Slide #10 (Child/Youth and Family Status Indicators). Child, youth and family status indicators measure the extent to which certain desired conditions are present in the life the focus child/youth and the child/youth’s parents and/or caregivers—as seen over the past 30 days. Status indicators measure constructs related to safety, permanence and well-being. Changes in status over time may be considered the near-term outcomes at a given point in the life of a case. Make connections to Pennsylvania’s Child Welfare Practice Model.
The Pennsylvania Child Welfare Resource Center

11. Practice Performance Indicators
Engagement Efforts
Role and Voice
Teaming
Cultural Awareness and Responsiveness
Assessment and Understanding
Long-Term View
Child/Youth and Family Planning Process
Planning for Transitions and Life Adjustments
Efforts to Timely Permanence
Intervention Adequacy and Resource Availability
Maintaining Family Relationships
Tracking and Adjusting
Discuss PowerPoint Slide #11. Practice indicators measure the extent to which core practice functions are applied successfully by practitioners and others who serve as members of the child/youth and family team. The core practice functions measured are taken from the child/youth and family team and provide useful case-based tests of performance achievement. The number of core practice functions and level of detail used in their measurement may evolve over time as advances are made in the state-of-the-art practice. Make connections to PA’s Child Welfare Practice Model.
The Pennsylvania Child Welfare Resource Center

12. What Time Period is Considered in Scoring?
Child/Youth and Family Status Indicators look at the past 30 days
Exceptions are: Stability and Academic Status
Practice Performance Status Indicators look at the past 90 days
Exceptions are: Planning for Transitions and Life Adjustments and Efforts to Timely Permanence
NOTE: Recent performance improvements have some influence on scores
Review PowerPoint Slide #12 (What Time Period is Considered in Scoring?)
Child/Youth and Family Status Indicators looks at the past 30 days.
Exceptions: Stability and Academic Status
Stability looks back over past 12 months and prospectively for the next 6 months.
Academic status looks back over past 30 days unless review is occurring when school is not in session (summer), then reviewers should rate the indicator based on final 30 days of the most recent school year.
Practice Performance Status Indicators looks at the past 90 days
Exceptions: Planning for Transitions and Life Adjustments and Efforts to Timely Permanence.
Planning for Transitions and Life Adjustments looks retrospectively over past 90 days and prospectively over the next 90 days.
Efforts to Timely Permanence – while “efforts” are measured over past 90 days, “timeliness” is rated for out-of-home cases ONLY and includes specific timeframes that must be considered.

13. How are Cases Scored? Each indicator is scored on a scale of 1 – 6
6 – Optimal
5 – Substantial Acceptable Range
4 – Fair
3 – Marginal
2 – Poor Unacceptable Range
1 – Adverse
Maintenance Zone: 6-5
Refinement Zone: 4-3
Improvement Zone: 2-1
Review PowerPoint Slide #13 (How are Cases Scored?) for a quick refresher on the scale.
The Pennsylvania Child Welfare Resource Center

14. The Central Scoring Rules
Focus on the focus measure and core concepts in each indicator.
Read each indicator’s instructions:
Stay within the time-based observation windows associated with each indicator.
Rate indicators based on events that have occurred or conditions that were present within the time-based observation window.
Review PowerPoint Slide #14 (Central Scoring Rules)
Focus on the Focus Measure and Core Concepts in each indicator.
Some indicators score results. Others score effort or intent, such as Practice Performance Indicators 1a and 8.
Not Applicable rating criteria are identified as appropriate.
Score each sub-category within each indicator, as appropriate.
Read each indicator’s instructions:
Stay within the time-based observation windows associated with each indicator.
Rate indicators based on events that have occurred or conditions that were present within the time-based observation window 14 14 14

15. Distinction Between 3 and 4
The Groundhog Rule
If this case were frozen in time as it is
today, would performance or status be
considered acceptable or unacceptable?
Review PowerPoint Slide #15 (Distinction between 3 and 4):
The Groundhog Rule: If this case were frozen in time as it is today, would performance or status be considered acceptable or unacceptable? 15 15

16. Case Review Information Focus Child/Youth’s Information
QSR Roll-Up Sheet
Case Review Information
Focus Child/Youth’s Information
Case Information
Child/Youth and Family Status Domain
Practice Performance Status Domain
Recommendations
New in Round 6, there will be use of a new web-based Roll-Up Sheet for data entry of all QSR Roll-Up Sheet data points. Refer to Appendix #1: PA QSR Protocol Version 4.0
The following are some benefits of the web-based Roll-Up Sheet:
Reviewers will write their rationale as they score, so the reasoning will be fresh in their mind
New format streamlines process – no longer a need for a separate written case review summary, as written case review summary will be automatically generated
Reviewers will have fully articulated their rationale prior to the caseworker/supervisor feedback session and case specific team debrief, so they will be better prepared for their feedback session and the debrief session
Web-based format allows for series of internal validation checks to ensure accurate data collection
Higher level analysis, to include improved qualitative content analysis, can be performed faster and more often because of the way the data is stored and readily accessible.
QSR reviewers will enter all data elements on a case-specific Roll-Up Sheet that will be activated by connecting to a personalized web link assigned to each review team. The assigned web link will be accessed via the flash drives that are disseminated by site leads.
One QSR Reviewer (usually the person who brought the laptop to the onsite review) will conduct the data entry via the web link while the other QSR reviewer will fill out a hardcopy of the QSR Roll-Up Sheet. This is being done as a back-up resources in case there are any connectivity issues to the web link. (NOTE: The hardcopy of the Roll-Up sheet can be scanned and ed to HZA if there are connectivity issues, so neither QSR reviewers nor site leads will have to conduct data entry due to technical issues.)
Refer to PowerPoint Slide #16. QSR Roll-Up Sheet Components have been streamlined and include the following:
Case Review Information
Focus Child/Youth’s Information
Case Information
Child/Youth and Family Status Domain
Practice Performance Status Domain
Recommendations
The Pennsylvania Child Welfare Resource Center

17. QSR Rating Rationale
Purpose: To provide a written narrative that explains the reasoning and justification for why the score was chosen by the reviewers.
Narrative should:
Tell a story of information gathered, including examples
Be succinct, yet descriptive
Narrative should not include:
Names of individuals interviewed
Reviewers should use case participants roles (i.e. mother, father, foster mother, attorney, etc.)
A reiteration of the QSR indicator score description
As the QSR review team enters their scores for the indicators and sub-indicators, QSR reviewers will discuss their rating rationale and provide written justification for each indicator and sub-indicators rating rationale. This written narrative will now take the place of what QSR reviewers previously wrote in the Written Case Review Summary.
Refer to PowerPoint Slide #17. The purpose QSR rating rationale is to provide narrative that explains the reasoning and justification for why the score was chosen by the reviewers. QSR reviewers will provide information about the favorable and unfavorable information that was learned as part of the case review and document that rationale.
Narrative should:
Tell a story of information gathered, including examples
Be succinct, yet descriptive
Narrative should not include:
Names of individuals interviewed
Reviewers should use case participants roles (i.e. mother, father, foster mother, attorney, etc.)
A reiteration of the QSR indicator score description
An example of a web-based Roll-Up Sheet can be shared with participants at this time.
Web Link: 
QSR reviewers will have an opportunity to view a short video explaining the navigation of the web-based Roll-Up Sheet. Site leads will also be available to offer support  
The Pennsylvania Child Welfare Resource Center

18. Case-Specific Team Debriefing: Case Presentation Outline
Brief synopsis of the child/youth/family story (2 minutes)
Child/Youth and Parent/Caregiver Status Indicator Rationale (4 minutes)
Practice Performance Indicator Rationale (4 minutes)
Next Steps (2 minutes)
Group Questions/Feedback (3 minutes)
The Case-Specific Team Debriefing is designed to provide QSR Reviewers with the opportunity to share what they learned during the case review and also provides an opportunity for the Site Leads to conduct second level quality assurance.
Review PowerPoint Slide #18 (Case Specific Team Debriefing: Case Presentation Outline)
Explain that each team presents their scores and rationale, and fellow reviewers and the Site Leads provide input and feedback to the scoring process. QSR Reviewers must follow the Case-Specific Team Debrief Case Presentation Outline, which can also be found in the QSR Protocol (Version 4.0, Page 59). 18 18

19. Feedback Session with Caseworker
Explain the purpose
Conduct an accuracy check, analysis, and strengths/challenge assessment
Note that identification of new facts is common
Identify the strengths of the case
Explain findings – Child/Youth and Family Status
Indicators
Explain findings – Practice Performance Indicators
Invite the worker to offer suggestions for improvement
Offer suggestions not identified by the worker
Review PowerPoint Slide #16 (Feedback Sessions with the Caseworker)
Explain the purpose again – accuracy check, analysis and strengths/challenge assessment.
Note that identification of new facts is common.
Identify the strengths of the case.
Confirm the facts of the case.
Explain findings – Child/Youth and Family Status.
Explain findings – Practice Performance.
Invite the worker to offer suggestions for next steps and suggestions for improvement.
QSR Reviewers should refer to the QSR Protocol (Version 4.0, Pages 61-63) for information on how to begin the conversation, discuss practice challenges and discuss recommendations. 19 19 19

20. Site-Specific QSR Information Shared
QSR Review Team Pairings
Dissemination of Case Summaries and Interview Sheets
Parking
Security
Weather
Other
Display PowerPoint Slide #20 (Site-Specific QSR Information Shared).
Site Leads should take this opportunity to share any specific local information that pertains to the onsite QSR.
Examples of things to discuss:
QSR Review Team Pairings
Dissemination of Case Summaries and Interview Sheets
Parking
Security
Weather 20 20

21. Questions/Answers
This is an opportunity for QSR reviewers to ask any additional questions that they might have concerning the material that was reviewed.
Display PowerPoint Slide #21 (Questions/Answers)
Answer participants’ questions
Instruct participants to complete training evaluation
Close by confirming attendance and that all participants that were present at the beginning are still present so that participants receive credit and certification as a QSR Reviewer can be confirmed.
Distribute or refer to Handout #9 (References)