1. 521: The Quality Service Review Process
3-16

2. Day 1 - Agenda Training Overview Overview of the QSR Process
Overview of the Indicators
Roll-Up Sheet
The QSR Interviews
Child/Youth and Family Status Indicators
The Pennsylvania Child Welfare Resource Center

3. Day 2 – Agenda Practice Performance Indicators
The QSR Rating Rationale
Case-Specific Team Debriefing
Feedback Session with the Caseworker/Supervisor
Closing and Evaluation

4. Learning Objectives Identify the components of the QSR Process;
Recognize the QSR Indicators;
Apply the interview techniques relative to the protocol;
Assign valid Status and Performance Indicator scores;
Explain rationale for indicator scores;
Demonstrate a Caseworker/Supervisor Feedback Session; and
Prepare for the Case-Specific Debriefing.

5. QSR’s Connection to CQI
Positive Change
QSR
Pennsylvania’s Continuous Quality Improvement (CQI) effort is the vehicle that will sustain positive change of our child/youth and family outcomes. The Quality Services Review (QSR) process is a component of the CQI effort.
PA CQI

6. DAPIMTM and “Learning by Doing”
Define
Work products – baseline surveys, alignment notes, initial feelings, ground rules, defined areas for improvements, desired future state
Monitor
Work products – monitoring quick wins and other changes, evaluations and measures of progress and impact
Assess
Work products – findings: strengths and gaps, root causes and general remedies
Performance
& Capacity
Implement
Work products – team activities, action plans, charters for working teams, communication efforts
Plan
Work products- quick wins, mid- and long-term improvement plans, communication and capacity plans

7. Purposes of the QSR Process
To appraise the current status of the focus child/youth and family and the performance of key practice areas;
To assure that the actions of a team of child welfare professionals are guided by clear and comprehensive policies and are rooted in Pennsylvania’s Practice Model/Standards;
To improve the quality and usefulness of information needed to evaluate the safety, permanence, and well-being of children and youth;

8. Purposes of the QSR Process (continued)
To assure the quality and effectiveness of services for children, youth, and families by regularly reviewing programs and practices;
To support the efforts of child welfare agencies to maintain an environment that encourages learning and program improvement; and
To stimulate actions and next steps, to build on strengths and support efforts to improve outcomes.

9. What is the QSR?
The QSR is designed to conduct an in depth review of front line practice in specific locations and points in time.
It is used for:
Appraising the current status of a child/youth in key life areas;
Appraising the current status of the parent/caregiver; and
Appraising the performance of key practices for the same child/youth/family.

10. How is the QSR Different From Other Quality Assurance Methods?
Traditional Review Were visits timely? Is the child’s IEP in the file? Was there a referral to mental health? Quality Service Review Are children currently safe? Is the child progressing well academically? Is the child functioning well emotionally?

11. How is the QSR Different From Other Quality Assurance Methods
How is the QSR Different From Other Quality Assurance Methods? (continued)
Traditional Review
Quality Service Review
Is there a current plan in the file?
Is the plan likely to lead to permanence?
Do the parents feel that the plan is their own?
Did the parents sign the plan?
Is there a written
Does the assessment accurately identify underlying needs?
assessment?

12. Skills and Abilities
Discuss with your partner the following questions:
Which three challenges or needs are most important for you to overcome in order to become a successful QSR reviewer?
On a scale of 1-10, with one being a minimal challenge or need and 10 being a major challenge or need, how would you rate your three identified challenges or needs?
What will it take for you to decrease your rating 1-2 points?

13. The QSR Process Review Team Pairings Sample Selection
Case Reviews – review of case record, interviews, scoring of indicators, and writing of rationale for scores
Case-Specific Team Debriefing
Feedback Session with the Caseworker/Supervisor
Exit Conference (optional for reviewers)
Focus Groups and Stakeholder Interviews
Final Report
Next Steps Meeting
County Improvement Plan

14. What Happens Prior to the Onsite Review?
Identification and pairing of reviewers;
Random sample of a county’s cases are selected and stratified by a service area (In Home or Out of Home);
Purpose of review shared with family and agreement to participate in the review obtained; and
Key interviews in each case are identified and appointments are scheduled.

15. Exit Conference – What Happens?
The Local and State Site Leads deliver the aggregate findings from all the cases reviewed along with trends discovered from the focus groups and stakeholder interviews.
A few reviewers may be asked to present the highlights from their case review experience.
Key strengths and areas needing improvement are identified.

16. Focus Groups and Stakeholder Interviews
Purpose:
Obtain feedback specific to systemic issues rather than case-specific information; and
Provide a forum for open dialogue about challenges and solutions within the system.

17. Focus Groups and Stakeholder Interviews
Invited to…
Focus Groups:
Caseworkers, supervisors, managers, service providers, and resource parents
Stakeholder Interviews:
Judges, attorneys, police, probation officers, county commissioners, etc.

18. Final Report – What Happens?
The QSR Final Report provides the finalized quantitative data collected during the week(s) of the onsite QSR, as well as relevant data from other sources. This includes demographic information and the final indicator scores for the Child/Youth and Family Status Indicators as well as the Practice Performance Indicators.
The QSR Final Report shall act as a springboard for the Continuing Quality Improvement (CQI) process, as the county will use the data and analysis to develop a County Improvement Plan.

19. Next Steps Meeting – What Happens?
Designed to be an opportunity for the county to begin/continue their CQI process.
Led by the Local and State Site Leads who discuss the recommendations for next steps in working toward an action plan for improving frontline practice and system performance.

20. What is Your Role? Conduct case reviews:
Each pairing of reviewers will:
Conduct case reviews:
Case file review - overview of the case record;
Interviews - meet with key players involved in the case;
Roll-Up Sheet - demographic information and scoring;
Present information gathered from case review and the rationale for scoring at the Case-Specific Team Debriefing;
Provide feedback to the caseworker (and supervisor) - immediate feedback including suggestions and potential next steps; and
Reviewers may be invited to attend the Exit Conference.

21. Child/Youth and Family Status Indicators
Safety: Exposure to Threats of Harm;
Safety: Risk to Self/Others;
Stability;
Living Arrangement;
Permanency;
Physical Health;
Emotional Well-Being;
Early Learning and Development;
Academic Status;
Pathway to Independence; and
Parent or Caregiver Functioning.

22. Practice Performance Indicators
Engagement Efforts;
Role and Voice;
Teaming;
Cultural Awareness and Responsiveness;
Assessment and Understanding;
Long-Term View;
Child/Youth and Family Planning Process;
Planning for Transitions and Life Adjustments;
Efforts to Timely Permanence;
Intervention Adequacy and Resource Availability;
Maintaining Family Relationships; and
Tracking and Adjusting.

23. What Time Period is Considered in Scoring?
Child/Youth and Family Status Indicators look at the past 30 days
Exceptions are: Stability and Academic Status
Practice Performance Status Indicators look at the past 90 days
Exceptions are: Planning for Transitions and Life Adjustments and Efforts to Timely Permanence
NOTE: Recent performance improvements have some influence on scores

24. How are Cases Scored? Each indicator is scored on a scale of 1 – 6
6 – Optimal
5 – Substantial Acceptable Range
4 – Fair
3 – Marginal
2 – Poor Unacceptable Range
1 – Adverse
Maintenance Zone: 6-5
Refinement Zone: 4-3
Improvement Zone: 2-1

25. Distinction Between 3 and 4
The Groundhog Rule
If this case were frozen in time as it is
today, would performance or status be
considered acceptable or unacceptable?

26. The Central Scoring Rules
Rules Include:
Concentrate on the focus measure and core concepts in each indicator.
Read each indicator’s instructions:
Some indicators score results, others score effort or intent.
Not Applicable rating criteria are identified as appropriate.
Score each sub-category within each indicator, as appropriate.
Stay within the time-based observation windows associated with each indicator.
Rate indicators based on events that have occurred or conditions that were present within the time-based observation window.

27. Roll-Up Sheet Purpose Purpose
Documenting the information gathered during the case review process including information learned about the child/youth/family, case information, and the indicator scores and rationale.

28. Roll-Up Sheet Completion
Some basic case information is filled out by county staff prior to onsite review.
Remaining information is gathered through case file review and interviews.
Reviewers will submit a validated Roll-Up Sheet to Site Leads for their review and approval. 

29. Roll-Up Sheet Components
Case Review Information
Focus Child/Youth’s Information
Case Information
Child/Youth and Family Status Domain
Practice Performance Status Domain
Recommendations

30. Planning QSR Interviews
Know the case history;
Plan relevant questions for each respondent;
Have accurate directions and be punctual; and
Organize the interview to help the respondent participate.

31. Prior to the Interview
Review the interview schedule and begin preparation by identifying potential questions;
Identify the lead interviewer and note-taker for each interview;
Remember that you are a reviewer and not a caseworker during the QSR process; and
Learn something about the interests of the child/youth to better engage him/her.

32. During the Interview Take notes to document the interview;
Take every opportunity to recognize family strengths;
Listen more than talk;
Use the list of QSR questions to keep you on track;
When needed, ask questions related to the data elements you need to answer on the Roll-Up Sheet;

33. During the Interview (continued)
Always ask about the child’s safety (if safety threats are identified, call Site Lead immediately);
Be patient with family stories;
Gather information, do not identify solutions to the case’s or family’s concerns;
Develop the relationship before inquiring about sensitive family issues; and
Avoid leading questions.

34. After the Interview
Do not share information gathered during one interview to the next interview;
Resist reaching conclusions prior to the completion of all interviews;
Contact your Site Lead or next interview if you are running a few minutes behind;
Note areas needing confirmation or follow-up; and
Contact your Site Lead if interviews lead to other key people in the child/youth’s life that you feel would be important to talk to regarding the case review.

35. The QSR Interview – Exploring Questions for Providers
For clinicians - What are your goals for this child/youth/family?
What are the child/youth/family’s goals?
What are the strengths of this child/youth/family?
What worries you most about this case?
Tell me what you are working on now.
Who do you see as members of this family’s team?
What’s the next big hurdle for this child/youth/family?
What will success look like?
On a scale of 1-10, how likely do you think success will be achieved? Ask follow-up questions.
Where do you see this case in six months?

36. Interviewing Practice – Sara’s Case
Both reviewers read the handout, Facts about the Fisher Family.
Reviewer playing the role of the mother reads the handout, Interview with Cynthia Fisher.
Reviewer playing the role of reviewer considers, What do you want to learn in the interview with Sara’s mother?
Conduct the interview.

37. Child/Youth and Family Indicators
With your partner, review Handouts #6-10.
Summarize the indicator, score the indicator, and identify two good questions for each indicator you are assigned.
Group 1: 5 and 1a
Group 2: 2 and 1b
Group 3: 4 and 5
Group 4: 3 and 6
Group 5: 7b and 4
Group 6: 6 and 9
Be prepared to join other teams in a group discussion about the scoring experience.

38. Child/Youth and Family Indicators
With your partner, review Handouts #6-10.
Summarize the indicator, score the indicator, and identify two good questions for each indicator you are assigned.
Group 1: 1a and 9
Group 2: 2 and 1b
Group 3: 4 and 5
Group 4: 3 and 6
Group 5: 7b and 4
Be prepared to join other teams in a group discussion about the scoring experience.

39. Practice Performance Indicators
With your partner, review Handouts #6-10.
Summarize the indicator, score the indicator, and identify two good questions for each indicator you are assigned.
Group 1: 1a and 5
Group 2: 9 and 11
Group 3: 2 and 10
Group 4: 3 and 7
Group 5: 4 and 8
Group 6: 1b and 6
Be prepared to join other teams in a group discussion about the scoring experience.

40. Practice Performance Indicators
With your partner, review Handouts #6-10.
Summarize the indicator, score the indicator, and identify two good questions for each indicator you are assigned.
Group 1: 1a and 5
Group 2: 6, 9, and 11
Group 3: 2 and 10
Group 4: 1b, 3, and 7
Group 5: 4 and 8
Be prepared to join other teams in a group discussion about the scoring experience.

41. QSR Rating Rationale
Purpose: To provide a written narrative the explains the reasoning and justification for why the score was chosen by the reviewers.
Narrative should:
Tell a story of information gathered, including examples
Be succinct, yet descriptive
Narrative should not include:
Names of individuals interviewed
Reviewers should use case participants roles (i.e. mother, father, foster mother, attorney, etc.)
A reiteration of the QSR indicator score description

42. Status Indicator 1a: Safety: Exposure to Threats of Harm
Substitute Home (Factors Contributing Favorably)
The child is safe and is in a desirable and enduring safety situation in her current foster home as no safety concerns have been raised regarding the current foster home based on interviews conducted and observations by reviewers. While in the foster home, no safety threats have been reported/observed by those that conduct regular visits to the foster home. None of the participants interviewed reported any concerns about the child’s safety. The child was interviewed by reviewers in the child’s home and there was no indications of any safety concerns, as the child has food, clothing, shelter that are all adequate to meet her basic needs.

43. Status Indicator 1a: Safety: Exposure to Threats of Harm
Family Home #1 (Factors Contributing Unfavorably) It is critical to note that during this Quality Service Review (QSR), information was gathered that during the visit at the paternal grandmother’s home, the sleeping arrangements presented significant safety concerns as it was learned that the child slept in the same bed with her father. Given the fact that the ABCCCYS was aware of the father’s sexual assault history, probable contact between the children and their father during the visit, and the paternal grandmother’s known space limitations, a concrete safety plan should have been in place prior to permitting the visit.

44. Status Indicator 1a: Safety: Exposure to Threats of Harm
Family Home #1 (Factors Contributing Unfavorably) (continued) It was also alleged, but not able to be confirmed by the reviewers, that the child’s father permitted the other children to drink alcohol during the visit. As a result of the lack of safety plan, the review findings indicate that the paternal grandmother’s home is a setting that poses a serious and worsening threat of harm for the child. Protective strategies have not been implemented in this home and therefore, without a safety plan, the child is at risk of continuing and worsening harm.

45. Practice Indicator 9: Intervention Adequacy and Resource Availability
Adequacy (Factors Contributing Favorably) The services the child receives (i.e. counseling, child advocacy, and IEP services) appear to be available and adequate. The therapy the mother receives has been helpful. Interviews revealed that she has a good relationship with her therapist with whom she has able to identify many of the issues she needs to address.

46. Practice Indicator 9: Intervention Adequacy and Resource Availability
Adequacy (Factors Contributing Unfavorably)
Services have not improved either parent’s capacity to provide long-term care for their children. Reunification services for the mother were closed out by ABCCCYS a couple months ago without a recommendation for a more intensive services or change in goal. No substantial progress has been made over the past four months. Most services the mother has received have been either inadequate or underpowered, as they have not led to enhancing her parental capacities to care for her child. The mother, father, and paternal grandmother all need concrete supports in securing housing large enough to provide for the children. This includes financial support as well as specific education on how to manage a budget and maintain a home. Access to welfare or housing benefits to support reunification are not available to either parents or grandparent if they do not have the children living with them.

47. Practice Indicator 9: Intervention Adequacy and Resource Availability
Availability (Factors Contributing Favorably)
The services that the child receives (i.e. counseling, child advocacy, and IEP services) appear to be available and adequate. Resources appear to be readily available in the area of the mother, father, substitute caregiver and child.

48. Practice Indicator 9: Intervention Adequacy and Resource Availability
Availability (Factors Contributing Unfavorably) The mother, father, and paternal grandmother all need concrete supports in securing housing large enough to provide for the children. This includes financial support as well as specific education on how to manage a budget and maintain a home. Access to welfare or housing benefits to support reunification are not available to either parents or grandparent if they do not have the children living with them.

49. Case-Specific Team Debriefing
Inter-rater reliability check for consistency between case description and scores.
Additional opportunity to practice sharing the rating and rationale based on the information gathered as part of the case review.
Identification of themes for feedback to the county.

50. Case-Specific Team Debriefing Outline
Case Presentation Outline – 15 minutes
Case Summary (2 minutes)
Child/Youth and Family Status Indicators – Strengths and Challenges (4 minutes)
Practice Performance Indicators – Strengths and Challenges (4 minutes)
Next Steps (2 minutes)
Group Questions/Feedback (3 minutes)

51. Caseworker/Supervisor Feedback Session
Thank the caseworker and supervisor for their time.
Review the QSR purpose. 
Outline the strengths of the case.
Remind caseworker of the high QSR standards. 
Very briefly summarize family story – ensure facts are correct.
Share new information, if applicable.
Ask caseworker for ideas for next steps.
Discuss the strengths and challenges of the indicators.
Share reviewer ideas for next steps.
Thank the caseworker/supervisor and emphasize that they had a meaningful role in making a difference for children, their families, and the system.

52. Feedback Session Practice
Please complete Handout #14, Notes for Providing Feedback to Caseworkers.
When completed, one partner will be the caseworker, the other partner, the reviewer.
Practice a feedback session providing the caseworker feedback on Sara’s case.
Compare your notes with your partner from Handout #14, did you identify similar:
Strengths/Accomplishments;
Practice Challenges/Limiting Conditions;
Opportunities for Advancing Practice/ Improving Local Conditions (Next Steps).

53. CQI Continues…
Remember QSR is NOT a one-time event. It’s part of the bigger CQI process that leads to improved outcomes of Safety, Permanence, and Well-Being for children, youth, and families.