1. 521: The Quality Service Review Process

2. The Pennsylvania Child Welfare Resource Center Day 1 - Agenda Training Overview Overview of the QSR Process Overview of the Indicators Roll-Up Sheet The QSR Interviews Child/Youth and Family Status Indicators 2

3. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Day 2 – Agenda Practice Performance Indicators Feedback Session with the Caseworker/Supervisor Case-Specific Team Debriefing The QSR Written Case Review Summary Closing and Evaluation 3

4. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Learning Objectives Identify the components of the QSR Process; Recognize the QSR Indicators; Apply the interview techniques relative to the protocol; Assign reliable status and performance indicator scores; Demonstrate a Caseworker/Supervisor Feedback Session; Identify the steps to prepare for the Case-Specific Debriefing; and Recognize the components of the QSR Written Case Review Summary. 4

5. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center PA CQI Positive Change QSR QSR’s Connection to CQI 5

6. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center DAPIM TM and “Learning by Doing” Implement Work products – team activities, action plans, charters for working teams, communication efforts Plan Work products- quick wins, mid- and long- term improvement plans, communication and capacity plans Define Work products – baseline surveys, alignment notes, initial feelings, ground rules, defined areas for improvements, desired future state Performance & Capacity Monitor Work products – monitoring quick wins and other changes, evaluations and measures of progress and impact Assess Work products – findings: strengths and gaps, root causes and general remedies 6

7. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Purposes of the QSR Process To appraise the current status of the focus child/youth and family and the performance of key practice areas; To assure that the actions of a team of child welfare professionals are guided by clear and comprehensive policies and are rooted in Pennsylvania’s Practice Model/Standards; To improve the quality and usefulness of information needed to evaluate the safety, permanence and well- being of children and youth; 7

8. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Purposes of the QSR Process (continued) To assure the quality and effectiveness of services for children, youth and families by regularly reviewing programs and practices; To support the efforts of child welfare agencies to maintain an environment that encourages learning and program improvement; and To stimulate actions and next steps, to build on strengths and support efforts to improve outcomes. 8

9. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center What is the QSR? The QSR is designed to conduct an in depth review of front line practice in specific locations and points in time. It is used for: Appraising the current status of a child/youth in key life areas; Appraising the current status of the parent/caregiver; and Appraising the performance of key practices for the same child/youth/family. 9

10. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center How is the QSR Different From Other Quality Assurance Methods? Traditional Review Were visits timely? Is the child’s IEP in the file? Was there a referral to mental health? Quality Service Review Are children currently safe? Is the child progressing well academically? Is the child functioning well emotionally? 10

11. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center How is the QSR Different From Other Quality Assurance Methods? (continued) Traditional Review Is there a current plan in the file? Did the parents sign the plan? Is there a written assessment? Quality Service Review Is the plan likely to lead to permanence? Do the parents feel that the plan is their own? Does the assessment accurately identify underlying needs? 11

12. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Skills and Abilities Discuss with your partner the following questions: Which three challenges or needs are most important for you to overcome in order to become a successful QSR Reviewer? On a scale of 1-10, with one being a minimal challenge or need and 10 being a major challenge or need, how would you rate your three identified challenges or needs? What will it take for you to decrease your rating 1-2 points? 12

13. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center The QSR Process 1.Review Team Pairings 2.Sample Selection 3.Focus Groups and Stakeholder Interviews 4.Case Reviews 5.Feedback Session with the Caseworker/Supervisor 6.Case-Specific Debriefing 7.QSR Written Case Review Summary 8.Exit Conference (optional for reviewers) 9.Final Report 10.Next Steps Meeting 11.County Improvement Plan 13

14. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center What Happens Prior to the Onsite Review? Identification and pairing of internal and external reviewers; Random sample of a county’s cases are selected and stratified by a service area; Family permission obtained to participate in the review and consents/releases obtained; and Key interviews in each case are identified and appointments are scheduled. 14

15. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Focus Groups and Stakeholder Interviews Purpose: –Obtain feedback specific to systemic issues rather than case-specific information; and –Provide a forum for open dialogue about challenges and solutions within the system. 15

16. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Focus Groups and Stakeholder Interviews Invited to… Focus Groups: Caseworkers, Supervisors, Managers, Service Providers, and Resource Parents Stakeholder Interviews: Judges, Attorneys, Police, Probation Officers, County Commissioners, etc. 16

17. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Exit Conference – What Happens? A few reviewers will be asked to present the highlights from their case review. The Local and State Site Leads deliver the aggregate findings from all the cases reviewed along with trends discovered from the focus groups and stakeholder interviews. Key strengths and areas needing improvement are identified. 17

18. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Final Report – What Happens? Final Report which includes analysis of demographic information, quantitative analysis of indicator scores, and information from the County Specific Data Packet is shared. The county will use the data and analysis to develop a County Improvement Plan. 18

19. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Next Steps Meeting – What Happens? Approximately 6-8 weeks after the onsite review; Led by the Local and State Site Leads; Lasts approximately two hours; Discuss the recommendations for next steps in working toward an action plan for improving frontline practice and system performance; and Begins the next step in the CQI process. 19

20. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center What is Your Role? Each pairing of reviewers will: Conduct case reviews:  Case file review - overview of the case record;  Interviews - meet with key players involved in the case;  Roll-Up Sheet - demographic information and scoring; Provide feedback to the caseworker (and supervisor) - immediate feedback including suggestions and potential next steps for the case; Present information gathered from case review and feedback on the family story at the Case-Specific Team Debriefing; Complete a QSR Written Case Review Summary; and Reviewers may be invited to attend the Exit Conference. 20

21. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Child/Youth and Family Status Indicators Safety: Exposure to Threats of Harm; Safety: Risk to Self/Others; Stability; Living Arrangement; Permanency; Physical Health; Emotional Well-Being; Early Learning and Development; Academic Status; Pathway to Independence; and Parent or Caregiver Functioning. 21

22. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Practice Performance Indicators Engagement Efforts; Role and Voice; Teaming; Cultural Awareness and Responsiveness; Assessment and Understanding; Long-Term View; Child/Youth and Family Planning Process; Planning for Transitions and Life Adjustments; Efforts to Timely Permanence; Intervention Adequacy and Resource Availability; Maintaining Family Relationships; and Tracking and Adjusting. 22

23. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center What Time Period is Considered in Scoring? Child/Youth and Family Status Indicators look at the past 30 days –Exceptions are: Stability and Academic Status Practice Performance Status Indicators look at the past 90 days –Exceptions are: Planning for Transitions and Life Adjustments and Efforts to Timely Permanence NOTE: Recent performance improvements have some influence on scores 23

24. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center How are Cases Scored? Each indicator is scored on a scale of 1 – 6 6 – Optimal 5 – Substantial Acceptable Range 4 – Fair 3 – Marginal 2 – Poor Unacceptable Range 1 – Adverse Maintenance Zone: 6-5 Refinement Zone: 4-3 Improvement Zone: 2-1 24

25. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Distinction Between 3 and 4 The Groundhog Rule If this case were frozen in time as it is today, would performance or status be considered acceptable or unacceptable? 25

26. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center The Central Scoring Rules Rules Include: Concentrate on the Focus Measure and Core Concepts in each indicator. Read each indicator’s instructions: –Some indicators score results, others score effort or intent. –Not Applicable rating criteria are identified as appropriate. –Score each sub-category within each indicator, as appropriate. Stay within the time-based observation windows associated with each indicator. Rate indicators based on events that have occurred or conditions that were present within the time-based observation window. 26

27. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Roll-Up Sheet Purpose Purpose Gather Data and Monitor Practice Model Implementation: –Demographic information –Case information –Indicator scores 27

28. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Roll-Up Sheet Completion Basic case information is filled out by the caseworker prior to onsite review. Remaining information is gathered through case file review and interviews. Reviewers will submit a validated Roll-Up Sheet electronically to Site Leads on a flash drive on the final day of onsite review. 28

29. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Roll-Up Sheet Components 1.General Review Information 2.Child/Youth’s Demographic Information 3.Child/Youth’s Current Placement 4.Child/Youth’s Educational Information 5.Case Information 6.Child/Youth and Family Status Domain 7.Practice Performance Status Domain 29

30. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Planning QSR Interviews Know the case history; Plan relevant questions for each respondent; Have accurate directions and be punctual; and Organize the interview to help the respondent participate. 30

31. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Prior to the Interview Review the interview schedule and begin preparation by identifying potential questions; Identify the lead interviewer and notetaker for each interview; Remember that you are a reviewer and not a caseworker during the QSR process; and Learn something about the interests of the child/youth to better engage him/her. 31

32. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center During the Interview Take notes to document the interview; Take every opportunity to recognize family strengths; Listen more than talk; Use the list of QSR questions to keep you on track; When needed, ask questions on the Roll-Up Sheet; 32

33. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center During the Interview (continued) Always ask about the child’s safety (if safety threats are identified, call site lead immediately); Be patient with family stories; Gather information, do not identify solutions to the case’s or family’s concerns; Develop the relationship before inquiring about sensitive family issues; and Avoid leading questions. 33

34. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center After the Interview Do not share information gathered during one interview to the next interview; Resist reaching conclusions prior to the completion of all interviews; Contact your site lead or next interview if you are running a few minutes behind; Note areas needing confirmation or follow-up; and Contact your site lead if interviews lead to other key people in the child/youth’s life that you feel would be important to talk to regarding the case review. 34

35. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center The QSR Interview – Exploring Questions for Providers For clinicians - What are your goals for this child/youth/family? What are the child/youth/family’s goals? What are the strengths of this child/youth/family? What worries you most about this case? Tell me what you are working on now. Who do you see as members of this family’s team? What’s the next big hurdle for this child/youth/family? What will success look like? On a scale of 1-10, how likely do you think success will be achieved? Ask follow-up questions. Where do you see this case in six months? 35

36. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Interviewing Practice – Sara’s Case Both reviewers read the handout, Facts about the Fisher Family. Reviewer playing the role of the mother reads the handout, Interview with Cynthia Fisher. Reviewer playing the role of reviewer considers, What do you want to learn in the interview with Sara’s mother? Conduct the interview. 36

37. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Child/Youth and Family Indicators With your partner, review Handouts #6-10. Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 5 and 1a Group 2: 2 and 1b Group 3: 4 and 5 Group 4: 3 and 6 Group 5: 7b and 4 Group 6: 6 and 9 Be prepared to join other teams in a group discussion about the scoring experience. 37

38. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Child/Youth and Family Indicators With your partner, review Handouts #6-10. Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 1a and 9 Group 2: 2 and 1b Group 3: 4 and 5 Group 4: 3 and 6 Group 5: 7b and 4 Be prepared to join other teams in a group discussion about the scoring experience. 38

39. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Practice Performance Indicators With your partner, review Handouts #6-10. Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 1a and 5 Group 2: 9 and 11 Group 3: 2 and 10 Group 4: 3 and 7 Group 5: 4 and 8 Group 6: 1b and 6 Be prepared to join other teams in a group discussion about the scoring experience. 39

40. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Practice Performance Indicators With your partner, review Handouts #6-10. Summarize the indicator, score the indicator and identify two good questions for each indicator you are assigned. Group 1: 1a and 5 Group 2: 6, 9, and 11 Group 3: 2 and 10 Group 4: 1b, 3, and 7 Group 5: 4 and 8 Be prepared to join other teams in a group discussion about the scoring experience. 40

41. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Caseworker/Supervisor Feedback Session Thank the caseworker and supervisor for their time. Review the QSR purpose. Outline the strengths of the case. Remind caseworker of the high QSR standards. Very briefly summarize family story – ensure facts are correct. Share new information, if applicable. Ask caseworker for ideas for next steps. Discuss the strengths and challenges of the indicators. Share reviewer ideas for next steps. Thank the caseworker/supervisor and emphasize that they had a meaningful role in making a difference for children, their families and the system. 41

42. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Feedback Session Practice Please complete Handout #13, Notes for Providing Feedback to Caseworkers. When completed, one partner will be the caseworker; the other partner, the reviewer. Practice a feedback session providing the caseworker feedback on Sara’s case. Compare your notes with your partner from Handout #13, did you identify similar: –Strengths/Accomplishments; –Practice Challenges/Limiting Conditions; –Opportunities for Advancing Practice/ Improving Local Conditions (Next Steps). 42

43. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Case-Specific Team Debriefing Inter-rater reliability check for consistency between case description and scores. Identification of themes for feedback to the county and written report. Preliminary communication about review findings to leadership. 43

44. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center Case-Specific Team Debriefing Outline Case Presentation Outline – 15 minutes Case Summary (2 minutes) Child/Youth and Family Status Indicators – Strengths and Challenges (4 minutes) Practice Performance Indicators – Strengths and Challenges (4 minutes) Next Steps (2 minutes) Group Questions/Feedback (3 minutes) 44

45. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center The QSR Written Case Review Summary - Purpose Explain the “story” behind the data; Provide feedback to the caseworker and local leadership; and Be utilized as an effective training tool within the county agency. 45

46. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center The QSR Written Case Review Summary- Additional Guidelines Should not tell the case story/history; but rather include information you gathered to support the ratings of the indicators; Written about the ratings and why the factors contributing to the indicators are either favorable or unfavorable; Practical and helpful recommendations; and Make a note of your due date:  Three day Review – Due prior to the conclusion of the 3 rd day  Two day Review – Due approximately 7-10 days following the onsite QSR NOTE: There will be an editing process done by the Site Lead Team, so you may be asked to edit your Written Case Review Summary and/or provide more supportive information. 46

47. 521: The Quality Service Review Process The Pennsylvania Child Welfare Resource Center CQI Continues… Remember QSR is NOT a one-time event. It’s part of the bigger CQI process that leads to improved outcomes of Safety, Permanence and Well-Being for children, youth and families. 47