1. Overview of Continuous Quality Improvement (CQI) and Quality Service Review (QSR) Process
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

2. Meeting Objectives
Introductions of QSR State Site Lead(s) and Local Site Lead(s)
Overview of the Continuous Quality Improvement (CQI) process and the role of Quality Service Review (QSR) in that process
Purpose of the QSR process – Case-specific review; Focus Groups and Key Stakeholder Interviews
Overview of the components of the QSR process and timelines associated with the onsite review
Overview of the QSR indicators
Composition of the review team
Introductions of QSR State Site Lead(s) and Local Site Lead(s) – make sure that all site leads have each other’s contact information. Can add information on to the QSR Team Chart – the team chart will also include the contact information for both the local and state reviewers.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

3. Continuous Quality Improvement
What it isn’t and what it is…
Continuous Quality Improvement (CQI) is not a time limited project or initiative. It is the ongoing process by which an agency makes decisions and evaluates its progress.
“A framework for implementation”
Casey Family Programs & NRCOI
Mission: “Pennsylvania’s Continuous Quality Improvement (CQI) effort is the vehicle that will sustain positive change for our child/youth and family outcomes...”
Implementing change at the local level is critical to the achievement of positive child, youth and family outcomes, particularly in a state-supervised and county-administered state. Continuous quality improvement has been defined as “the ongoing process by which an agency makes decisions and evaluates its progress.” (Casey Family Programs and NRCOI)
CQI benefits for PA
Achieving and Sustaining Change at the local level
Shift from compliance to quality
Framework for aligning resources
Connects to Practice Model and standards
In the CQI Self Assessment you were asked to identify any existing Quality Assurance (QA) or Quality Improvement (QI) efforts that currently exist within your agency.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 3

4. What’s in it for your County?
Establishes an environment that encourages learning and program improvement.
Provides a tool for the evaluation of best case practice rooted in Pennsylvania’s Practice Model/Standards
In depth review of child/youth and family’s safety, permanency and well being outcomes as well as the agency’s practice performance.
Provides information/data that will help support/guide ongoing improvement efforts
Practice improvements
Organizational and systemic improvements
Improved outcomes for children, youth and families
It will be helpful to have the county talk about why they chose to participate in the CQI effort – including sharing information that was outlined in the self-assessment.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

5. The DAPIMTM Model: A “Flywheel”
Define
Monitor
Assess
Performance
& Capacity
PA’s Model of CQI is the DAPIM framework
Since this is a flywheel which revolves continuously there is no one specific start point – counties may be at very different places along the flywheel at any given time.
We are going to begin the discussion focusing on the MONITORING PHASE. The Quality Service Review case review process is ONE potential source to monitor your agency’s achievement of safety, permanency and well-being outcomes as well as practice performance of the system. It is also a tool to ASSESS your agency.
Implement
Plan
© 2009 American Public Human Services Association
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 5

6. Organizing for Continuous Improvement
Work Team III
Work Team II
Sponsor Group
Provide high level oversight; obtain resources; set expectations that align to strategy
Continuous Improvement (CI) Team
Initiate and manage continuous improvement effort; maintain hands-on responsibility for CI efforts during and after the facilitated process
TIME
Work Team I:
Implement plans that require complex and/or extended efforts (e.g., process design)
There needs to be a structure in place to assure that the DAPIM framework is supported and that there are people and resources identified for supporting and monitoring the continuous improvement efforts.
Continuous improvement relies on strong internal sponsorship to become a way of doing business. When an agency embarks on continuous quality improvement efforts, executive teams become “Sponsor Teams,” who are accountable for defining the high level vision of continuous improvement efforts and securing resources required for success. One crucial resource is people power; therefore, one of the Sponsor Team’s most important tasks early in the process is the creation and chartering of an “Improvement Team.”
An “Improvement Team” is the coach and guide of continuous improvement. Members of this team set the direction for and guide continuous improvement work day-to-day.  This team has hands-on responsibility for the improvement efforts and maintains ongoing responsibility for monitoring the continuous improvement efforts.  Teams should be large enough to represent key internal stakeholders, but small enough to make recommendations and decisions. (10-15 members suggested in large organizations.) Members of these teams should be considered content experts, be committed to continuous improvement and a willingness to understand Organizational Effectiveness models (such as APHSA’s DAPIM™ approach). Members should have the ability to build trusting relationships (internal and external to the organization).
Often times, members of these teams will also charter and oversee the efforts of smaller work teams while also being a communicator to the Sponsor Team about the status of the continuous improvement efforts and provide recommendations to the Sponsor Team based on the monitoring of the continuous improvement efforts.
© 2011 American Public Human Services Association. All rights reserved.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

7. Purpose of the Quality Service Review?
Strategies for Success: Tools to Support Supervision and Quality Practice
Purpose of the Quality Service Review?
5/6/2010
Measures child, youth, and family outcomes
Reveals the practice model being used in actual cases.
QSR is an organizational learning process offering helpful ways of knowing what’s working and not working in practice -- for which children and families and why.
QSR connects results to local frontline conditions.
QSR supports teaching and action learning processes that clarify expectations, provide useful feedback, and affirm good work.
QSR stimulates actions taken to improve practice and results at all levels of the organization.
Acknowledge QSR Protocol is copyrighted by Human Systems & Outcomes
QSR is an organizational learning process offering helpful ways of knowing what’s working and not working in practice -- for which children and families and why.
We can use it to find out how children and families are benefiting from services received and how well locally coordinated services are working for children and families.
QSR reveals the practice model being used in actual cases.
Connects the values and principles to the case practice – which is different from last PA QSR which focused solely on CFSR outcomes of safety, permanency, well-being and 7 systemic factors – but did not connect to practice model or standards
QSR connects results to local frontline conditions.
QSR Identifies LOCAL conditions under which practice is being done; provides timely results to specific cases as well as systemic/practice strengths and concerns
QSR supports teaching & action learning processes that clarify expectations, provide useful feedback, and affirm good work.
Strength-based approach. Unlike previous PA QSR this process provides immediate feedback to the CW and Supervisor for that child/family, as well as practice performance areas that are needed to improve outcomes
QSR stimulates actions taken to improve practice and results at all levels of the organization.
Recommendations and sharing strengths, sharing areas needing improvement, sharing perspective learned after talking with everyone on the TEAM.
QSR FOCUSES ON “FUNCTIONAL PRACTICE PERFORMANCE RATHER THAN SIMPLE COMPLIANCE with Policies and procedures.” The case review is a PROFESSIONAL (honest and informed) APPRAISAL of: (1) the status of a focus child and parent/caregiver on key indicators; (2) recent progress made on applicable change indicators; and (3) adequacy of performance of essential service functions for that child and parent/caregiver. Each focus child served is a UNIQUE and VALID point-in-time “test” of frontline practice performance in a local system.
2010 Spring Supervisor Training Event
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 7

8. Strategies for Success: Tools to Support Supervision and Quality Practice
5/6/2010
QSR Process…
State and local review team members review case of one focus child/youth
Brief review of the record
Focused interviewers with members of the child/youth/family team
Rates the status of the child/youth/family and the system performance on a 6-1 scale
Provides written rationale for the ratings as well as case-specific and agency-specific recommendations
QSRs contain a very small RANDOM sample but it may only be an inch wide but it is a mile deep. One case: In-depth case reviews to measure current status, recent progress, and adequacy of current practices in getting results for children/youth and families being served in local sites. [Drill Downs] Case review provides a MICRO VIEW of a case – Each family served is a unique “test” of the service system.
Brief review of record– includes having the reviewers gain some case knowledge to prepare for interviews
Focused interviews – incorporates in person/phone interviews of all integral members of the child/family team; focus is on gathering information, not on sharing information
Rating – uses a scale of 1-6 to determine the status of the child/family and the performance of the team in working with that child/family; Uses aggregate quantitative patterns of qualitative indicators to reveal and describe the quality and consistency of local practice.
Rating Rationale and Recommendations - Uses the power of the story to reveal what is happening by providing narrative and examples based on information gathered as part of the case review process. [Stories Teach] Power of story – we can’t understand what is behind the #’s (the data) without knowing the story behind the data
2010 Spring Supervisor Training Event
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 8

9. QSR Process…continued
Immediate feedback to the Caseworker and Supervisor with strengths and needs, as well as recommendations
Case specific
Agency specific
Focus groups with stakeholders and key stakeholder interviews
Debriefing with QSR Team
Exit Conference of preliminary findings
Final findings report
Next Steps Meeting
County Improvement Plan (CIP)
Immediate Feedback to CW/Sup/JPO(if applicable) : Purposes – 1) Understand current status and facts of the case, as well as sharing new facts of the case that come to light during the interview process; 2) Offer helpful successful recommendations for the child/family situation and for the organization (team) as a whole on how to improve practice with the child/family; Focus is on learning and improvement. They will also receive a Written Case Review Summary that details the information shared during the feedback session.
Focus Groups/Key Stakeholder Interviews: Local focus group and key stakeholder interviews along with case stories, data patterns, and local working conditions to find and affirm what’s working now and to surface areas where even better results might be achieved in the future.
Focus groups – to provide a macro-view of local site and local conditions
Provides information about local practice/resources/collaboration/coordination/working conditions-helps contextualize/explain findings
Debriefing with QSR Team re: findings: Second level QA on the reviews of the cases; providing an overview of the preliminary rating scores; discussion about validity of findings based on agency’s perspective – does this surprise you, are there “a-ha” moments?
Exit Conference of Preliminary findings: preliminary data and themes identified during focus groups are presented to the agency at the conclusion of the onsite review week by the local and state site leads.
Final findings report: Review of demographic data; Final findings following 2nd level QA.
Next Steps Meeting: a starting point or continuation for the county agency’s efforts to begin/continue the CQI process of developing an action plan (County Improvement Plan) for enhancing case practice and system performance.
County Improvement Plan: turning the flywheel of DAPIM – applying the findings/data to drive continuous quality improvement
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

10. Connecting QSR and Licensure
QSR Case Reviews
QSR Focus Groups and Key Stakeholder Interviews
DHS
Licensure
Interviews with case participants
Caseworkers
Same cases as QSR sample
Feedback to Caseworker/Supervisor
Additional intake, foster home, and personnel files
Supervisors
Roll-Up Sheet Data, Rationale and Recommendations
Others: may consider youth, birth families, foster families, courts, etc.
Connecting QSR and Licensure
Case Reviews:
In-depth review of small sample of cases - interviews with family team members. Immediate feedback to caseworkers & supervisors – strength-based review of case-specific findings
Oral presentations of QSR case findings (team debriefings) – allows for identification of recurring patterns and also allows for 2nd level quality assurance of reviewer’s scoring
Roll-Up Sheet Data and Rationale – Outlines facts about the case and provides narrative about the factors contributing to the favorable and unfavorable findings. These are/were provided back to the county worker/sup/admin to use to help move the case forward – review team’s recommendations.
Focus groups/key stakeholder interviews: Provides additional opportunity to look at issues that affect many different stakeholders in the system and to elicit ways to improve the system. Caseworker and Supervisor focus groups are mandatory for counties who are new to the CQI effort. Repeat counties are asked to conduct a minimum of two focus groups with groups and focus of their choosing.
Aggregate and Quantitative Results: Preliminary results of aggregate quantitative results from case-specific reviews and information about trends discovered from focus groups provided. Final Report – developed with site lead team and HZA
DPW Licensure: Process of licensing has been abbreviated for counties involved in a state-supported QSR – the sample pulled from QSR were records reviewed as part of annual licensing. Licensing Review focused on themes of: Safety Assessment and Monitoring; Family Engagement; Family Service Planning; Casework visitation with family members; Permanency Activities; Placement settings; Data tracking (re-abuse, re-entry); Fostering Connection activities; and Shared Case Responsibility
Review for regulatory compliance
Aggregate quantitative results
Summation, Discussion, Next Steps Planning, County Improvement Plan Development
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 10

11. PA QSR Protocol: Child/Youth and Family Status Indicators
1a. Safety: Exposure to Threats of Harm
1b. Safety: Risk to Self/Others
2. Stability
3. Living Arrangement
4. Permanency
5. Physical Health
6. Emotional Well-being
7. Learning & Development
8. Pathway to Independence
9. Parent & Caregiver Functioning
The new PA QSR protocol may be accessed via the following link:
QSR Protocol gives specific set of indicators to use when examining status of child/youth and parent/caregiver, and when analyzing responsiveness and effectiveness of core practice functions
Child/Youth and Family Status Indicators measure extent to which desired conditions are present in the life of focus child, child’s parents/caregivers over past 30 days (with a few exceptions)
(Site Leads may want to use the actual QSR Protocol, or the 2-page At-a-Glance Summary of Indicators (attachment within the QSR Protocol) to walk group through the indicators)
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 11

12. PA QSR Protocol: Practice Performance Indicators
1a. Engagement Efforts
1b. Role and Voice
2. Teaming
3. Cultural Awareness & Responsiveness
4. Assessment & Understanding
5. Long-Term View
6. Child/Youth and Family Planning Process
7. Planning for Transitions and Life Adjustments
8. Efforts to Timely Permanence
9. Intervention Adequacy & Resource Availability
10. Maintaining Family Connections
11. Tracking & Adjusting
Practice Indicators measure extent to which core practice functions are applied successfully by practitioners, and others on the child/youth and family team over the past 90 days (with a few exceptions)
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 12

13. Interpretive Guide for Child/Youth and Family Status Indicator Ratings
Acceptable Range: 6 - 4
Unacceptable Range: 3 - 1
Maintenance Zone: 6–5
Refinement Zone: 4–3
Improvement Zone: 2–1
Status is favorable. Efforts should be made to maintain and build upon a positive situation.
Status is minimum or marginal, may be unstable. Further efforts are necessary to refine the situation.
Status is problematic or risky. Quick action should be taken to improve the situation. 6 5 4 3 2 1
Optimal Status
Substantial Status
Fair Status
Marginal Status
Poor Status
Adverse Status
The best of most favorable status presently attainable for this individual in this area (taking age and ability into account).
Substantially and dependably positive status for the individual in this area with an ongoing positive pattern. Status is good and likely to continue.
Status is at least minimally or temporarily sufficient for the individual to meet short-term needs or objectives in this area.
Status is mixed, limited or inconsistent and not quite sufficient to meet the individual’s short-terms needs or objectives now in this area.
Status is and may continue to be poor and unacceptable. Any risks may be mild to serious.
The individual’s status in this area is poor, unacceptable and worsening.
Ratings range from fall within the acceptable range and 3-1 fall within unacceptable range which but can also break down scores even further within the acceptable and unacceptable categories.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

14. Final Report
The Final Report provides an analysis of the qualitative and quantitative data collected during onsite review, as well as relevant data from other sources.
The information in the Final Report includes: demographics information specific to the case sample and compared to the county as a whole; quantitative data regarding the case-specific ratings for indicators; any additional data from the county’s data packets that may relate to an item; case-specific examples gleaned from the analysis of the rating rationale; a summary of results that presents overarching areas of strength and opportunities for improvement. Report is developed by HZA with help from state and local site leads and then provided to the county for review/edits – final is provided to administrator by OCYF RO.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a

15. Purpose: The Next Steps Meeting is designed to be a starting point or the continuation of the county’s efforts in monitoring the Continuous Quality Improvement (CQI) process which focuses on the development and/or monitoring of an action plan for enhancing case practice and system performance.
The Next Steps Meeting is designed to be a starting point or continuation for the county agency’s efforts to begin/continue the CQI process of developing an action plan (County Improvement Plan) for enhancing case practice and system performance.
During this meeting we will:
Provide an overview and outline key components of the statewide CQI effort that is being phased in to shift from compliance to quality;
Review the QSR Process and the strengths and concerns outlined from the QSR;
Review the themes focused on in the licensing review and note how this review coincides with the QSR results;
Review the concepts of Sponsor Team and Improvement Teams and how these entities will be integral in the continuous improvement process;
Outline of the components of the County Improvement Plan; and
Discuss the Statewide TA Collaborative effort and the County’s-specific TA Collaborative and how these entities can support the ongoing continuous improvement work.
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 15

16. County Improvement Plan (CIP)
Purpose: The County Improvement Plan (CIP) outlines the priorities the county agency chooses to focus on to improve specific outcomes as a result of a comprehensive review of their practice. This review is not limited to the QSR findings, and may also include a review of additional data such as the County data packages provided by the State, quantitative measures produced by the county, as well as the results of other qualitative data.
Development of County Improvement Plan, to be submitted within 120 days of the last day of the onsite QSR – unless an extension is granted by OCYF.
When planning, a county should consider the three types of continuous improvement planning:
Quick wins
Mid-term improvement planning
Long-term improvement planning
Pennsylvania's QSR Manual Version 4.0 Appendix 5a 16

17. Questions and comments
Pennsylvania's QSR Manual Version 4.0 Appendix 5a