Targets & Improvement Activities State Improvement Planning Carol Massanari, MPRRC Western Regional Resource Center APR Clinic 2010 November 1-3, 2010 San Francisco, California

It’s been six years already? Old age arrives suddenly, as does the snow. One morning, on awakening, one realizes that everything is white. Western Regional Resource Center APR Clinic 2010

Getting here has not always been obvious. Western Regional Resource Center APR Clinic 2010

A New Opportunity Western Regional Resource Center APR Clinic 2010 Using prior data to look at trends over time. Reviewing and evaluating what we’ve been doing. Shaping the form that is emerging from the chaos.

State Improvement Planning SPP/APR as a management tool Evaluating to document and report results

Using the SPP/APR as a Management Tool Jeanna Mullins Jim Leinen, WRRC Western Regional Resource Center APR Clinic 2010 November 1-3, 2010 San Francisco, California

Development Started with information from the November 2009 WRRC Think Tank on “Managing Improvement” Field tested with KY Part B staff Gathered input at Spring TA&D Conference and from RRC Program staff

Design TA Tool for TA providers at the national, regional, State or local levels Guide conversations on SPP process and management structures Serve as an informal formative evaluation Assist in identifying potential areas for improvement

Attributes of Promising Practices Topical Areas Leadership Staff/Workforce Assignments Strategic Planning Technical Assistance and Improvement Activities Measurement, Analysis and Knowledge Management Customer Focus/Results Western Regional Resource Center APR Clinic 2010

Rubric Rating of Attributes Fully Operational Under Development Under Consideration Not Considered at This Time Not Applicable Western Regional Resource Center APR Clinic 2010

Needs Identification Strengths Potential opportunities for change Comparison to federal, State, regional or local priorities for targeting improvement Resources available to support improvement Western Regional Resource Center APR Clinic 2010

Using the SPP/APR as a Management Tool Activity: Select one of the state systems’ Attributes: Leadership, Staff/Work Force Assignments Strategic Planning Technical Assistance and Improvement Activities Measurement Analysis & Knowledge Management Customer Focus/Results Answer the questions related to your selected Attribute Western Regional Resource Center APR Clinic 2010

Activity #1 Team Report Out: What attribute did you select and why? Share1-2 highlights about what you discovered while applying this tool Any comments about the tool? Western Regional Resource Center APR Clinic 2010

Guiding Evaluation and Improvement Planning Logic Models & Evaluation Plans Jeanna Mullins Jim Leinen, WRRC Western Regional Resource Center APR Clinic 2010 November 1-3, 2010 San Francisco, California

Logic Model Elements  Inputs (resources to be used)  Processes (activities and strategies to be used)  Outputs (immediate, countable results produced by activities), and  Outcomes (benefits and changes that are realized)

Logic Model Elements  Inputs (resources to be used)  Processes (activities and strategies to be used)  Outputs (immediate, countable results produced by activities), and  Outcomes (benefits and changes that are realized)

A New Resource EVALUATING SPP/APR IMPROVEMENT ACTIVITIES Developed by members of the Systems and Improvement Planning Priority Team (NECTAC and RRCP) December 2009

Evaluating SPP/APR Improvement Activities Provides: 1. information about evaluation, improvement planning and strategic systems thinking 2. guidance on selecting an appropriate design for evaluating different types of improvement activities 3. additional resources and tools that support SPP/APR implementation and evaluation Western Regional Resource Center APR Clinic 2010

Foundation: Systems Thinking Systems are made up of interrelated, interconnected components Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders Desired changes in one part or level of the system must be accompanied by changes in other parts or levels Western Regional Resource Center APR Clinic 2010

Foundation: Logic Models InputsProcessOutputs Outcomes ‘theory of change’ model to show the process by which change is expected to occur illustrated through a visual depiction or logic model shows how the inputs (resources) and processes (activities) connect to outputs (products or units produced, such as number of staff trained) and outcomes (intended results)

What is a Logic Model? “A systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve.” WK Kellogg Foundation (2002)

Why Use a Logic Model?* Serves as a clear roadmap for the work Provides a framework for planning, implementation and evaluation Graphically illustrates key elements of an improvement activity and how they are interrelated *Source: Evaluation of Improvement Activities: Considerations and Strategies. George Liacopoulos, Ph.D. Mid-South Regional Resource Center’s Systems Improvement Forum, Baltimore, MD. April, 2010.

Benefits of Logic Models* Ensures stakeholders understand the purpose of the activities, the resources needed, the activities to be conducted and their capacity to effect change Tracks what has worked and what has not worked well so that success can be replicated Identifies and prioritizes questions to be asked in an evaluation *Source: Evaluation of Improvement Activities: Considerations and Strategies. George Liacopoulos, Ph.D. Mid-South Regional Resource Center’s Systems Improvement Forum, Baltimore, MD. April, 2010.

Foundations: SPP as long term plan for systems change EI/ECSE are systems of complex, interrelated components with goal of achieving outcomes for children and families Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results

Selecting and Reviewing a Set of Improvement Activities e.g. Was a root cause analysis conducted to drive the activities? Was a logical linkage established between the root cause(s) and proposed activity outcomes? Have specific action steps been developed (e.g., task, activity, resources, timelines)? Have personnel been identified who will develop, implement, monitor and evaluate the improvement activity? Have methods to collect the outcome data been identified? Has a data collection timeline been developed? (And more … see full paper)

Steps for Evaluating Individual Improvement Activities 1.What is the goal or purpose? 2.What methods will you use for data collection? 3.What are your timelines for collecting, analyzing and reporting the data? 4.What is your baseline data that you will monitor for improvement? 5.How will the data be analyzed? 6.How will the evaluation results be used? 7.Who will be responsible for the various aspects of the evaluation?

Scenario: Child Outcomes Data Situation: Considerable amount of missing data on C3/B7 Local programs are unclear about when the data should be collected and reported, etc. Written policies and procedures are unclear, e.g. roles and responsibilities of Part C v. 619 Not all local staff have been trained

State Improvement Activities to address missing data issue: 1.Collaborate across State Part C and Section 619 agencies to clarify roles and responsibilities 2.Revise written policies and procedures to clarify e.g. roles/responsibilities 3.Provide additional training on the child outcomes data collection and reporting process Scenario: Child Outcomes Data

Short term goals of the 3 activities: To clarify roles and responsibilities of Part C and Section 619 school district staff in the collection and reporting To improve the written policies and procedures so that local program staff find them clear and comprehensive To increase local provider/program understanding of the child outcomes data collection and reporting process Long term goal of these 3 activities: To increase statewide reporting of child outcomes data (and decrease the missing data in the child outcomes indicator) Scenario: Child Outcomes Data

Logic Model for Scenario: Child Outcomes Data

Contact Information Western Regional Resource Center APR Clinic 2010 Christina Kasprzak, NECTAC/ECO Jeanna Mullins, MSRRC

Key Considerations Considering it may not be possible to evaluate all improvement activities, you could develop criteria to determine which activities you would like to evaluate:  Where is the state investing most resources?  “More bang for your buck” – What improvement activities can be clustered and/or influence multiple Indicators?  Improvement activities associated with indicators for which no progress is being evidenced  Indicators for which you have no or little data  Extent of implementation (e.g., statewide or more targeted)  Degree of confidence in the improvement activity Western Regional Resource Center APR Clinic 2010

Evaluation Plan/Logic Model Team Activity Instructions: Select an improvement activity from your SPP Explain reasons for selecting this improvement activity Draft an evaluation plan or an evaluation logic model for this activity Western Regional Resource Center APR Clinic 2010

Activity #2 Team Report Out: What indicators or cluster of indicators did you select? What improvement activities did you select and why? Any questions or comments about the process? Western Regional Resource Center APR Clinic 2010