Using Data for Program Improvement Christina Kasprzak, NECTAC/ECO Ann Bailey, NCRRC July 2010 Early Childhood Outcomes Center

Systems Thinking: Systems are made up of interrelated, interconnected components Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders Desired changes in one part or level of the system must be accompanied by changes in other parts or levels Early Childhood Outcomes Center

SPP as long term plan for systems change: EI/ECSE are systems of complex, interrelated components with goal of achieving outcomes for children and families Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results Early Childhood Outcomes Center

Developing improvement activities: Effective Policies, Procedures, and MOU Data Systems and Monitoring TA and Professional Development Fiscal Management Evaluation Early Childhood Outcomes Center

Evaluating SPP/APR Improvement Activities Early Childhood Outcomes Center

The Need for Evaluation Provides an organized way of assessing work in progress and results obtained Assesses the impact of an activity on the area targeted for improvement Identifies strengths and weaknesses in the implementation of the improvement activity Keep in mind that it is important to develop a comprehensive plan for improvement, but it is equally important to assess the effectiveness of implementing that plan and the impact of the improvement efforts thru your improvement activities. The purposes for conducting evaluation are numerous and can vary depending on the needs of the agency. Thinking critically about evaluation is the key to ensuring that the agency will obtain information that accurately reflects the progress and impact of the improvement activity. Evaluation can help the agency team to judge whether the identified improvement activity has been successful in strengthening the designated area for improvement. The evaluation can: Provide an organized way of assessing work in progress and results obtained. Assess the impact of an activity on the area targeted for improvement-i.e., specific indicators. Identify strengths and weaknesses in the implementation of the improvement activity.

Types of Evaluation Process evaluation Evaluates the improvement process itself Impact evaluation Evaluates the results produced by the process There are two types of evaluation that can and should be used to evaluate improvement activities : 1.Evaluating the improvement process itself (Process Evaluation) 2.Evaluating the results of the improvement process (Impact Evaluation)

Process Evaluation Questions Process evaluation questions might answer To what extent is the improvement activity being implemented as intended? To what extent is the improvement activity reaching the target audience (i.e., children, staff, parents)? Is everyone doing what they said they would do? Are resources still available to adequately support this improvement activity? Some questions can be answered through a process evaluation include: To what extent is the improvement activity being implemented as intended? To what extent is the improvement activity reaching the target audience (i.e., children, staff, parents)? Is everyone doing what they said they would do?

Impact Evaluation Questions Impact evaluation questions might include Did the improvement activity accomplish what it was supposed to? Which parts of the improvement activity worked well? Which parts of the improvement activity did not work well? Should the agency continue the improvement activity? What has changed as a result of implementing the improvement activity? Some questions to Consider when conducting an impact evaluation of an improvement activity: Did the improvement activity accomplish what it was supposed to? Which parts of the improvement activity worked well? Which parts of the improvement activity did not work well? Should the agency continue the improvement activity? What has changed as a result of implementing the improvement activity?

Reviewing Improvement Activities Improvement activities are aligned to the indicators, including if reflected across related indicators. Improvement activities reflect state priorities. Improvement activities are actionable. Improvement activities are realistic. So before thinking about the entire evaluation plan you may want to assess what it is you already know about your improvement activities. Improvement activities are aligned to the indicators-including if reflected across related indicators. Improvement activities reflect state priorities. Improvement activities are actionable. Improvement activities are realistic. Improvement activities include measures of performance. Improvement activities include time lines. Improvement activities identify responsibility for implementation. Improvement activities include technical assistance needs.

Reviewing Improvement Activities Improvement activities include measures of performance. Improvement activities include timelines. Improvement activities identify responsibility for implementation. Improvement activities include technical assistance needs. So before thinking about the entire evaluation plan you may want to assess what it is you already know about your improvement activities. Improvement activities are aligned to the indicators-including if reflected across related indicators. Improvement activities reflect state priorities. Improvement activities are actionable. Improvement activities are realistic. Improvement activities include measures of performance. Improvement activities include time lines. Improvement activities identify responsibility for implementation. Improvement activities include technical assistance needs.

Categorizing Improvement Activities Training and Professional Development Improve Data Collection Improve Systems Administration and Monitoring Improve Collaboration and Coordination Program Development Clarify/Examine/Develop Policies & Procedures Provide Technical Assistance Increase/Adjust FTE Evaluation Another way of thinking about how to assess the effectiveness of your improvement activities is to try to categorize them prior to evaluating them. This may help in looking overall at your improvement activities. In most instances, improvement activities can be categorized in the following ways: Training and Professional Development Improve Data Collection Improve Systems Administration and Monitoring Improve Collaboration and Coordination Program Development Clarify/Examine/Develop Policies & Procedures Provide Technical Assistance Increase/Adjust FTE Evaluation Because it may too daunting of a task to try to evaluate every single improvement activity included in your SPP/APR, you may want to focus your evaluation on one or two of the categories that seem to be most prevalent. For example, you may want to start out by looking system wide at training and PD and Program Development. Try to determine which category contains the most improvement activities and begin with that.

Developing a Plan for Evaluation Identify the goal of the evaluation. Frame the evaluation questions to be answered. Identify evaluation methods and measurement options. Identify data sources. Determine data analysis techniques. Establish timelines. There are several steps that needs to completed in order to develop an evaluation plan for improvement efforts. These steps include: Identify the goal of the evaluation. Frame the evaluation questions to be answered. Identify evaluation methods and measurement options. Identify data sources. Determine data analysis techniques. Establish timelines. I will discuss each step in more depth on the following slides.

What’s Next? Moving beyond data quality Understanding and manipulating the data you have Using data to make program improvements

Looking at Data: To better understand issues and areas of concern to focus improvement activities... What does the data analysis say? What are the ‘root causes’ of issues or challenges? Early Childhood Outcomes Center

Using data for improvement Evidence Inference Action Early Childhood Outcomes Center

Evidence Evidence refers to the numbers, such as “89% of families reported ...” The numbers are not debatable Early Childhood Outcomes Center

Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News we can’t interpret? To reach an inference, sometimes we analyze data in other ways (ask for more evidence) Early Childhood Outcomes Center

Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders Early Childhood Outcomes Center

At the state level – TA, policy Program improvement: At the state level – TA, policy At the regional or local level – supervision, guidance At the service/classroom level– implement high quality individualized family centered services Different program improvement levers at different levels. Going to be focusing primarily on the state level use of information. Some state applications translate directly to smaller units. How interventionists or teacher use outcome data for program improvement is a completely different topic – very important but we are not going to cover it here. Early Childhood Outcomes Center

Key points Evidence refers to the numbers and the numbers by themselves are meaningless Inference is attached by those who read (interpret) the numbers You have the opportunity and obligation to attach meaning You cannot prevent the misuse of data but you can set up conditions to make it less likely. Early Childhood Outcomes Center

Continuous Program Improvement Reflect Are we where we want to be? Check (Collect and analyze data) Plan (vision) Program characteristics Child and family outcomes Implement Early Childhood Outcomes Center

Tweaking the System Reflect Check Plan (vision) Implement Is there a problem? Reflect Are we where we want to be? Why is it happening? Is it working? What should be done? Check (Collect and analyze data) Plan (vision) Program characteristics Child and family outcomes Implement Is it being done? Early Childhood Outcomes Center

Outcome questions for program improvement, e.g. Do outcomes vary by Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received? Early Childhood Outcomes Center

Looking at Family Outcomes by Subgroups System Characteristics Family Characteristics Child Characteristics Service Characteristics Early Childhood Outcomes Center

Are there differences in outcomes across family characteristics? Race/ethnicity Family Income Primary language Family structure Etc Early Childhood Outcomes Center

Are there differences in outcomes across child characteristics? Race/ethnicity Type of disability Length of time in services Etc Early Childhood Outcomes Center

Examples of process questions Are ALL services high quality? Are ALL children and families receiving ALL the services they should in a timely manner? Are ALL families being supported in being involved in their child’s program? What are the barriers to high quality services? Early Childhood Outcomes Center

Working Assumptions There are some high quality services and programs being provided across the state. There are some families who are not getting the highest quality services. If we can find ways to improve those services/programs, these families will experience better outcomes. Early Childhood Outcomes Center

Action Given the inference from the numbers, what should be done? Develop improvement activities that are: Targeted based on data analysis Based on evidence based practices Interconnected, work together to accomplish the desired result Early Childhood Outcomes Center

Small Group Scenarios Early Childhood Outcomes Center

Small Group Scenarios Each table is assigned a scenario Choose a table / scenario As a group, walk through the scenario. Discuss and answer questions. Jot down your ideas and be prepared to share back some highlights. Early Childhood Outcomes Center