Early Childhood Outcomes Center1 Using Data for Program Improvement Christina Kasprzak, NECTAC/ECO Ann Bailey, NCRRC July 2010

Measuring Child and Family Outcomes Early Childhood Outcomes Center2 Systems Thinking: Systems are made up of interrelated, interconnected components Systems change involves changing the capacity, interrelationships, and interdependencies among parts, levels and stakeholders Desired changes in one part or level of the system must be accompanied by changes in other parts or levels

Measuring Child and Family Outcomes Early Childhood Outcomes Center3 SPP as long term plan for systems change: EI/ECSE are systems of complex, interrelated components with goal of achieving outcomes for children and families Changes to EI/ECSE systems require a combination of improvement activities that are interconnected and support changes to infrastructure that work together to achieve the desired results

Measuring Child and Family Outcomes Early Childhood Outcomes Center4 Developing improvement activities: Effective Policies, Procedures, and MOU Data Systems and Monitoring TA and Professional Development Fiscal Management Evaluation

Measuring Child and Family Outcomes Early Childhood Outcomes Center5 Evaluating SPP/APR Improvement Activities

Measuring Child and Family Outcomes 6 The Need for Evaluation Provides an organized way of assessing work in progress and results obtained Assesses the impact of an activity on the area targeted for improvement Identifies strengths and weaknesses in the implementation of the improvement activity

Measuring Child and Family Outcomes 7 Types of Evaluation Process evaluation Evaluates the improvement process itself Impact evaluation Evaluates the results produced by the process

Measuring Child and Family Outcomes 8 Process Evaluation Questions Process evaluation questions might answer To what extent is the improvement activity being implemented as intended? To what extent is the improvement activity reaching the target audience (i.e., children, staff, parents)? Is everyone doing what they said they would do? Are resources still available to adequately support this improvement activity?

Measuring Child and Family Outcomes 9 Impact Evaluation Questions Impact evaluation questions might include Did the improvement activity accomplish what it was supposed to? Which parts of the improvement activity worked well? Which parts of the improvement activity did not work well? Should the agency continue the improvement activity? What has changed as a result of implementing the improvement activity?

Measuring Child and Family Outcomes 10 Reviewing Improvement Activities Improvement activities are aligned to the indicators, including if reflected across related indicators. Improvement activities reflect state priorities. Improvement activities are actionable. Improvement activities are realistic.

Measuring Child and Family Outcomes 11 Reviewing Improvement Activities Improvement activities include measures of performance. Improvement activities include timelines. Improvement activities identify responsibility for implementation. Improvement activities include technical assistance needs.

Measuring Child and Family Outcomes 12 Categorizing Improvement Activities Training and Professional Development Improve Data Collection Improve Systems Administration and Monitoring Improve Collaboration and Coordination Program Development Clarify/Examine/Develop Policies & Procedures Provide Technical Assistance Increase/Adjust FTE Evaluation

Measuring Child and Family Outcomes 13 Developing a Plan for Evaluation Identify the goal of the evaluation. Frame the evaluation questions to be answered. Identify evaluation methods and measurement options. Identify data sources. Determine data analysis techniques. Establish timelines.

Measuring Child and Family Outcomes What’s Next? Moving beyond data quality Understanding and manipulating the data you have Using data to make program improvements

Measuring Child and Family Outcomes Early Childhood Outcomes Center15 Looking at Data: To better understand issues and areas of concern to focus improvement activities... What does the data analysis say? What are the ‘root causes’ of issues or challenges?

Measuring Child and Family Outcomes Early Childhood Outcomes Center16 Using data for improvement E vidence I nference A ction

Measuring Child and Family Outcomes Early Childhood Outcomes Center17 Evidence Evidence refers to the numbers, such as “89% of families reported...” The numbers are not debatable

Measuring Child and Family Outcomes Early Childhood Outcomes Center18 Inference How do you interpret the #s? What can you conclude from the #s? Does evidence mean good news? Bad news? News we can’t interpret? To reach an inference, sometimes we analyze data in other ways (ask for more evidence)

Measuring Child and Family Outcomes Early Childhood Outcomes Center19 Action Given the inference from the numbers, what should be done? Recommendations or action steps Action can be debatable – and often is Another role for stakeholders

Measuring Child and Family Outcomes Early Childhood Outcomes Center20 Program improvement: At the state level – TA, policy At the regional or local level – supervision, guidance At the service/classroom level– implement high quality individualized family centered services

Measuring Child and Family Outcomes Early Childhood Outcomes Center21 Key points Evidence refers to the numbers and the numbers by themselves are meaningless Inference is attached by those who read (interpret) the numbers You have the opportunity and obligation to attach meaning

Measuring Child and Family Outcomes Early Childhood Outcomes Center22 Continuous Program Improvement Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be?

Measuring Child and Family Outcomes Early Childhood Outcomes Center23 Tweaking the System Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?

Measuring Child and Family Outcomes Early Childhood Outcomes Center24 Outcome questions for program improvement, e.g. Do outcomes vary by Region of the state? Level of functioning at entry? Services received? Age at entry to service? Type of services received?

Measuring Child and Family Outcomes Early Childhood Outcomes Center25 Looking at Family Outcomes by Subgroups System Characteristics Family Characteristics Child Characteristics Service Characteristics

Measuring Child and Family Outcomes Early Childhood Outcomes Center26 Are there differences in outcomes across family characteristics? Race/ethnicity Family Income Primary language Family structure Etc

Measuring Child and Family Outcomes Early Childhood Outcomes Center27 Are there differences in outcomes across child characteristics? Race/ethnicity Type of disability Length of time in services Etc

Measuring Child and Family Outcomes Early Childhood Outcomes Center28 Examples of process questions Are ALL services high quality? Are ALL children and families receiving ALL the services they should in a timely manner? Are ALL families being supported in being involved in their child’s program? What are the barriers to high quality services?

Measuring Child and Family Outcomes Early Childhood Outcomes Center29 Working Assumptions There are some high quality services and programs being provided across the state. There are some families who are not getting the highest quality services. If we can find ways to improve those services/programs, these families will experience better outcomes.

Measuring Child and Family Outcomes Early Childhood Outcomes Center30 Action Given the inference from the numbers, what should be done? Develop improvement activities that are: Targeted based on data analysis Based on evidence based practices Interconnected, work together to accomplish the desired result

Measuring Child and Family Outcomes Early Childhood Outcomes Center31 Small Group Scenarios

Measuring Child and Family Outcomes Early Childhood Outcomes Center32 Small Group Scenarios Each table is assigned a scenario Choose a table / scenario As a group, walk through the scenario. Discuss and answer questions. Jot down your ideas and be prepared to share back some highlights.