1. The Current Status of States' Early Childhood Outcome Measurement Systems Kathy Hebbeler, SRI International Lynne Kahn, FPG Child Dev Inst October 17, 2009 DEC's 25th Annual International Conference on Young Children with Special Needs and Their Families 1

2. We’ve come a long way, BABY! 2

3. In the Beginning- Federal Accountability 1994 Government Performance and Results Act (GPRA) 2002 Performance Assessment Rating Tool (PART) –Both Part C and Part B Preschool categorized as “Results Not Demonstrated” due to lack of outcome data. –OMB recommended OSEP develop a strategy for collecting outcome data

4. The Development of Outcome Statements 2004-early 2005: ECO generates discussion and gathers input on child and family outcomes Summer 2005: OSEP announces the child and family outcomes States must report on through their SPP/APRs Early Childhood Outcomes Center 4

5. Three Child Outcomes Percent of children who demonstrate improved: –Positive social emotional skills (including positive social relationships) –Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) –Use of appropriate behaviors to meet their needs Early Childhood Outcomes Center 5

6. Key Concepts to Communicate Outcomes are functional– term has multiple definitions Each outcome is integrated across domains Challenge: Reporting language does not clearly define the outcomes Early Childhood Outcomes Center 6

7. Reporting Child Progress December 2005: States submit plans via SPP on how they will collect outcome data September 2006: OSEP finalizes the child outcome reporting categories Early Childhood Outcomes Center 7

8. Child Progress- the % of children who a. did not improve functioning b. improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers c. improved functioning to a level nearer to same-aged peers but did not reach it d. improved functioning to reach a level comparable to same-aged peers e. maintained functioning at a level comparable to same- aged peers Early Childhood Outcomes Center 8

9. Key Concepts to Communicate Progress categories require 2 data points for each child, are based on growth trajectories, compare a child to him or herself over time, and also compare each child to age expectations Early Childhood Outcomes Center 9

10. 10 EntryExit

11. States Submit Data to OSEP in their Performance Reports Feb 2007: “status at entry” data Feb 2008: child progress data for children who exited 7/1/06 through 6/30/07 Feb 2009: child progress data for children who exited 7/1/07 through 6/30/08 Early Childhood Outcomes Center 11

12. State Approaches to Measuring Child Outcomes Approach Part C (56 states/jur) Preschool (59 states/jur) One tool statewide7/56 (13%)9/59 (15%) Publishers’ online analysis 3/56 (5%)6/59 (10%) COSF 7 pt. scale41/56 (73%)38/59 (64%) Other5/56 (9%)7/59 (10%)

15. Number of Children Included in Feb ‘09 SPP/APR Data Part C (56) Range: 5-6452 <30 = 3 30-99= 10 100-499= 25 500-999= 6 1000+ = 12 Preschool (59) Range: 3-10157 <30 = 3 30-99 = 9 100-499= 14 500-999= 10 1000+ = 22

18. Key Concepts to Communicate Meaningfulness of the progress categories Establishing expectations for how the percentages “should” look Exploring relationships between state eligibility definitions and expectations for outcomes Early Childhood Outcomes Center 18

19. Setting Targets for Improving Child Outcomes 2008 and 2009: State input into summary statements to be the basis of target setting Currently: summary statements have been out for public comment, and are in draft form in SPP Measurement tables February 2010: Baseline data and target setting on summary statements

20. Summary Statement Data Required Summary Statement 1: Of those children who entered or exited the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program. Required Summary Statement 2: The percent of children who were functioning within age expectations in each Outcome by the time they exited the program. 20

21. Key Concepts to Communicate The meaning of the summary statements Ensuring that data is clean before making program changes based on it Choosing/ developing strategies that will improve child outcomes Early Childhood Outcomes Center 21

22. So where are the states now?

23. States and Outcomes Measurement Many states have embraced the ideas of measuring outcomes –Want information for own purposes –Program improvement The power of GOOD outcomes data is widely recognized

24. States and Outcomes Measurement States have invested resources and time in building an outcomes measurement system. –Involving stakeholders –Identifying an approach –Training providers in approach –Building a data system And collecting data!

25. States have made incredible progress States started at various times Have varying amounts of data Examining their data (and procedures) –The Quality Question?? This is VERY DIFFICULT work

26. Some of the Challenges Need for good assessment tools –Need for more providers skilled in assessment Need strong grounding in typical child development Adequate training statewide in data collection procedures –Understanding and commitment Resources for new data systems Capability to analyze, report and interpret data –Generating reports for local programs Resources in general

27. Next steps: Putting it all together

28. Early Childhood Outcomes Center 28 Making Good Use of Data: Some Necessary Ingredients Set of good questions Data set with the required elements Analytic capability to analyze the data to address the question Commitment to use the information for program improvement

29. Examining and Tweaking the Service System Plan (vision) Program characteristics Child and family outcomes Implement Check (Collect and analyze data) Reflect Are we where we want to be? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?

30. Early Childhood Outcomes Center 30 Information Infrastructure: Data Needed for Program Improvement WHOSERVICES COSTQUALITY OUTCOMES

31. Early Childhood Outcomes Center 31 Where is your state (or program) now? WHOSERVICES OUTCOMES What do you have? How much is linked?

32. WHO SERVICES COST QUALITY OUTCOMES WHO SERVICES COST QUALITY OUTCOMES WHO SERVICES COST QUALITY OUTCOMES Preschool Kindergarten Grade 1 Grade 2 WHO SERVICES COST QUALITY OUTCOMES Tracking outcomes over time

33. An outcomes measurement system is more than hardware, software, and data

34. Framework Development ECO is working with 7 partner states over the next two years to develop a framework for outcomes measurement. Framework provides the basis for a self assessment to allow states to track their own progress in building an outcomes measurement system.

35. Areas of DRAFT Outcomes Measurement System Framework Data Collection and Transmission Analysis Information Sharing Action Vision for What Is to Be Learned Evaluation

36. Data Collection and Transmission State has a method capable of accurately capturing outcomes information State has effective procedures for professional development around data collection. State has effective procedures for moving data from the local to the state level Etc.

37. State Self Assessment Is each area of the outcomes measurement system implemented? Is the area implemented in a way that reflects high quality?

38. The power of data The widespread availability of quality data on child and family outcomes has the potential to radically change service delivery. –If the data are high quality –If the data are used for improving services

39. We’ve come a long way, BABY! 39

40. …and we still have a long way to go…...but won’t it be great when we get there?

41. For more information (and for a copy of this presentation) www.the-eco-center.org