1. Presented at Division for Early Childhood National Harbor, Maryland November, 20111 Child Outcomes: What We Are Learning from National, State, and Local Data Kathy Hebbeler ECO at SRI International Lynne Kahn ECO at FPG Child Development Institute

2. Topics Why outcomes data are collected? State approaches National data Use of the data at state and local levels 2 Early Childhood Outcomes Center

3. Let’s get acquainted 3 Early Childhood Outcomes Center

4. Why does the federal government want data on child outcomes? Government Performance and Results Act (GPRA) Program Assessment Rating Tool (PART) Individuals with Disabilities Education Act (IDEA) 4

5. Requires goals and indicators be established for federal programs, including IDEA Indicators and data collection for school age population included data on outcomes Previously, for early childhood data had been reported on: Number of children served (Part C) Settings (both Part C and 619 Government Performance and Results Act (GPRA) passed in 1993

6. OSEP: PART evaluation results (2002) 130 programs examined in 2002; 50% programs had no performance data Programs looking at inputs, not results Part C and Section 619  No child outcome data - “results not demonstrated”  Department of Education needs to develop a strategy to collect annual performance data in a timely manner 6 Early Childhood Outcomes Center

7. 9 SEC. 616. > MONITORING, TECHNICAL ASSISTANCE, AND ENFORCEMENT. ``(a) Federal and State Monitoring.….. (2) Focused monitoring.--The primary focus of Federal and State monitoring activities described in paragraph (1) shall be on-- (A) improving educational results and functional outcomes for all children with disabilities; IDEA 2004 The word “results” appears 65 times in the legislation.

8. Early Childhood Outcomes Center8 $373,351,000 $438,548,000

9. State and Local Uses Accountability –Justifying the investment in EI and ECSE Program Improvement –Using data to identify program strengths and share them –Using data to identify program weaknesses and address them 9 Early Childhood Outcomes Center

10. Reporting Requirement for Early Intervention and Preschool Special Education 10 Early Childhood Outcomes Center

11. 11 OSEP Reporting Requirements: Child Outcomes –Positive social emotional skills (including positive social relationships) –Acquisition and use of knowledge and skills (including early language/ communication and early literacy) –Use of appropriate behaviors to meet their needs

12. What States Report: OSEP Reporting Categories Percentage of children who: a.Did not improve functioning b.Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c.Improved functioning to a level nearer to same- aged peers but did not reach it d.Improved functioning to reach a level comparable to same-aged peers e.Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers 12

13. Early Childhood Outcomes Center13

14. Reporting details Progress for all children who exited between July 1, 2009 and June 30, 2010 Stayed in the program at least 6 months Data reported to OSEP in February of each year. Progress data first submitted in 2008 14 Early Childhood Outcomes Center

15. The Summary Statements 1.Of those children who entered the program below age expectations in each outcome, the percent who substantially increased their rate of growth by the time they turned 3 [6] years of age or exited the program. 2.The percent of children who were functioning within age expectations in each outcome by the time they turned 3 [6] years of age or exited the program. 15 Early Childhood Outcomes Center

16. Formula for SS 1 (c+d)/(a+b+c+d) 16

17. Formula for SS 2 (d+e)/(a+b+c+d+e) 17

18. Timeline 18 Early Childhood Outcomes Center WhenCritical Event January 2004 – January 2005Stakeholder input gathered on 3 child outcomes July 2005 (revised September 2006)OSEP releases reporting requirements for state programs February 2008States submit first data on 5 progress categories: Children who exited between July 1, 2006 and June 30, 2007 February 2010States establish baseline and set targets on the Summary Statements for first time. February 2011States submit data on 5 categories for the 4 th time.

19. State Approaches to Measuring Child Outcomes Approach Part C (56 states/jur) Preschool (59 states/jur) One tool statewide7/56 (13%)9/59 (15%) Publishers’ online analysis 3/56 (5%)6/59 (10%) COS 7 pt. scale41/56 (73%)38/59 (64%) Other5/56 (9%)7/59 (10%)

20. …and now there are national data 20 Early Childhood Outcomes Center

21. …one day we get a request From the U.S. Department of Education Include the child outcomes data as a GPRA indicator? Also, in President’s budget justification for Part C and Preschool 619 funding? Initial request received in 2010, repeated in 2011. 21 Early Childhood Outcomes Center

22. What would be the impact of state variation in data quality on the national numbers? The Dilemma Variations in quality of state data –Some states started earlier –Some states had devoted more attention to improving quality 22 Early Childhood Outcomes Center

23. Our Response: Compute the analyses several ways Identify the states with the highest quality data and use only their data. Stratify by number of children served and weight data to produce national estimate. 23 Early Childhood Outcomes Center Use data from all states. Weight data to represent the nation. Weighting necessary because a few states are sampling. Also, many states not reporting data on all children.

24. Early Childhood Outcomes Center24 Note: Based on 29 States with highest quality data

25. Early Childhood Outcomes Center25 Note: Based on 29 States with highest quality data

26. Early Childhood Outcomes Center26 Note: Based on 33 States with highest quality data

27. Early Childhood Outcomes Center27 Note: Based on 33 States with highest quality data

28. Can we trust these data? 28 Early Childhood Outcomes Center

29. Pattern checking for validity Checking across years –How do the 2009-10 data compare to the data for 2008-09? Checking across methods –How do the data for all states compare to states with highest quality data? 29 Early Childhood Outcomes Center

30. Part C: Social Relationships

32. Part B Preschool: Knowledge and Skills

34. Possible interpretation of the data Nationally, a high proportion of children who receive Part C and ECSE services are showing greater than expected progress Nationally, many (over half) are exiting the program functioning like same age peers in at least one of the outcomes. 34 Early Childhood Outcomes Center

35. Would you agree? 35 Early Childhood Outcomes Center

36. Should each state’s data look like the national data? Probably not More important that each state continue to focus on the quality of its own data –Getting outcomes data on all children who exit –Working with programs whose data look unusual to address possible data quality issues 36 Early Childhood Outcomes Center

37. For more information For updates to the framework and the self- assessment and resources to support the quality indicators: www.the-eco-center.org 37 Early Childhood Outcomes Center

38. 38 Early Childhood Outcomes Center

39. Additional Information Early Childhood Outcomes Center39

40. Criteria for States with Quality Data 1.Low percentage of missing data 2.No odd patterns in “a” or “e” categories 3.Did not use questionable data collection methods 40 Early Childhood Outcomes Center

41. Calculating Missing Data for Part C Proxy for missing data = Number with data for C3/ Exiting Data (618) 41 Early Childhood Outcomes Center Do not expect this number to be 100%..but we don’t expect it to be 10% either

42. Part C: Percent of Exiters included in Outcomes Data 08-09 <10% = 10* 10- 20% = 4 20- 30% = 8 30- 40% = 11 40- 50% = 8 50- 60% = 8 60- 70% = 4 70- 80% = 2 >80% = 1 09-10 <10% = 5* 10- 20% = 4 20- 30% = 6 30- 40% = 8 40- 50% = 5 50- 60% = 11 60- 70% = 9 70- 80% = 1 >80% = 0 *3 states are sampling for Part C. Cut off was > 27%.

43. Calculating Missing Data for 619 Proxy for missing data = Number with data for B7/ Child count 43 Early Childhood Outcomes Center Do not expect this number to be 100%..but we don’t expect it to be 10% either

44. Odd Patterns in a or e a = % of children who show no new skills –Except this to be very small. e = % of children who maintained functioning comparable to age expectations. –Don’t expect this to be large. Quality defined as <10% in a and <65% in e. 44 Early Childhood Outcomes Center

45. 45

46. Percent of Child Count included in Outcomes Data for ECSE 08-09 <10= 11* 10- 20%= 15 20- 30%= 12 30- 40%= 12 40-50% =1 >50% = 2 09-10 <10= 6* 10- 20%= 11 20- 30%= 12 30- 40%= 16 40-50% =4 >50%= 0 *4 States are sampling for 619 Cutoff was > 11%.