1. National Picture – Child Outcomes for Early Intervention and Preschool Special Education Kathleen Hebbeler Abby Winer Cornelia Taylor August 26, 2014

2. On Today’s Call Brief review of the national data Data quality – Completeness of data – State-to-state variation – Change over time 2

3. State Approaches to Measuring Child Outcomes – 2012-13 3 Approach Part C (N=56) Preschool (N=59) COS 7 pt. scale42/56 (75%)37/59 (63%) One tool statewide8/56 (14%)9/59 (15%) Publishers’ online analysis 1/56 (2%)6/59 (10%) Other5/56 (9%)7/59 (12%)

4. Method for Calculating National Estimates Weighted average of states that met minimum quality criteria Minimum Quality Criteria: – Reporting data on enough children Part C – 28% or more of exiters Part B Preschool – 12% or more of child count – Within expected patterns in the data category ‘a’ not greater than 10% category ‘e’ not greater than 65% 4

5. Number of States that Met Criteria for Inclusion in the National Analysis 5 08-0909-1010-1111-1212-13 Part C1929393341 Part B Preschool1533363941

6. Part C: Reason States Were Excluded from Analyses (out of 51) 6 Reason Part C state was excluded 2010-112011-122012-13 State is sampling 321 No outcomes data reported 000 Reported outcomes data on less than 28% of reported exiters 364 Had at least one outcome with category a greater than 10% or category e greater than 65% 453 Reported outcomes data on less than 28% of reported exiters AND Had at least one outcome with category a greater than 10% or category e greater than 65% 142 Questionable data quality based on review of SPP/APR and knowledge gained through TA 100 States included in the analysis393341

7. Part B Preschool: Reason States Were Excluded from Analyses (out of 51) 7

8. 8

9. 9

10. 10

11. 11

12. What We See Continuing to see consistency over time Increasing number of states who meet minimum quality criteria for national analysis Increasing number of children in the child outcomes data 12

13. Current Emphasis of State Requests Data Quality – Increasing the number of children/families in the data – Pattern checking to identify data quality issues – Training, guidance, supervision, etc. Using Data for Program Improvement – SSIP data analysis Identifying trends in the data Identifying areas of low and high performance Identifying meaningful differences 13

14. Part C: Percent of States by Completeness of Child Outcomes Data* * Completeness = (total with outcomes data/total exiters) 14

15. Part B Preschool: Percent of States by Completeness of Child Outcomes Data* * Completeness = (total with outcomes data/child count) 15

16. State Level Variation and Patterns 16

17. Part C: Positive Social Emotional Skills Progress Category ‘b’ – All States 17 National: 21%

18. 18 National: 66% Part C: State Variation: Greater than Expected Growth – Social Emotional, 2012-2013, All States

19. Part B Preschool: Knowledge and Skills Progress Category ‘b’ – All States 19 National: 15%

20. 20 National: 53% Part B Preschool: State Variation: Exited within Age Expectations – Knowledge and Skills, 2012- 2013, All States

21. Part C: Exited within Age Expectations by State Percent of Exiters Not Eligible for Part B* 2012-13 - All States 21

22. Part C: Average Percentage Who Exited within Age Expectations by State Percent Served*, 2012-13 – All States 22 *http://therightidea.tadnet.org/assets/2514

23. Part C: Average Percentage Who Exited within Age Expectations by ITCA Eligibility Category, 2012-13, All States 23

24. Part B Preschool: Average Percentage Who Exited within Age Expectations by State Percent Served, 2012-13, All States 24

25. Variation Over Time 25

26. What Types of Change are Important? 26 Types of Change2008-09 2012-13 Small variations from year to year are expected Large consistent increases are good news particularly when linked to programmatic changes Large consistent decreases require explanation (e.g. changing population) Large up and down changes are an indicator of questionable data quality and require explanation

27. Part C: Longitudinal Patterns All States Outcome 1 Summary Statement 1 27

28. Part C: Longitudinal Patterns All States, Last 3 Years, Outcome 1 Summary Statement 1 28

29. Part C: Longitudinal Patterns States Included in National Estimate, Last 3 Years Outcome 1 Summary Statement 1 (30 states) 29

30. Part B 619: Longitudinal Patterns All States Outcome 1 Summary Statement 1 30

31. Part B 619: Longitudinal Patterns All States, Last 3 Years, Outcome 1 Summary Statement 1 31

32. Part B 619: Longitudinal Patterns States Included in National Estimate, Last 3 Years Outcome 1 Summary Statement 1 (25 states) 32

33. Part C: Statistically Significant* Change between 2011-12 and 2012-13 : 41 States That Met the Criteria Statistically Significant ChangeOC1-SS1OC2-SS1OC3-SS1OC1-SS2OC2-SS2OC3-SS2 Negative 433867 None 32 332827 Positive 565587 33 *p<.01

34. Part B Preschool: Statistically Significant* Change between 2011-12 and 2012-13: 39 States** That Met the Criteria Statistically Significant ChangeOC1-SS1OC2-SS1OC3-SS1OC1-SS2OC2-SS2OC3-SS2 Negative 644426 None 262430293228 Positive 7115655 34 ** Two states excluded because there was insufficient data from 2011-12 to make the comparison *p<.01

35. Conclusions The data continue to be used by the federal government to justify funding. Results Driven Accountability is shining a spotlight on each state’s child outcomes data. States can expect more scrutiny around data quality. – The criteria used for the national analyses do not set a high bar for data quality. 35

36. How We Can Help! State data quality profiles for FFY 2012-13 were sent out – Email Abby Winer with questions: abby.winer@sri.comabby.winer@sri.com Contact us for help with data quality analysis and quality assurance activities Contact us for help with program improvement planning and data analysis 36

37. State Child Outcomes Data Quality Profiles FFY 2012-13 38

38. Updated National Graphing Template 38 http://www.ectacenter.org/eco/pages/summary.asp#summarygraph

39. Data Quality: Pattern Checking http://www.ectacenter.org/eco/assets/pdfs/Pattern_Checking_Table.pdf Training materials on looking at data: http://www.ectacenter.org/eco/assets/ppt/LookingAtData_revised.ppt http://www.ectacenter.org/eco/assets/ppt/LookingAtData_revised.ppt 40

40. Other Resources Additional data quality resources – http://www.ectacenter.org/eco/pages/quality_assurance.asp http://www.ectacenter.org/eco/pages/quality_assurance.asp Data analysis for program improvement http://www.ectacenter.org/eco/pages/usingdata.asp http://www.ectacenter.org/eco/pages/usingdata.asp SSIP-related Resources – http://www.ectacenter.org/~calls/2014/ssip/ssip.asp http://www.ectacenter.org/~calls/2014/ssip/ssip.asp 40

41. Reminders & Announcements We hope to see you at the Improving Data, Improving Outcomes Conference! The Outcomes for Children and Family Outcomes flyers will be available at the conference Stay tuned for an upcoming presentation of Family Data: Indicator C4 Highlights 41

42. 42 Find More Resources At: http://www.ectacenter.org/eco