1. A National Picture: Child Outcomes for FFY2013-14
Abby Winer
Cornelia Taylor
Welcome to this year’s presentation of the national data for child outcomes for both Part C and Part B 619 preschool.
September 16, 2014
NECTAC/ECO/WRRC 2012

2. On Today’s Call
Learn about the latest national child outcomes data data
Examine data quality patterns in the national data
Completeness of data
State-to-state variation
Change over time
Today we’re going to present the overall outcomes data and then we’re going to dig in to some patterns; patterns that we see across states, patterns that we see across characteristics of states, and patterns within states.
NECTAC/ECO/WRRC 2012

3. State Approaches to Measuring Child Outcomes – FFY 2013-14
Part C
(N=56)
Preschool
(N=59)
COS 7 pt. scale
42/56 (75%)
37/59 (63%)
One tool statewide
8/56 (14%)
9/59 (15%)
Publishers’ online analysis
1/56 (2%)
6/59 (10%)
Other
5/56 (9%)
7/59 (12%)
First, we want to orient everyone to the point that states use multiple approaches to present this child outcomes data. This table shows the percentage of states for Part C and Part B preschools that use each of these four methods. You can see that the majority of states for both Part C and Part B use the Child Outcomes Summary Process to report on the child outcomes. And for both C and 619, about 15% of states use one tool statewide such as the Battelle Developmental Inventory (BDI). For Part C we have one state that uses the publishers online analysis, for example the AEPSI or Teaching Strategies Gold, and for Preschool we see 6 states that use the publisher’s online analysis. For both Part C and preschool we have states that use a variety of other methods to report on these data.

4. State Approaches: Part C
This is a map that shows the same information that was just presented for Part C. Most of the states are green because that represents the Child Outcomes Summary Process. We have a few states that use one tool statewide, some states that use another method, and one that uses the publisher’s on-line system.

5. State Approaches: Part B Preschool
For Part B preschool you’ll that we have fewer states that use the Child Outcomes Summary Process, it’s still the majority, but there’s more variance in the methods that are used and many more states using the publishers online system.

6. Method for Calculating National Estimates & Critieria
Weighted average of states that met minimum quality criteria
Minimum quality criteria for inclusion in national analysis:
Reporting data on enough children
Part C – 28% or more of exiters
Part B Preschool – 12% or more of child count
Within expected patterns in the data
category ‘a’ not greater than 10%
category ‘e’ not greater than 65%
Next, we’re going to present the national estimates for child outcomes for this year. We’re going to re-orient people to how we compute these national estimates. We take a weighted average of the states that meet a set of minimum quality criteria. When we say a weighted average, we’re taking the child count for each state included in the analysis and looking back to determine the amount of influence that is has on the average. So the big states have more influence on the average and the smaller states have less of an influence on the average. The minimum quality criteria for inclusion in the national analysis has not changed since we started doing the national analysis. Part of the reason we haven’t changed it is that we want to be able to track how states are progressing on these original set of quality indicators that we developed. We use two criteria, the first is the number of children included in the outcomes data compared to the number of children that should be included. For Part C, we include states that have 28% of the children exiting or more. For Part B Preschool we include states that have 12% or more of the children that were represented in their child count. The second feature we look at is the patterns in the progress category data. We include states where progress category A is not greater than 10% and we include states where progress category E is not greater than 65%. The rationale for category A not being greater than 10% is that it is intended to measure children who enter the program and do not progress at all, do not gain any skills and perhaps may be losing skills during their participation in the program and we think that’s a small percentage of the state population. For progress category E, those are children that enter and exit at age expectations in the outcomes. For our review of national data and what we know about the children that are served in these programs, we don’t expect that more than 65% of children should be entering and exiting at age expectation.

7. Number of States that Met Criteria for Inclusion in the National Analysis
We’ve been tracking the number of states that are included in the national analysis since fiscal year On the left hand side you see the number of states included in the analysis which runs from 0-51, the blue line is Part B preschool and the orange line is Part C. The table below shows each of the reporting years, from fiscal year ‘08 to fiscal year ‘13. For both Part B and Part C the number of states in the national analysis has been increasing. We had a small dip in 2011 because a lot of states were changing methodologies.

8. Part C: Reason States Were Excluded from Analyses (out of 51)
Reason Part C State Was Excluded
State is sampling 2 1
Missing data
(less than 28% of reported exiters) 6 4
‘a’ and ‘e’ patterning
(Had at least one outcome with category a greater than 10% or category e greater than 65%) 5 3
AND
Patterning
States included in the analysis 33 41 47
This table shows details about the reasons why states were excluded from the analysis. We don’t include states that are sampling because we can’t look at their completeness in the same way that we can for states that are doing a population count. For Part C, 1 state was excluded from the national analysis because they were sampling, no states were excluded solely because of missing date, 1 state was excluded because of their progress category “a” and “e” patterning, and 2 states were excluded because of both missing data and patterning. What you can see is a trend of decreasing for the missing data category, decreasing for the “a” and “e” category, and we’re at the same number of states that we were at last year for the missing data and patterning category.

9. Part B Preschool: Reason States Were Excluded from Analyses (out of 51)
Reason Part B 619 State Was Excluded
State is sampling 2 3
No progress category data reported 1
No child count data available
Missing Data
(Reported outcomes data on less than 12% of child count) 4
‘a’ and ‘e’ patterning
(Had at least one outcome with category a greater than 10% or category e greater than 65%)
AND
States included in the analysis 39 41 43
This is the exact same table for Part B preschool. We see 3 states that are sampling, 1 state were we didn’t have child count data so couldn’t estimate their completeness, 1 state had less than 12% of their child count, 2 states were excluded for patterning and 1 state was excluded for both missing data and patterning.

10. Part C: Greater than Expected Growth
These are the national numbers. What you see along the y-axis is the percent of children in the nation exiting with greater than expected growth (summary statement 1). Each of these clusters of bars is one of the outcomes. The first cluster of bars is for outcome A – social relationships, the second cluster of bars is for Outcome B – knowledge and skills, and the third cluster of bars is for outcome C – action to meet needs. When we look within each cluster we see a bar for fiscal year 2008, fiscal year 2009, fiscal year 2010, all the way to fiscal year 2013, our most current year of data. This is for Part C. When we look at the patterns of the national estimates for the percent of students making greater than expected growth in these outcomes we see a decreasing trend. In 2008, in social relationships, the national estimate was that 70% of children made greater than expected growth in social relationships during their participation in the program. In 2013, we’re seeing that 66% are making greater than expected growth in their social relationships.

11. Part C: Exited within Age Expectations
Now we’re going to look at the same chart for the percent of children that exit within age expectations for Part C. While the previous chart was for Part C summary statement 1, we’re now looking at Part C summary statement 2. Again, the axis is the percentage from the national estimate, the clusters are the outcomes, and the bar represents the fiscal year. We see more stability in summary statement 2 and that is partly because there are more children included in summary statement 2. The calculation of summary statement 1 excludes children that entered and exited at age expectation, so the total number of children included in summary statement 1 is lower. You can see that in social relationships the pattern is very flat, in knowledge and skills there’s a slight decrease as well as in action to meet needs.

12. Part B Preschool: Greater than Expected Growth
This is the same data, but for Part B Preschool. This is for greater than expected growth, or summary statement 1, and again each cluster of bars is outcome A, B and C. For Social Relationships, we see a decreasing trend from 83% in fiscal year 2008 to 77% in fiscal year We also see a decreasing trend for knowledge and skills, as well as a decreasing trend for action to meet needs.

13. Part B Preschool: Exited within Age Expectations
Here we are looking at the same chart, this is Part B Preschool summary statement 2, those children that exited the program within age expectations. Similar to what we saw for Part C, we see more stability in summary statement 2 than we saw in summary statement 1. There is, again, a decreasing trend for summary statement 1 but it’s very slight. We’ve actually seen a increase in summary statement 2, although it is just by 1 percentage point. There’s a slight decrease in action to meet needs.

14. What We See Continuing to see consistency over time
Increasing number of states who meet minimum quality criteria for national analysis
Increasing number of children in the child outcomes data
To sum up what we see from this national analysis is that we’re continuing to see consistency across time. We think that’s good because if we’re seeing a lot of bouncing around that would make us question the quality of the data. We’re seeing an increasing number of states who meet the minimum quality criteria for the national analysis and an increasing number of children in the child outcomes data, and by that we mean the percent of children in the child outcomes data. We want to emphasize that the criteria for being included in the national analysis is not sufficient for having high quality data. Data quality is a multi-faceted construct, so there are a lot of characteristics of high quality data and within your state you need to think about your system, think about the data you have available to you, and think about data quality from your state perspective.

15. Current Emphasis of State Requests
Data Quality
Increasing the number of children/families in the data
Pattern checking to identify data quality issues
Training, guidance, supervision, etc.
Using Data for Program Improvement
Identifying trends in the data
Identifying areas of low and high performance
Identifying meaningful differences
Have a child outcomes TA request?
your ECTA state contact or one of us!
We wanted to talk a little bit about the state requests that we’ve been receiving for TA. We continue to receive a lot of state requests around data quality, including how states can increase the number of children in their data, technical support for pattern checking, and also support in training staff in the child outcomes process. In addition, we receive requests around using data for program improvement. Those of you in Part C have definitely been using data as part of the state systemic improvement process. Those of you in Part B preschool are hopefully participating in that process or leading it and using data. We think that has generated more requests around this area, so as we work with states and learn about the great things that people are doing it really shapes our thinking and so please let us know if you need support in either of these areas. You can your ECTA state contact, or you can always contact Abby or Cornelia, whose s are at the end of this PowerPoint.

16. Part C: Percent of States by Completeness of Child Outcomes Data*
We’re going to take another look at the completeness of child outcomes data. In this chart, we’re defining completeness as the total number of children with outcomes data divided by the total number of exiters. The axis is showing the percent of children in each of these three categories. The clusters of bars are fiscal years. The blue bars are the states that had less than 40% of their reported exiters with outcomes data, the orange bar is the number of states that has between 34 to less than 70% of their exiters with outcomes data, the black bar is the number of states that had 70% or more of their exiters. In fiscal year 2009 we had 12 states that had 70% or more of their exiters and this year, in fiscal year 2013, we had almost double: 22 states that had 70% or more of their exiters. We can see that the blue bars, those with less than 40% of their exiters, has been decreasing for the most part and this year has been the lowest out of the last couple of years. We can see that 65% of states are between 34 to less than 70% of exiters in their child outcomes data.
* Completeness = (total with outcomes data/total exiters) 14

17. Part B Preschool: Percent of States by Completeness of Child Outcomes Data
Now we’re going to look at data for Part B preschool and the completeness of the child outcomes data. The axis is the percent of states, the cluster of bars is by fiscal year. At the end, the number of states included varied a little and that’s because we don’t have access to child count for some of the states so we don’t have access to the denominator. The percentage for the blue bar was calculated as the total number of outcomes data divided by the child count. So the blue bar is the percent of states with less than 12% of their child count represented in their outcomes data. The orange bar is the percent of states with 12 to less than 33% of their child count represented in their outcomes data. The black bar is the percent of states with more than 33% of their child count represented in their outcomes data. If we focus on the black bar, you can see that has increased from 27% to 39% of states from fiscal year 2009 to fiscal year There’s been more stability with states that have between 12 and 33% and that continues to be the largest chunk of states.
* Completeness = (total with outcomes data/child count) 15
NECTAC/ECO/WRRC 2012

18. State Level Variation and Patterns
Next we are going to talk about state level variation because we look at the stability of national numbers and we want to make sure that everyone knows that that stability is laid on top of a lot of variability across states and their progress categories.

19. Part C: State Variation- Exited within Age Expectations Kowledge and Skills
National: 52%
Each bar in this graph represents a state. The height of the bar is their value for Summary Statement 2, Outcome B, so for example, in the lowest state, 9% of children exited the program at age expectation in their knowledge and skills and in the highest state, 72% of children exited the program within age expectation for their knowledge and skills. This is for Part C. And for the this outcome and summary statement, the national estimate is 52%. So you can see that there’s states above and below, states far above and far below, so although those numbers have remained fairly stable over time we continue to see a large difference across states.

20. Part B Preschool: State Variation: Exited within Age Expectations – Knowledge and Skills, , All States
National: 52%
This is the same chart for Part B preschool. The first bar represents a state where 10% of children exit the program with age expectations in their knowledge and skills, and the last, highest part represents a state where 74% of children exit the program within age expectations in their knowledge and skills. Again, you can see that the national estimate is 52%. The estimate for both Part B and Part C for this is the same, and the range is also almost identical.

21. Part C: Exited within Age Expectations by State Percent of Exiters Not Eligible for Part B* All States
This next chart is for Part C. We are looking at summary statement 2, exiting within age expectations, and we’re crossing that by the percent of children exiting the program who were identified as not eligible for Part B. You can see down in the fine print that not eligible for Part B includes those children who were no longer eligible for Part C prior to reaching age three, those who were identified as not eligible for Part B and exited with referrals to other programs and those who were not eligible for Part B and were exited with no referrals. The assumption is that the more children exiting your program without referrals to Part B, the higher percentage of children we would expect to be exiting at age expectation. So states that have high percentages of children, which we’ve defined as 30% or greater percent of children exiting the program not eligible for Part B, we would expect to see a higher percent of children exiting at age expectation. For states with relatively lower percent of children exiting who are not eligible for Part B, we would expect a lower summary statement. In this case, the bars are clustered by the outcomes. The first cluster is outcome A, social relationships, the second cluster is outcome B, knowledge and skills, and the third cluster is outcome C, actions to meet needs. This axis is the average percent of children so what we did was we grouped states that had less than 20% of children who were identified as not eligible for Part B at exit, states that had 20 to less than 30% of children who were identified as not eligible for Part B at exit, and states that had 30% or greater percent of children who were identified as not eligible for Part B at exit. For those groups of states we computed the average percentage so for social relationships we can see that there’s a difference between states where less than 20% exit not eligible for Part B and states where more than 20% exit not eligible for Part B, but we don’t see a variation between the orange bar and the black bar. For knowledge and skills we see a nicer stair step pattern where as the percentage of children exiting not eligible for Part B increases so does the value for summary statement 2, those exiting at age expectations in knowledge and skills. With actions to meet needs we see again a strong differentiation between those states where less than 20% are exiting not eligible for Part B, and states where 20% or greater are exiting not eligible for Part B, but it’s difficult to distinguish between the middle and the highest category.
NECTAC/ECO/WRRC 2012

22. Part C: Average Percentage Who Exited within Age Expectations by State Percent Served*, – All States
Here we’re looking at group of states again for Part C. We’re looking at groups of states that had less than 2.5% served, 2.5 to less than 3.9% served, and 3.9 or greater percent served. Again for each of these groups of states we computed the average percent who exited within age expectation (summary statement 2). This is, again, clustered by outcomes. The pattern is the same for all three outcomes. You see that the states who serve a very low percentage of children have fewer children exiting within age expectations than those who serve more than 2.5%. With those two categories we see a relationship between percent served and percent exiting at age expectation and the expected pattern which is the assumption that states that have a lower percent served are serving children with more severe disabilities. We do see that in these patterns.
Note: Percent served is based on a population estimate of the number of children birth to 3 from the Census. *

23. Part C: Average Percentage Who Exited within Age Expectations by ITCA Eligibility Category, , All States
Another way to group states by a variable that is a proxy for the types of severity of disability of the children who are served is to look at the ITCA eligibility category. States select these categories from three different categories, the category that best describes their state eligibility category. We have category C which is restrictive or narrow eligibility, we have category B which is medium eligibility and we have category A which is broad eligibility. The assumption is that states with a broad eligibility would have more children exiting at age expectations than states with a narrow eligibility. Again, this is clustered by outcomes. We can see that there is differentiation between states with a restrictive eligibility and states with a broad eligibility in the expected direction for social relationships but there’s a split when we look from restrictive to medium where the percent of children exiting within age expectation in social relations is actually lower for those medium eligibility states than it is for restrictive states. For knowledge and skills we see a nice stair step pattern across the three eligibility categories in the predicted direction. We also see a stair step for action to meet needs, although the difference between the restrictive and the medium is less distinguished.
Note: •Category A - At Risk, Any Delay, Atypical Development, one standard deviation in one domain, 20% delay in two or more domains, 22% in two or more domains , 25% delay in one or more domains •Category B – medium 25% in two or more domains, 30% delay in one or more domains, 1.3 standard deviations in two domains, 1.5 standard deviations in any domain , 33% delay in one domain. Category C – restrictive – 33% delay in two or more domains, 40% delay in one domain, 50% delay in one domain, 1.5 standard deviations in 2 or more domains, standard deviations in one domain, 2 standard deviations in one domain, 2 standard deviation sin two or more domains 23
NECTAC/ECO/WRRC 2012

24. Part B Preschool: Average Percentage Who Exited within Age Expectations by State Percent Served, , All States (N=50)*
This graph shows, for Part B preschool, the average percentage who exited within age expectations by percent served. We didn’t include one state because we didn’t have their child count which is the numerator for this percent served variable. We can see no differentiation between states that were grouped as having less than 5.7% served, those that were 5.7 to less than 7.5 and those that were 7.5 or greater. We see no distinction across those 3 percent served categories in social relationships. We see a stair step in knowledge and skills in the predicted direction. In action to meet needs we see that those states with the lowest percent served actually have the highest percentage of students exiting within age expectations, which is counter to the predictions.
*child count data was not available for 1 state to compute percent served

25. Variation Over Time
Another way to look at variation is by looking at trends over time.
NECTAC/ECO/WRRC 2012

26. What Types of Change are Important?
Small variations from year to year are expected
Large consistent increases are good news particularly when linked to programmatic changes
Large consistent decreases require explanation (e.g. changing population)
Large up and down changes are an indicator of questionable data quality and require explanation
This should be a familiar illustration for you if you’ve looked at your state child outcome data quality profiles. This a quick summary of the types of change that are generally seen, how to interpret different patterns and change over time.
Small variations from year to year much like we see with the national estimate data aren’t necessarily cause for concern or really give you anything to interpret. But when you see large increase that could be good news, especially when you’re linked to programmatic changes.
Hopefully as you begin implementing changes related to your SSIP you’re going to start to se some of these increase patterns in trends and will be able to directly link them to changes you’ve made. And when you see large consistent decreases, that isn’t necessarily bad, it just warrants explanations and looking deeper into. It could be that your initial estimate when you first gathered data in ’08-’09 the providers weren’t really well trained so maybe they were overestimating children progress and now you’ve done a lot of trainings and they are much more accurate. So you’ve seen a decrease over time. Or maybe you’re collecting data on much more children than you first did and so now as you’ve grown the number of children that are included in your outcomes data you’ve seen a little bit of a decrease over time.
Large up and down changes, instability, is usually related to questionable data quality or some significant change that occurred in your data collection method. For instance, if you changed the tool you were using, from providers getting to choose what tool they wanted to having one tool statewide, that’s going to affect the pattern of results. You also may have had, during that transitional period, much fewer children who had complete data on that tool or if you changed data systems in that year they were only able to report on a small subset of kids so that really changed that states outcome data for that one year period.
It could also be that there just isn’t reliability in your outcomes data due to a lack of inter-rater reliability or consistency in the data collection.
Finally, for states that serve a very small number of children, the data is more prone to fluctuate from year to year due to small n size. Changing population of children served will have a greater impact on a small n size (under 100) than a large state.
NECTAC/ECO/WRRC 2012

27. Part C: Trends Over Time: Greater Than Expected Growth– Social Emotional, 2013-2014, All States
This graph (we like to refer to as a spaghetti plot) shows the trends over time for all Part C states in the percent of children showing greater than expected growth (summary statement 1) in social and emotional outcomes.
Each line is a single state.
As you look across states this gives you a sense of the variation that between states but also over time, which tell us two different things. It tells us about between-state variability and within-state variability.
You can see that there is a lot of between-state variability. For this outcome and summary statement, state range anywhere from 100% down to 30%.
As we look across states you can also see a fair amount of clustering in the 60-80% range.
When we look at within state variability – following the pattern of each line, in general, most states’ lines are pretty flat or have slight fluctuations with a few states who have large fluctuations or inconsistent patterns.

28. Part C: Longitudinal Patterns States Included in National Estimate, Last 3 Years Outcome 1 Summary Statement 1 (31 states)
What happens when we look at just those states who were included in the national analysis last 3 years?
Looking only at states that met the quality criteria for inclusion in the national analysis all 3 of the last 3 years based on the missing data criteria, patterning criteria, and APR/SPP reviews.
We see that there is less variation across states, although there is still some, and more stability within each state. The lines are almost all pretty straight without large increases or decreases. 29
NECTAC/ECO/WRRC 2012

29. Part B Preschool: Trends Over Time: Greater Than Expected Growth– Social Emotional, , All States
For Part B preschool we see a similar patterns over time for ALL states. We see that there’s a good amount of variability but less than what we saw in Part C. 619 states vary from as high as 100% to as low as about 50%.
This related to how the national estimates differ for Part C and 619 – for Part B 619 the national estimate has been a little higher for summary statement 1 in general.
Also looking across states, we see that most states cluster close to the 70-90% range in terms of kids showing greater than expected growth in positive social emotional skills.
Within state variation also shows that generally most states are stable over time.

30. Part B 619: Longitudinal Patterns States Included in National Estimate, Last 3 Years Outcome 1 Summary Statement 1 (34 states)
What happens when we look at just those states who were included in the national analysis last 3 years? As with Part C, for Part B Preschool we see that there is less variation across states (the lines are closer together) and more stability within each state. The lines are almost all pretty straight without large increases or decreases. 32
NECTAC/ECO/WRRC 2012

31. Part C: Statistically Significant
Part C: Statistically Significant* Change between and : All States (N=51)
Statistically Significant Change
OC1-SS1
OC2-SS1
OC3-SS1
OC1-SS2
OC2-SS2
OC3-SS2
Negative 2 9 4 10 7 8
None 44 36 39 34 41
Positive 5 6 3
This table is another way of looking at change within a state or compared to itself year to year.
We use the same test used as meaningful differences calculator but set at a p<.05 or 95% confidence interval which aligns with determinations. The meaningful differences calculator is set at p< .10 or 90% confidence interval. We compute standard error, then compute difference between the two values and divide by the standard error to get a z-score. We record into positive change, negative change, and no change.
If we take a look at significant change and ask, was change meaningful between last year and this year when we look at all states. Most states don’t have significant change.
This goes into that stability in the national estimate that shows the majority of states for Part C didn’t show any change. A little more change for summary statement 2 than for 1, a few states on either end, significant positive or significant negative change. Hopefully the work that’s being implemented through SSIP work will shift things in a more positive direction
*p<.05
NECTAC/ECO/WRRC 2012

32. Part B Preschool: Statistically Significant
Part B Preschool: Statistically Significant* Change between and : All States (N=51)
Statistically Significant Change
OC1-SS1
OC2-SS1
OC3-SS1
OC1-SS2
OC2-SS2
OC3-SS2
Negative 9 7 14 11 13
None 31 34 35 30 32 33
Positive 10 8 6 4
*p<.05
This is the same table for Part B preschool comparing the change between last year estimates and this year estimates in summary statements for all states. The majority of states show no change, there’s stability. But we see more states showing significant positive change specifically for knowledge and skills for greater than expected growth. Again hopefully be seeing a trend showing that more states are moving towards more significant positive change as result of the SSIP work that’s being done.
NECTAC/ECO/WRRC 2012

33. Conclusions
The data continue to be used by the federal government to justify funding.
Results Driven Accountability is shining a spotlight on each state’s child outcomes data.
States can expect more scrutiny around data quality.
Data quality is not as simple as yes/no, there is a continuum of quality and multiple criteria.
The criteria used for the national analyses do not set a high bar for data quality.
To conclude, the data continue to be very important nationally. They continue to be used by the Department of Ed. to justify funding for Part B preschool and Part C. As we move to the results driven accountability and results indicators are being incorporated into determinations, you are all probably all very aware that it puts a new spotlight on your child outcomes data and not just results but it puts greater scrutiny around data quality. Are the results we are capturing accurate and meaningful?
Data quality is not as simple as yes/no, there is a continuum of quality and multiple criteria.
The criteria used for the national analyses do not set a high bar for data quality and currently have some differences from the criteria used by OSEP for determinations.
We hope that we’ve stimulated some interest on your part into some of the data quality questions, but we also want you to know that we have a lot of resources available for you.
NECTAC/ECO/WRRC 2012

34. How We Can Help!
State data quality profiles for FFY were sent out to C/619 coordinators
Abby Winer with questions:
Contact us for help with data quality analysis and quality assurance activities
Contact us for help with program improvement planning and data analysis
How can we help? We would like to remind you that the state child outcomes data quality profiles for FFY are now available.
They include information comparing state to national, current criteria for inclusion in national estimate (completeness, progress category patterns), and state summary statements over time – from through
We sent them to the Part C and 619 coordinators – please share with your staff. If you have any questions or did not receive it, please contact us.
Also, feel free to contact us if you need assistance with data quality, quality assurance activities or program improvement and data analysis.
We also have several sessions at conferences related to data analysis and quality and program improvement so those are good avenues for stimulation of thought and resources for doing that work.
NECTAC/ECO/WRRC 2012

35. State Child Outcomes Data Quality Profiles FFY 2013-14
This is a reminder of what those data quality profiles are so if you didn’t see the ones from last year I’ll just briefly talk about what they include.
The profiles basically take the info that we collect as part of doing our national analysis and then give it back to you in an individualized state profile form.
We put your state summary state values compared to the national estimate, we show you the data we have on the completeness of the data, which is limited to looking at the number of exiters or child count, but you can see based on the numbers we have if you meet that criteria for inclusion based on completeness.
Also for the progress categories you can see if you have no flags for patterning.
If you meet completeness and have no flags and are not sampling, then you would be included in the national analysis.
Then we also graph the state trends over time for you. The nice piece is that these graphs include what you wouldn’t be able to do on your own, which is that, since we have national data, we’re able to add in a line for the national average and then also a line for one standard deviation above and one standard deviation below so you can get a sense of not only how comparable are you to the national average but are you within that 1 standard deviation range of the national average to give you another comparison point.
Updated profiles were sent to C/619 coordinators 38
NECTAC/ECO/WRRC 2012

36. Part C Specific Updates: Completeness
State
As we have said, quality is not a yes/no, it is a continuum and involves multiple criteria. The state child outcomes data quality profiles are primarily based on the criteria used for inclusion in the national analysis and estimates, which are not identical to those used for Part C determinations this year.
Therefore, for the Part C profiles, we made additions to make comparisons to or align with the criteria for inclusion in the national analysis and the Part C determinations criteria used by OSEP this federal fiscal year.
The two main revisions were that we included the data completeness criteria for determinations and color coded lines to the completeness graphs to be able to compare where your state data fall in relation to the national average, 1 standard deviation above and below as well as the low and higher completeness cutoffs used for determinations.
We also show whether there are unexpected patterns for all 5 progress categories, to be more comprehensive, but we still focus on a and e for the national analysis. 38
NECTAC/ECO/WRRC 2012

37. Part C Specific Updates: Progress Categories
Determinations - Anomalies
ECTA Expected Patterns
Also for progress categories, we added in some explanation of how anomalies in progress categories were defined in the determinations and how they differ from ECTA’s expected patterns for progress categories. In a nutshell, the ECTA expected patterns are based upon previous national data and other data sources used to establish expected percentages and reasonable patterns for the progress categories. For the purpose of determinations, a measure of data anomalies was used and was based upon 1 or 2 standard deviations above or below the average across all states and territories. 38
NECTAC/ECO/WRRC 2012

38. Updated National Graphing Template
This is the updated national graphic template. This is posted on the website. This is just the general one that has our overall national estimates for Part C and B 619 that you can add your state data to if you want to print your own graphs to put into different things.
NECTAC/ECO/WRRC 2012

39. Child Outcomes Summary (COS) Process Module
Many of you have already heard and seen the new Child Outcomes Summary (COS) Process Modules that ECTA and DaSy have released – 3 sessions were released in May 2015 the 4th was just released and there will be a few more coming!
Session 1: Introduction
Session 2: Overview of the COS Process
Session 3: Completing the COS Process
Session 4: The 7-Point Scale 40
NECTAC/ECO/WRRC 2012

40. Frameworks Data System Framework Child Outcomes Measurement Framework
ECTA System Framework
Data System Framework
Child Outcomes Measurement Framework
ECTA and DaSy also have frameworks and staff to support you to unpack your child outcomes data collection to see where you can improve your quality to improve the usability

41. Additional Resources Data quality: Pattern checking
Training materials on looking at data:
Additional data quality resources
Data analysis for program improvement
SSIP-related Resources
Finally, these are additional relevant, but now new, resources related to data quality
if you want to dig into data quality or have concerns.
There’s a pattern checking table which describes strategies for using data analysis to improve the quality of state data by looking for patterns that indicate potential issues for further investigation.
There are also additional training materials for looking at data.
There’s a section on quality assurance with data quality resources.
There’s a using data section with different tools and templates.
And there is an SSIP section of the ECTA website has SSIP-related resources gathered.
NECTAC/ECO/WRRC 2012

42. How to Stay in Touch Email Visit the ECTA and DaSy websites
Abby Winer
Cornelia Taylor
Visit the ECTA and DaSy websites
Thank you and if you have any questions, please Abby Winer or Cornelia Taylor or visit the ECTA and DaSy websites.
NECTAC/ECO/WRRC 2012

43. Conclusion
The contents of this presentation were developed under a grant from the U.S. Department of Education, # H373Z120002, and a cooperative agreement, #H326P120002, from the Office of Special Education Programs, U.S. Department of Education. However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the Federal Government. DaSy Center Project Officers, Meredith Miceli and Richelle Davis and ECTA Center Project Officer, Julia Martin Eile.
NECTAC/ECO/WRRC 2012