Child Outcomes: Understanding the Requirements in order to Set Targets Presentation to the Virginia Interagency Coordination Council Infant & Toddler Connection of Virginia 1

Topics to be Covered Today The three child outcomes OSEP reporting requirements How the data is gathered OSEP Progress Categories Summary Statements Considerations when reviewing baseline data for setting targets OSEP Infant & Toddler Connection of Virginia 2

OSEP Reporting Requirements: Child Outcomes Positive social emotional skills (including positive social relationships) Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) Use of appropriate behaviors to meet their needs Infant & Toddler Connection of Virginia 3

OSEP Reporting Categories Percentage of children who: a. Did not improve functioning b.Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c.Improved functioning to a level nearer to same- aged peers but did not reach it d.Improved functioning to reach a level comparable to same-aged peers e.Maintained functioning at a level comparable to same-aged peers Infant & Toddler Connection of Virginia 4

OSEP Reporting Categories 3 indicators X 5 Reporting Categories 15 numbers to Report to OSEP Infant & Toddler Connection of Virginia 5

Reporting Schedule 2008 and 2009 Reporting Schedule  Data in reporting categories at exit for all children who have been in the program for at least 6 months  First submission was due February 2008 for the year July 1, June 30, Infant & Toddler Connection of Virginia 6

Reporting Schedule 2010 in SPP format: – Baseline – Targets for 2 reporting years – Improvement activities for 2 reporting years 2011 and 2012 in APR format: – Actual data, progress and slippage, etc. – Local reporting of [summary statement %s] Infant & Toddler Connection of Virginia 7

Understanding the Three Child Outcomes In Virginia, we refer to the OSEP child outcomes as child indicators to distinguish between the individual child outcomes and the OSEP outcomes. Infant & Toddler Connection of Virginia 8

The Child Indicator Summary Form (CISF) 7 point rating scale Team summarizes multiple data sources (NOT an assessment) Rating the status of child’s functioning at entry and again at exit Comparing child’s functioning to what is expected at his/her age Infant & Toddler Connection of Virginia 9

The two CISF questions a. To what extent does this child show age-appropriate functioning, across a variety of settings and situations, on this outcome? (Rating: 1-7) b. b. Has the child shown any new skills or behaviors related to [this outcome] since the last outcomes summary? (Yes-No) Infant & Toddler Connection of Virginia 10

Essential Knowledge for Completing the CISF Between them, team members must: Know about the child’s functioning across settings and situations Understand age-expected child development Understand the content of the three child outcomes Know how to use the rating scale Understand age expectations for child functioning within the child’s culture Infant & Toddler Connection of Virginia 11

7 Point Rating Scale See handout Infant & Toddler Connection of Virginia 12

Infant & Toddler Connection of Virginia 13 Understanding the OSEP Progress Categories

Helping Children Move Toward Age-expected Functioning Assumption: Children can be described with regard to how close they are to age expected functioning for each of the 3 outcomes By definition, most children in the general population demonstrate the outcome in an age- expected way By providing services and supports, EI is trying to move children closer to age expected behavior Infant & Toddler Connection of Virginia 14

Key Points The OSEP categories describe types of progress children can make between entry and exit Two scores or ratings (entry and exit) are needed to calculate what OSEP category describes a child progress Service providers are not responsible for calculating progress (this happens at the state level) Infant & Toddler Connection of Virginia 15

The “a” category a. Percent of infants and toddlers who did not improve functioning Children who acquired no new skills or regressed during their time in the program Didn’t gain or use even one new skill Children with degenerative conditions/ significant disabilities Infant & Toddler Connection of Virginia 16

CISF ratings and the “a” category Rated lower at exit than entry; OR Rated 1 at both entry and exit; AND Scored “No” on the progress question (b) Infant & Toddler Connection of Virginia 17

Early Childhood Outcomes Center 18 EntryExit

Early Childhood Outcomes Center 19 EntryExit

The “b” category b. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers Children who acquired new skills but continued to grow at the same rate throughout their time in the program Gained and used new skills but did not increase their rate of growth or change their growth trajectories while in services Infant & Toddler Connection of Virginia 20

CISF ratings and the “b” category Rated 5 or lower at entry; AND Rated the same or lower at exit; AND “Yes” on the progress question (b) Infant & Toddler Connection of Virginia 21

Early Childhood Outcomes Center 22 EntryExit

Early Childhood Outcomes Center 23 EntryExit

Early Childhood Outcomes Center 24 EntryExit

The “c” category c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach it Children who acquired new skills but accelerated their rate of growth during their time in the program Made progress toward catching up with same aged peers but were still functioning below age expectations when they left the program Changed their growth trajectories --“narrowed the gap” Infant & Toddler Connection of Virginia 25

CISF ratings and the “c” category Rated higher at exit than entry; AND Rated 5 or below at exit Infant & Toddler Connection of Virginia 26

Early Childhood Outcomes Center 27 EntryExit

The “d” category d. Percent of infants and toddlers who improved functioning to reach a level comparable to same-aged peers Children who were functioning below age expectations when they entered the program but were functioning at age expectations when they left Started out below age expectations, but caught up while in services Infant & Toddler Connection of Virginia 28

CISF ratings and the “d” category Rated 5 or lower at entry; AND Rated 6 or 7 at exit Infant & Toddler Connection of Virginia 29

Early Childhood Outcomes Center 30 EntryExit

The “e” category e. Percent infants and toddlers who maintained functioning at a level comparable to same-aged peers Children who were functioning at age expectations when they entered the program and were functioning at age expectations at exit Infant & Toddler Connection of Virginia 31

CISF ratings and the “e” category Rated 6 or 7 at entry; AND Rated 6 or 7 at exit Infant & Toddler Connection of Virginia 32

Early Childhood Outcomes Center 33 EntryExit

Early Childhood Outcomes Center 34 EntryExit

The Challenge…. Progress data included 5 progress categories For each of 3 outcomes Total of 15 numbers reported each year Too many interrelated targets to understand OSEP asked for a recommendation Infant & Toddler Connection of Virginia 35

Development of the Summary Statements ECO presented options to states and ECO work groups via conference calls Two sessions at December, 2008 EC Conference Posted on the ECO web site for comments ECO made recommendation to OSEP Infant & Toddler Connection of Virginia 36

Final Deliberation OSEP put the summary statements out for public comment Comments came in that were thoughtful, but not necessarily consistent with one another Advantages and disadvantages to all options Paper documenting the process on the ECO website mary_of_target_setting-2.pdf Infant & Toddler Connection of Virginia 37

The Summary Statements Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program. The percent of children who were functioning within age expectations in each Outcome by the time they exited the program. Infant & Toddler Connection of Virginia 38

Understanding the Data Sample data will be used in the next several slides to demonstrate the process for calculating the summary statements from the data in the 5 OSEP categories Infant & Toddler Connection of Virginia 39

Example of State Progress Data for Positive social-emotional skills (including social relationships): Number of children % of children a. Percent of infants and toddlers who did not improve functioning 404 b. Percent of infants and toddlers who improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers c. Percent of infants and toddlers who improved functioning to a level nearer to same-aged peers but did not reach d. Percent of infants and toddlers who improved functioning to reach a level comparable to same- aged peers e. Percent of infants and toddlers who maintained functioning at a level comparable to same-aged peers TotalN= %

Summary Statement Data Required Summary Statement 1: Of those children who entered the program below age expectations in each Outcome, the percent who substantially increased their rate of growth by the time they exited the program= 75% Required Summary Statement 2: The percent of children who were functioning within age expectations in each Outcome by the time they exited the program= 54% Infant & Toddler Connection of Virginia 41

Where do the #s come from? 42 Prog cat #% a 404 b c d e (a, b, c, and d) or 76% of the children entered the program functioning below age expectations 240 (e) or 24% of the children entered and exited functioning at age expectations

Where do the #s come from? Measurement for Summary Statement 1: Percent = # of infants and toddlers reported in progress category (c) plus # of infants and toddlers reported in category (d) divided by [# of infants and toddlers reported in progress category (a) plus # of infants and toddlers reported in progress category (b) plus # of infants and toddlers reported in progress category (c) plus # of infants and toddlers reported in progress category (d)] times 100. Infant & Toddler Connection of Virginia 43

Where do the #s come from? 44 Prog cat #% a 404 b c d e (c and d) of the 760 (a, b, c, and d) changed their growth trajectories (made greater than expected progress) = = 75%

45 Summary Statements Calculator Summary Statements Calculator -April 14, 2009

Where do the #s come from? Measurement for Summary Statement 2: Percent = # of infants and toddlers reported in progress category (d) plus [# of infants and toddlers reported in progress category (e) divided by the total # of infants and toddlers reported in progress categories (a) + (b) + (c) + (d) + (e)] times 100. Infant & Toddler Connection of Virginia 46

Where do the #s come from? 47 Prog cat #% a 404 b c d e = = 54% 30% of the children reached age expectations by exit and 24% of the children entered and exited at age expectations

What can we say about the children’s progress? Part C Outcome 1: successful social relationships with peers and adults, following rules for social interactions 96% of children participating in Part C made progress in their social relationships while they were enrolled. The 4% of children who did not make progress included children with the most severe disabilities and/or degenerative conditions. Can you describe them? Infant & Toddler Connection of Virginia 48

Positive Social Relationships 24% of the children participating in Part C were functioning at age expectations at entry and at exit in this outcome area. 54% of the children were functioning at age expectations in this outcome area when they exited the program. (summary statement 2) 30% started out behind and caught up 24% entered and exited at age expectations Infant & Toddler Connection of Virginia 49

Positive Social Relationship 75% of the children who entered the program below age expectations made greater than expected gains; they made substantial increases in their rates of growth. i.e. changed their growth trajectories (summary statement 1) Infant & Toddler Connection of Virginia 50

Setting Targets Quality Data is essential for setting realistic targets Two strategies for examining data Data quality Potential for program improvement Parameters, guidance for target setting Infant & Toddler Connection of Virginia 51

Can you trust the data? Mechanisms that impact quality of data: ITOTS mechanisms to prevent inaccurate data entry Personnel Training Initial Statewide training Training resources, including online training modules and Implementation Manuals Local training and oversight Review of Data – TA Guidance at Regional Meetings Infant & Toddler Connection of Virginia 52

Can you trust the data? Look at the data across all systems in order to analyze data quality: Begin by identifying outliers Examples: look at the percentages reported for certain categories across local programs Infant & Toddler Connection of Virginia 53

Percentages reported in category “a” across 30 (sample) local programs 54

Look at the Impact of the Outliers on the Data State percentage for “e” with all data= 32.1% Revised percentage for “e” with outliers removed= 27.7% 55

Example of data with outliers removed Progress Category Original %Clean % a 42 b 1517 c 2730 d 31 e 2420 Sum St Sum St Clean data (without the outliers) may be a more accurate picture of where you are starting 56

Suggested Strategy Analyze your data with your local system outliers included and excluded so you can gauge the impact they are having on your state level data. Infant & Toddler Connection of Virginia 57

Important Note about Reporting Data to OSEP Consider clean data when deciding about reasonable targets, BUT Turn in the original data to OSEP in the SPP report! You can discuss the clean data in the rationale for your targets. Infant & Toddler Connection of Virginia 58

Which local systems can be targeted for program improvement? Compare the summary statement data by local programs to identify which programs have the most potential for improvement. Infant & Toddler Connection of Virginia 59

Summary Statement Percentages by Local Program 60

Data Quality Considerations What do you know about the local systems with the least and the most progress in the summary statements? i.e. the programs with: the lowest and highest percentages of children at age expectation at exit the lowest and highest percentage of children making greater than expected gains Infant & Toddler Connection of Virginia 61

Examples of Key Questions Are the children similar at entry? Are the higher performing local systems participating in special projects or have they instituted training programs? Are there systems issues in lower performing local systems that would explain differences in outcomes? e.g. personnel shortages Infant & Toddler Connection of Virginia 62

Bottom Line Question Could either system or practice focused improvement activities targeted toward the lowest performing local systems improve the child outcomes? Infant & Toddler Connection of Virginia 63

The Math of Target Setting How much would the data change if the lowest local programs moved toward the mean? Improvements in the lowest programs will result in improvement in your statewide data Experiment with your data to determine what targets are reasonable in your state Infant & Toddler Connection of Virginia 64

Next Steps VICC Subgroup meeting (1/2 day) to review Virginia’s data in detail and develop recommendations for targets. Presentation of recommendations from the VICC subgroup to the full VICC at the December VICC meeting Infant & Toddler Connection of Virginia 65

Additional Information Options and ECO Recommendations for Summary Statements for Target Setting target_setting-2.pdf Summary Statement Slides Target Setting Slides Infant & Toddler Connection of Virginia 66