1 Birth to 3 Child Outcomes Maryland’s Approach to Converting Assessment Data to OSEP Outcome Categories August 28, 2007 Deborah Metzger Funded by IDEA General Supervision Enhancement Grant (GSEG) CFDA84.326X

2 Outcome Data Collected When the Child Enters and When the Child Exits Example Child enters at 20 months Initial Present Levels of Development data are extracted from the IFSP database Child exits at 36 months Present Levels of Development data at Exit are extracted from the IFSP database and compared to entry data to determine progress Status At Entry Data Progress At Exit Data

3 How Do We Get The Data? (for Status At Entry and Progress at Exit) Data Extracted from Present Levels of Development and Electronically Linked to 3 Outcomes to Produce Answers Is the child’s acquisition and use of knowledge and skills (including early language/communication) at the level expected for his or her age? Are child’s social-emotional skills (including social relationships) at the level expected for his or her age? Does the child use appropriate behavior to meet his or her needs at the level expected for his or her age? Alignment of broad outcomes to Present Levels of Development

4 Protocols for Linking Age Levels/Age Ranges with Outcomes For the outcome “acquisition and use of knowledge and skills (including early language/communication),” two domain categories (cognitive and communication) will be used. If both domains have quantitative data, the category that has the lowest range of data will be used. When an age range has been entered, the midpoint of the range will be used.

5 Rationale for Maryland’s Approach Decision to align outcome process with the IFSP process Focus on improving evaluation and assessment practices (Online Tutorial) Focus on ensuring data is collected in all domains (Monitoring) Have a data system that collects Present Levels of Development (PLOD) Could get started by generating electronic reports from data entered into PLOD Response to local input

6 Most Commonly Used Tools in Maryland Early Learning Accomplishment Profile (E-LAP) Early Intervention Developmental Profile (EIDP) Hawaii Early Learning Profile (HELP) Battelle Developmental Inventory II Ages and Stages

7 How will Local Programs Measure and Report Status at Exit Data Assessment at exit results will be entered on the IFSP form, Section II: Present Levels of Development (PLOD). The form will be revised so that providers can check off if the PLOD are from an entry, interim, or exit assessment. The name of the assessment(s) that were used will be documented on the IFSP form. The results of the assessments at exit will be entered into the new screens in the IFSP database.

8 Need to Validate the Data Compare Domain Results to Functional Results

9 Child Outcomes—Validation WHY? Maryland has elected to use the information from domain-specific assessment results to determine the results of functional child outcomes. Therefore it is important that the electronic results be validated. In other words, we are asking the following: “Are the responses derived from the electronically- extracted domain data consistent with direct responses from providers about a child’s functioning in the three outcomes?”

10 How Will We Validate the Results? In December 2006, local Infants and Toddlers Programs began completing the Child Outcome Summary Form as soon as possible following initial evaluation and assessment. Local programs are completing COSFs at exit for children: Who were referred since December 2006 Who received services for at least six months, and For whom a COSF was completed at entry COSF results will be entered into the IFSP database.

11 What if the “electronic results” and “validation results” are different? During , we will begin to conduct a validation study by comparing electronic results and COSF results, conducting focus groups with local programs, and discussing discrepancies in results with local provider teams. Based on the validation study results, we will determine how our approach should be modified or changed, whether we will continue our progress reporting by using Present Levels of Development data or through the COSF.

12 Converting Assessment Data to OSEP Outcome Categories

13 Measuring Progress Based on the Rate of Growth Between Entry and Exit Working with Evaluation and Assessment Consultant to identify a methodology for measuring developmental gains during participation in early intervention Testing child data using two existing indices: Intervention Efficacy Index Proportional Change Index

14 Intervention Efficacy Index Relates changes in child capabilities to time spent in program; describes individual and group progress in terms of developmental gains within and across domains for each month in an intervention program.(Bagnato & Neisworth)

15 Intervention Efficacy Index IEI = Developmental gain in months Time in intervention in months IEI = Exit DA - Entry DA Time in Intervention IEI = 34 months-20 months 12 months IEI = 1.17

16 Proportional Change Index Controls for children’s developmental status before intervention Calculation is a ratio of a child’s rate of development at pretesting to the rate of Development during intervention defined at posttesting (Wolery, 1983)

17 Proportional Change Index PCI = Exit DA-Entry DA / Entry DA Time in Intervention Entry CA PCI = 34months-20months/ 20 months 12 months 24 months PCI = 14 / PCI = 1.40

18 Linking Results to OSEP Categories Test both Indexes with real-child data Determine numerical ranges for linkage to OSEP categories Decide which index yields most meaningful results and most accurately matches the OSEP categories

19 Pros, Cons, Challenges! Pros: Existing IFSP data and statewide database Use of COSF for validation has generated good discussions of functional performance Possibility of more meaningful results using factors such as time in intervention and relationship of chronological age to developmental age at entry

20 Pros, Cons, Challenges! Cons: Using domain-based assessment results to Measure developmental progress in functional Outcomes Using multiple assessment tools, rather than single or limited number of tools Challenges: Making decisions thoughtfully with regard to Impact on local programs and families, but quickly enough to ensure meaningful data and analysis Sleeping at night without outcome nightmares!

21 Ultimate Goal: Positive Results for Infants, Toddlers, and Families