Measuring Child Progress: Two State’s Journeys Barbara Jackson, NE Beppie Shapiro, HI Measuring Child and Family Outcomes Albuquerque, NM April 25, 2006

Format of Presentation Meet each stateMeet each state Overview of the 4 key questions to be discussedOverview of the 4 key questions to be discussed For each key question:For each key question: –Description of each state’s plans –Questions and comments from participants

What Counts: Measuring Benefits of Early Intervention in Hawai`i Beppie Shapiro, University of Hawai`i

About Hawai`i About 17,000 births per year Over 7% of each birth cohort served in Part C No majority ethnicity 4.1 % of population are immigrants, primarily Asians and Pacific Islanders Geographic isolation Five main islands – different access to services

Special education in Hawai`i Part C under Department of Health Part B under Department of Education Outcomes process differs from C to 619 SEA = 1 LEA

Part C in Hawai`i, 2006 Broad eligibility Three DOH agencies provide services statewide Recent development of statewide IFSP to be used by all programs No Part C database – 2 agencies have their own

Standardized IFSP form, process 2005: All Part C programs trained to use new statewide IFSP form –Present Levels of Development –Family-driven Outcomes* and Objectives –Services to support Objectives –Transition

What Counts?.... Measuring the Outcomes of Early Intervention in Nebraska Barbara Jackson, Munroe Meyer Institute

About Nebraska  Birth Mandate State  Co-Leads for Part C: Health and Human Services & Education  Outcome data process will be the same across the birth through 5 age group

Nebraska (continued)  460 School Districts  Number of children birth to 3: 1303  Number of children ages 3 and 4: 2811  Number of children 3 through 5: 4707

Key Question………… 1.How will the state determine children’s status on each of the outcomes?

Assessment in Hawai`i Part B Section 619, Hawai`i Assessment: Early Brigance Part C: Since 2004: 3 assessment tools approved for CDE – Hawaii Development Charts (HELP), “Michigan” EIDP, or ASQ

Hawai`i’s Part C Assessment : Ages and Stages For babies eligible due to environmental risk Forms for specific age levels Scores indicate only “Typical”, “Monitor”, “Refer” Referred for CDE and services by program serving DD Care coordination may remain with original program

1. 1.What Counts Design Team convenes monthly to develop/review progress 2. 2.How can we assign a “score” to each child on each EI Goal? What Counts: Measuring the Benefits of Early Intervention in Hawai`i:

Hawai`i’s Part C Outcomes (Goals) Measurement Process Data Collection Intervals & Rationale – – At every IFSP: initial, review, annual – –Starting when child is at least 4 months old

Hawai`i’s Part C Outcomes (Goals) Measurement Process Assigning scores: decisions & rationale WHO: IFSP team Following “Present levels of Development” description HOW: all team members reach consensus on rating using modified ECO Child Summary Reporting Form

Hawai`i’s 619 Outcomes Measurement Process Family involved in assessment process Family provides input on written form with open-ended questions A professional assigns rating on each goal Other providers asked for input to rating Ratings assigned at entry and annually

Nebraska’s Process  Child Outcome Task Force convened to guide process

Nebraska calls for Child Assessment that….  Is based on ongoing observation of children engaged in real activities, with people they know, in natural settings  Engage families and primary caregivers as active participants  Is individualized to address each child’s unique ways of learning  Reflects that development and learning are rooted in culture and supported by the family  Integrates information across settings

Nebraska’s Assessment Process  Which assessments?  AEPS  Creative Curriculum  High Scope COR for Infants/ Toddlers & Preschoolers

Why Selected?  Assessment approach parallels other Early Childhood Program assessment processes  Use information from multiple sources (e.g., family, providers) and multiple observations  Curriculum-based Assessment – can be used for multiple purposes  Can build the capacity of our system to support children and their families

Nebraska’s Process for parent input?  Parents provide input during the assessment process

How is the assessment information “transformed”?  Working with publishers to determine feasibility of computer- based formulas  Scores will be reviewed by team to assure validity of score

Nebraska’s Assessment Process Data Collection Schedule  Districts will be mandated to report entry and exit data  Entry data will collected within the days after IFSP/IEP meeting  Exit data collected within 2 months of leaving the program  Districts will be encouraged follow publishers guidelines for frequency of assessments

Key Question………… 2. What reporting categories has the state chosen to use?

Reporting Categories in Hawai`i What Counts: Measuring Benefits of Early Intervention in Hawai`i

Reporting Categories in Hawai`i Part C Decision & Rationale Use ECO 5 Categories A. A.maintained typical functioning B. B.made progress to achieve typical functioning C. C.moved nearer to typical functioning D. progress but not enough to move nearer to typical E. did not make progress Meaningful, Program Improvement

Reporting Categories in Hawai`i 619 –OSEP Categories

Reporting Categories in NE  Four Categories  OSEP’s 3 categories  Plus % of children who attain typical development

Key Question………… 3. How and in what form will data get from local programs to the state?

Getting Part C Data to the State In Hawai`i Simpler than most states? More difficult than many states? Each “Agency” will collect and summarize data from its programs

Data to explain results Length of enrollment Age at enrollment ICD9s/conditions

Getting Data to the State: Part C Electronic data entry at local program Transmitted to Agency Agency calculates number of children in each category Agency sends these numbers to State State creates OSEP and cross-Agency reports

Unresolved data questions Local programs send “explanatory data” to Agency –How much of this data goes to State? How? How to identify potential duplicates

Getting Data to the State: 619 Teacher enters data into stand-alone EXCEL spreadsheet Spreadsheets collected at school, sent to State

NE: Getting Data to the State  Using the publisher’s internet system- State will be the licensed manager  Link with the State Data System

Key Question………… 4. What will you do to maximize the reliability of the data?

Training, Support to Maximize Data Reliability in Hawai`i Training:Support:Web: 7 hrs, all EI programs, by community 1-2 months post training: On site? Conference call? FAQs, listserv

Maximizing Data Reliability in Hawai`i: Evidence for Rating IFSP teams document evidence used to select rating category Pilot showed need for training on what makes good evidence Supervisors provide support and QA Criteria for Evidence (in progress) Different contexts Highest level of achievement Specific to Goal Area

Examining Reliability in Hawai`i Another measurement of “how well child is doing” –Compare ratings Compare ratings for children with different conditions/diagnoses

Challenges for Hawai`i: Quality Assurance - Part C How reliable are assessments? Does every IFSP team have someone knowledgeable about typical child development? How reliable are summary form ratings?

Reliability of Data in NE: Training  Intensive training on assessments throughout state with consistent trainers

Reliability of Data in NE: Implementation  Review of data at the state level  Ongoing questions/answers communication document based on local questions  Team approach to assessment process  Pilot process to determine reliability issues