Update from ECO: Possible Approaches to Measuring Outcomes Kathleen Hebbeler SRI International Prepared for the OSEP Early Childhood Meeting Washington, DC February 2005

Today’s presentation Update on the outcomes Approaches to measurement Next steps and timeline

Update on the Child and Family Outcomes Have been gathering input for 12 months Input from advisors, work groups, presentations, conference calls, public input solicited through web site

What should be in the outcomes? The outcomes should be statements of what EI and ECSE are trying to do for families and children. Statements at the most fundamental level of what these programs are all about.

Considerations and criteria Consistent with IDEA and legislative intent Apply to the entire birth to 5 age span Apply to all families and children in EI and ECSE Easily compatible with outcomes of other early childhood initiatives/ programs

Input Recieved 2 conference calls with >200 participants total On-line survey (61 responses) Emails (35) On-line discussion forum (1)

Who responded? Fairly equally divided between Part C and Preschool About half local and half state level About 10% family members

Family Outcomes posted for comments Families know their rights and advocate effectively for their child. 2. Families understand their child's abilities and special needs 3. Families help their child develop and learn. 4. Families have the supports they want. 5. Families participate in desired services and activities that are available to all families in their community.

Feedback from the field The word “advocate” in #1 has adversarial connotations to some #2 should include strengths Controversy over outcomes #4 and #5: The word “want” in 4 4 and 5 (and some thought 3) went beyond the requirements of Part B Section 619 Preschool Programs

Family Outcomes Families understand their children’s strengths, abilities and special needs Families know their rights and effectively communicate their children’s needs. Families help their children develop and learn Families feel they have adequate social supports Families are able to access services, programs, and activities that are available to all families in their communities

All five family outcomes apply to Part C Recommendations All five family outcomes apply to Part C Family outcomes numbers 1 and 2, and maybe 3, apply to 619.

Child Outcomes as posted for comment Children have positive social relationships Children acquire and use knowledge and skills Children take action to meet their needs

Feedback from the field The child outcomes were supported by the majority (>80%) of those providing comments, across Part C, Part B 619, and families Outcome #3 was the most difficult to understand; respondents suggested adding “appropriate”

1. Children have positive social relationships Child Outcomes 1. Children have positive social relationships 2. Children acquire and use knowledge and skills 3. Children take appropriate action to meet their needs

Recommendation All three child outcomes apply to both Part C and 619

Feedback on the Preface Nearly all supported the concepts Many excellent suggestions for wording refinements/ minor edits

Measurement of Child Outcomes

Definition of Measurement Measurement - Tools and techniques to quantify the child’s status on aspects of functioning; for example: Norm-based assessments Curriculum-based measures Rating scales

Criteria and Considerations Children with disabilities already have assessment data -- but not from the same assessments Highly probable most data will be collected by local providers Need to minimize burden; try to incorporate into existing procedures

Criteria and Considerations Children with disabilities may be participating in other assessment systems Outcome measurement should be as valid and reliable as possible Quality of data will get better over the first few years

Pitfall Alert! What we want for children and families (desired outcomes) reflects our values What we want for young children may not be the same as what can be easily measured When what is valued and what can be measured easily or well do not line up, EITHER: Outcomes will be determined by what can be measured easily or well, OR Some of the outcomes will be more easily measured or more adequately measured than others

There is no perfect system… …but you get to choose the shortcomings you will live with

Standardized Assessment and Young Children Reliability Interested in behavior over multiple settings but many assessments reflect behavior in one setting (validity, generalizability) Appropriateness of assessment for children with disabilities Accommodations Qualifications of assessors

Measurement Options for Producing Aggregated Data at the State Level All children in state assessed with same measure Different assessments are used and they are converted to the same metric 2 Options available to states that can result in aggregated data across the state.

State Local Local State Local OSEP State State

Two alternative scales are possible So What Is the Metric? Some kind of scale Two alternative scales are possible Age-anchored Continuous progress Each has strengths and weaknesses

Measurement Decision Givens: Different children will have different kinds of assessment data The same child will have several different kinds of assessment data Challenge: How to convert multiple pieces of information (e.g., scores from several standardized assessments, informed clinical opinion, parent report) to a single score?

Informed Clinical Opinion Assessment 1 Assessment 2 Conversion Process Single Score Parent Report Informed Clinical Opinion

How to get to a single score from multiple sources of information Mathematical conversion from assessment data Mathematical conversion with an exception for clinical judgment or parent input Team rating that uses all information on the child to derive a score

Team derives rating Structured process to derive rating Guidance materials would be produced for how to derive ratings incorporating mathematical conversions for common (good) assessment instruments Training materials would be developed

Team derives rating Pro IDEA places great value on team decision-making (eligibility, service decision) Literature suggests teams can produce reliable ratings data Could be part of IFSP or IEP process (does this make it more acceptable to providers?)

Team derives rating Consistent with best practices in child assessment – Uses multiple sources of information Does not put undue weight on standardized assessments “Develop systems of analyses so that test scores are interpreted as part of a broader assessment that may include observations, portfolios, or ratings from teachers and/or parents.” - Nat’l Institute for Early Education Research

Team derives rating Con Requires functioning team Requires some knowledge of typical child development depending on rating scale Would require lots of training

Team derives rating: Unanswered questions What would be required to get reliable ratings? Would the resulting data be seen as credible?

Next steps Feb - Fall 2005– Collect input on measurement approach Spring 2005 – Develop measurement approach and accompanying materials. Small scale pilot work on measurement approach for 1-2 child outcomes and all family outcomes Summer 2005 – Continued development work and pilot work on the other outcomes

Next steps (continued) Fall 2005 – Large scale field testing of measurement of child and family outcomes Winter 2005 – Revisions based on field testing Early Spring 2006 – Release of procedures and materials for use by all states Summer 2006 and onward – Revision to materials based on research on implementation. Ongoing development of training materials.

All of this is very tentative Each next step assumes the previous step is completed on time and the recommendation is accepted.

Follow developments on the web site Contact us with questions