1. Update from ECO: Possible Approaches to Measuring Outcomes
Kathleen Hebbeler
SRI International
Prepared for the OSEP Early Childhood Meeting
Washington, DC
February 2005

2. Today’s presentation
Update on the outcomes
Approaches to measurement
Next steps and timeline

3. Update on the Child and Family Outcomes
Have been gathering input for 12 months
Input from advisors, work groups, presentations, conference calls, public input solicited through web site

4. What should be in the outcomes?
The outcomes should be statements of what EI and ECSE are trying to do for families and children.
Statements at the most fundamental level of what these programs are all about.

5. Considerations and criteria
Consistent with IDEA and legislative intent
Apply to the entire birth to 5 age span
Apply to all families and children in EI and ECSE
Easily compatible with outcomes of other early childhood initiatives/ programs

6. Input Recieved
2 conference calls with >200 participants total
On-line survey (61 responses)
s (35)
On-line discussion forum (1)

7. Who responded?
Fairly equally divided between Part C and Preschool
About half local and half state level
About 10% family members

8. Family Outcomes posted for comments
Families know their rights and advocate effectively for their child.
2. Families understand their child's abilities and special needs
3. Families help their child develop and learn.
4. Families have the supports they want.
5. Families participate in desired services and activities that are available to all families in their community.

9. Feedback from the field
The word “advocate” in #1 has adversarial connotations to some
#2 should include strengths
Controversy over outcomes #4 and #5:
The word “want” in 4
4 and 5 (and some thought 3) went beyond the requirements of Part B Section 619 Preschool Programs

10. Family Outcomes
Families understand their children’s strengths, abilities and special needs
Families know their rights and effectively communicate their children’s needs.
Families help their children develop and learn
Families feel they have adequate social supports
Families are able to access services, programs, and activities that are available to all families in their communities

11. All five family outcomes apply to Part C
Recommendations
All five family outcomes apply to Part C
Family outcomes numbers 1 and 2, and maybe 3, apply to 619.

12. Child Outcomes as posted for comment
Children have positive social relationships
Children acquire and use knowledge and skills
Children take action to meet their needs

13. Feedback from the field
The child outcomes were supported by the majority (>80%) of those providing comments, across Part C, Part B 619, and families
Outcome #3 was the most difficult to understand; respondents suggested adding “appropriate”

14. 1. Children have positive social relationships
Child Outcomes
1. Children have positive social relationships
2. Children acquire and use knowledge and skills
3. Children take appropriate action to meet their needs

15. Recommendation
All three child outcomes apply to both Part C and 619

16. Feedback on the Preface
Nearly all supported the concepts
Many excellent suggestions for wording refinements/ minor edits

17. Measurement of Child Outcomes

18. Definition of Measurement
Measurement - Tools and techniques to quantify the child’s status on aspects of functioning; for example:
Norm-based assessments
Curriculum-based measures
Rating scales

19. Criteria and Considerations
Children with disabilities already have assessment data -- but not from the same assessments
Highly probable most data will be collected by local providers
Need to minimize burden; try to incorporate into existing procedures

20. Criteria and Considerations
Children with disabilities may be participating in other assessment systems
Outcome measurement should be as valid and reliable as possible
Quality of data will get better over the first few years

21. Pitfall Alert!
What we want for children and families (desired outcomes) reflects our values
What we want for young children may not be the same as what can be easily measured
When what is valued and what can be measured easily or well do not line up, EITHER:
Outcomes will be determined by what can be measured easily or well, OR
Some of the outcomes will be more easily measured or more adequately measured than others

22. There is no perfect system…
…but you get to choose the shortcomings you will live with

23. Standardized Assessment and Young Children
Reliability
Interested in behavior over multiple settings but many assessments reflect behavior in one setting (validity, generalizability)
Appropriateness of assessment for children with disabilities
Accommodations
Qualifications of assessors

24. Measurement Options for Producing Aggregated Data at the State Level
All children in state assessed with same measure
Different assessments are used and they are converted to the same metric
2 Options available to states that can result in aggregated data across the state.

25. State
Local
Local
State
Local
OSEP
State
State

26. Two alternative scales are possible
So What Is the Metric?
Some kind of scale
Two alternative scales are possible
Age-anchored
Continuous progress
Each has strengths and weaknesses

27. Measurement Decision Givens:
Different children will have different kinds of assessment data
The same child will have several different kinds of assessment data
Challenge: How to convert multiple pieces of information (e.g., scores from several standardized assessments, informed clinical opinion, parent report) to a single score?

28. Informed Clinical Opinion
Assessment 1
Assessment 2
Conversion Process
Single
Score
Parent Report
Informed Clinical Opinion

29. How to get to a single score from multiple sources of information
Mathematical conversion from assessment data
Mathematical conversion with an exception for clinical judgment or parent input
Team rating that uses all information on the child to derive a score

30. Team derives rating
Structured process to derive rating
Guidance materials would be produced for how to derive ratings incorporating mathematical conversions for common (good) assessment instruments
Training materials would be developed

31. Team derives rating Pro
IDEA places great value on team decision-making (eligibility, service decision)
Literature suggests teams can produce reliable ratings data
Could be part of IFSP or IEP process (does this make it more acceptable to providers?)

32. Team derives rating
Consistent with best practices in child assessment –
Uses multiple sources of information
Does not put undue weight on standardized assessments
“Develop systems of analyses so that test scores are interpreted as part of a broader assessment that may include observations, portfolios, or ratings from teachers and/or parents.”
- Nat’l Institute for Early Education Research

33. Team derives rating
Con
Requires functioning team
Requires some knowledge of typical child development depending on rating scale
Would require lots of training

34. Team derives rating: Unanswered questions
What would be required to get reliable ratings?
Would the resulting data be seen as credible?

35. Next steps Feb - Fall 2005– Collect input on measurement approach
Spring 2005 – Develop measurement approach and accompanying materials. Small scale pilot work on measurement approach for 1-2 child outcomes and all family outcomes
Summer 2005 – Continued development work and pilot work on the other outcomes

36. Next steps (continued)
Fall 2005 – Large scale field testing of measurement of child and family outcomes
Winter 2005 – Revisions based on field testing
Early Spring 2006 – Release of procedures and materials for use by all states
Summer 2006 and onward – Revision to materials based on research on implementation. Ongoing development of training materials.

37. All of this is very tentative
Each next step assumes the previous step is completed on time and the recommendation is accepted.

38. Follow developments on the web site
Contact us with questions