1. Measuring Early Childhood Special Education Outcomes in Massachusetts
Pat Cameron, Department of Early Education and Care
Donna Traynham, Department of Education
October 2, 2007

2. What We Will Cover Why collect outcomes data?
Understanding the child outcomes
Assessing the accomplishment of the 3 child outcomes
Using the Child Outcomes Summary Form (COSF)
Collecting and reporting data using the COSF

3. Essential Knowledge for Completing the COSF
Team members know about:
The child’s functioning across settings and situations
Age-expected child development
Content of the 3 outcomes
How to use the rating scale

4. Why Collect Outcomes Data?

5. Public Policy Context Age of accountability
Accountability increasingly means looking at results – not just process
Office of Special Education Programs (OSEP) is under increasing pressure to produce outcomes data on children participating in early intervention and early childhood special education programs
Federal government is the driving force behind the move to collect outcomes data
“The Department has no performance information on preschool children with disabilities served by this program.” from Expectmore.gov

6. OSEP’s Response
Required states to submit outcomes targets and data in their State Performance Plans and Annual Performance Reports (SPP/APRs)
Funded the Early Childhood Outcomes Center to make recommendations, and to assist states in collecting, reporting and USING outcome data

7. OSEP Reporting Requirements: Child Outcomes
Positive social emotional skills (including positive social relationships)
Acquisition and use of knowledge and skills (including early language/communication [and early literacy])
Use of appropriate behaviors to meet their needs
Lynne elaborates on these as
outcome 1 – getting along with others;
outcome 2 children as learners;
outcome 3 meeting their needs
The and early literacy portion is relevant for 619, not included in the Part C child outcome statements.

8. OSEP Reporting Categories
Percentage of children who:
▪ Did not improve functioning
▪ Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers
▪ Improved functioning to a level nearer to same-aged peers but did not reach it
▪ Improved functioning to reach a level comparable to same-aged peers
▪ Maintained functioning at a level comparable to same-aged peers
Note, previously there were only 3 reporting categories in case this seems confusing. OSEP revised the requirements, and released the new 5 category version in August, States will be expected to report data in these five categories beginning in 2008 (using 2 points of data collected, with entry between July 1, 2005-June 30, 2006 and exit anytime 6 months after that up until the June 30, 2007 window for data closes). [Note, some states may have received approval to slightly modify the fiscal year calendar to end Sept 30 of each year rather than June 30th…
3 outcomes x 5 “measures” = 15 numbers

9. Schedule for State reporting to OSEP for Cohort 2
Status-at-entry data must be reported to OSEP in February, 2008
Progress data must be reported to OSEP in February, 2009
Progress data are based on the difference between each child’s status at entry and child’s status at exit

10. How the Data inform State and Local Practices
Providing data for the federal government is not the only reason to collect outcomes data.
Data on outcomes are important for state and local purposes
To document program effectiveness
Support continued or increased funding
To improve programs
Identify strengths and weaknesses
Determine technical assistance and/or staff development needs

11. Think of all that we can learn for program planning and improvement purposes
What would you like to know about YOUR program?
Which demographic group of children make the most/least progress?
Which service delivery options are associated with better progress toward outcomes?

12. Values guide choices of methods
Although all states are required to measure child outcomes for early childhood programs, the strategies each state has chosen are based on the values held by the state about assessing young children.

13. Understanding
the 3 Child Outcomes

14. 3 Child Outcomes
Children have positive social-emotional skills (including social relationships)
Children acquire and use knowledge and skills (including early language/ communication [and early literacy])
Children use appropriate behaviors to meet their needs

15. Where did they come from?
National consensus on purpose of Part C and ECSE programs:
“…To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings – in their homes with their families, in child care, in preschool programs, and in the community.”

16. Outcomes are Functional
Functional refers to things that are meaningful to the child in the context of everyday living
Refers to an integrated series of behaviors or skills that allow the child to achieve the outcomes
They are not
a single behavior, nor are they
the sum of a series of discrete behaviors

17. Outcomes are Functional
They cross domains– do not separate child development into discrete areas (communication, gross motor, etc.)
Emphasis is on how the child is able to carry out meaningful behaviors in a meaningful context
Focus is functionally meaningful actions. Lots of assessment tools have items that are individual skills that aren’t individually meaningful – only meaningful when integrate it with other things to accomplish something, turn the little thing into a whole.

18. A Puzzle 8, 5, 4, 1, 7, 6, 3, 2 Where does 9 go? Where does 0 go?
The numbers are arranged alphabetically. But, without telling people that they find the ordering mysterious and confusing because they expect them to be in number order. This slide can be used to make the point that we all have expectations on how information is organized. Once you know the new system, it is fairly easy to do, but it can take a while to get used to making the shift to the new organizational set.

19. Thinking Functionally
Knows how to make eye contact
Smiles
Give hugs when prompted
Functional?

20. Thinking Functionally
Can imitate a gesture when prompted by others
Functional?

21. Thinking Functionally
Uses finger in pointing motion
Functional?

22. Thinking Functionally
Takes 4 steps on 6 inch balance beam
Climbs 6 stairs, one foot on each step
Functional?

23. Children Have Positive Social Relationships
Involves:
Relating with adults
Relating with other children
For older children- following rules related to groups or interacting with others
Includes areas like:
Attachment/separation/ autonomy
Expressing emotions and feelings
Learning rules and expectations
Social interactions and play

24. Children Acquire and Use Knowledge and Skills
Involves:
Thinking, reasoning, remembering, problem-solving
Using symbols and language
Understanding physical and social
worlds
Includes:
Early concepts – symbols, pictures, numbers, classification, spatial relationships
Imitation
Object permanence
Expressive language and communication
Early literacy

25. Children Take Appropriate Action to Meet Their Needs
Involves:
Taking care of basic needs
Getting from place to place
Using tools
In older children, contributing to their own health and safety
Includes:
Integrating motor skills to complete tasks
Self-help skills (e.g., dressing, feeding, grooming, toileting, household responsibility)
Acting on the world to get what one wants

26. Elaboration of the ECO Outcomes
Children have positive social relationships
Children acquire and use knowledge and skills
Children take appropriate action to meet their needs
Relation-ships with adults
Relation-ships with peers
Follows group rules
Masters the environ-ment
Symbol use, abstract thinking
Applies
knowledge
Listening
Play
Exploring
Playing
Being curious
Practicing
Touching
Attending
Engaging
Persisting
Knowledge of physical world & culture
Self-care, health and safety
To be active and successful participants
now and in the future
in a variety of settings
To be active and successful participants
now and in the future
in a variety of settings

27. Assessing the Accomplishment
of the 3 Outcomes

28. What is Assessment?
“Early childhood assessment is flexible, collaborative decision-making process in which teams of parents and professionals repeatedly revise their judgments and reach consensus about the changing developmental, educational, medical, and mental health services needs of young children and their families.”
Bagnato and Neisworth, 1991
Quoted in DEC Recommended Practices, 2005
Stress difference again between formative assessment and eligibility determination.

29. DEC Recommended Practices for Assessment
Involves multiple sources (e.g., families, professional team members, service providers, caregivers)
Involves multiple measures (e.g., observations, criterion-curriculum-based instruments, interviews, informed clinical opinion)

30. Assessment Instruments
Potential: Assessment tools can inform us about children’s functioning in each of the 3 outcome areas
Challenge: There is no assessment tool that assesses the 3 outcomes directly

31. The Assessment Tool Lens
Each assessment tool carries its own organizing framework
Many are organized around domains
But what is covered in the domains isn’t always the same, even if the names are the same

32. Currently Available Assessment Tools
There are not right and wrong assessment tools
Key question to ask about any assessment tool:
How much and what information will the
tool provide about the attainment of the
3 functional child outcomes?

33. Using Information from Assessment Tools
ECO Center has “crosswalked” assessment tools to the outcomes
Crosswalks show which sections of assessment are related to each outcome
The number of items addressing an outcome does not necessarily mean that the assessment captures functioning across settings

35. Making Use of Information from Assessment Tools
Information from formal or published assessment tools can be very useful, as long as it is used in the context of achievement of the three functional outcomes
The information almost always needs to be supplemented with additional information

36. MA Direction and Decisions
Using the Child Outcome Summary Form (COSF)
Rating children annually in the fall and spring
Using information from assessment tools currently in use in local districts

37. Using the
Child Outcomes
Summary Form
(COSF)

38. Why is There a Need For the Child Outcomes Summary Form?
No assessment instrument assesses the 3 outcomes directly
Many states will allow local programs to use different assessment instruments, and outcomes data will need to be aggregated
The summary form’s 7 point rating scale defines a child’s current functioning in a metric that can be compared over time to reflect child progress

39. Features of the Child Outcomes Summary Form
Not an assessment tool
Uses information from assessment tools and other data sources to determine an overall rating of how the child is functioning in each outcome area, at one point in time

40. Features of the Child Outcomes Summary Form
7-point rating scale
Rating is based on child’s functioning:
How a child functions across settings and situations
Compared to what is expected of a child his/her age

41. Thinking About the Achievement of Each Child Outcome
Figure represents range of skills and behavior observed in all children in the general population.
Yellow circle represents those with age-expected behavior. That includes quite a range of skills. By definition, most children in the general population fit into the yellow circle. There are some children at the fringe of the yellow circle (just beyond the dotted line, still in yellow) that show age-expected skills now, but for whom providers and parents have concerns that if they continue on this developmental course, then in future years they might no longer show age-expected skills. (Perhaps they have the skills now, but don’t appear to be gaining new skills at a rate or in quality that parallels other children their age). Then, there are children who right now do not show age-expected skills and behaviors. Some are pretty close, but not quite at age-expected levels (like the magenta -- you might see a mix of age-expected and not age-expected skills in different situations or at different times). Others are farther from age-expected levels – like the blue ring where a child might show skills that are the immediate foundation for age-expected skills (not displaying age-expected skills yet, but a step just before that in the sequence of development). Still other children are farthest from age-expected levels. Those in the outermost pink ring also show skills, but they don’t show those expected for their age or even those that are the close precursors to those skills. Over time, children might move from the inside of the circle out away from age expected (e.g., the bar gets higher – look more age expected as a baby, but fall farther away with increasing age) or move from the outside of the circle in (e.g., child exhibiting delays gets needed services, accommodations and begins to catch up in functioning to the skills shown by same-age peers). Goal of early intervention and preschool special education is to help children move closer to the yellow circle. Not all will reach it, but in the work that we do, we hope to either minimize children’s movement away from yellow (e.g., in cases like Down Syndrome, where this typically does happen) or to help a child move from a more-outer circle as close into yellow as possible.

42. Helping Children Move Toward Age-Expected Functioning
Assumption: Children can be described with regard to how close they are to age-expected behavior in each of the 3 outcomes
By definition, most children in the general population demonstrate the outcome in an age-expected way
By providing services and supports, ECSE is trying to move children closer to age-expected behavior

43. Measuring Functioning Compared to Age-Expectations
Documenting children’s movement toward age-expected development is one type of evidence that program services are effective
The Child Outcome Summary Form (COSF) was designed to measure this type of progress

44. Essential Knowledge for Completing the COSF
Team members need to know:
The child’s functioning across settings and situations
Age-expected child development
Content of the 3 outcome areas
How to use the rating scale

45. Child Outcomes Summary Form (COSF)

46. Summary Ratings (1-7)
Provide an overall sense of the child’s current functioning in 3 areas
They are not:
Information on the services provided themselves
The family’s satisfaction with services
An explanation of why the child’s functioning is at that level

47. Summary Ratings (1-7)
Reduce rich information from assessment and observation into a rating to allow a summary of progress across children
Do not provide information for planning for the individual child
Information at the rich, detailed level will be more helpful for intervention planning purposes.

48. A Domain Score on an Assessment Tool Does Not Necessarily Translate Directly into an Outcome Rating
Ratings require
Looking at functional behaviors, and
Collecting and synthesizing input from many sources familiar with the child across different settings and situations.

49. Summary Ratings are Based on…
Sources of Evidence
Parents and family members
Service providers
Therapists
Physicians
Child care providers
Teachers
People familiar with the child in all the settings and situations that he/she is in
Types of Evidence
Curriculum-based assessments (e.g., HELP)
Norm-referenced assessments (e.g.,BDI-2)
Developmental screenings (e.g., Ages and Stages)
Parent and professional observation and report

50. Ratings on the 3 Child Outcomes
Ratings on all 3 outcomes should be reported for every child enrolled
Ratings are needed in all areas even if:
No one has concerns about a child’s development
A child has delays in one or two outcome areas, but not in all three outcome areas
Even if the domain is not in the area of disability

51. Assistive Technology Considerations
Ratings should reflect the child’s level of functioning using whatever assistive technology or special accommodations are present in the child’s typical settings
Children who could benefit from assistive technology but don’t have it will get lower scores
This does not reflect on the child’s inability inasmuch as the fact the child does not have the necessary equipment/services

52. Children who have Articulation Problems only
How are articulation difficulties impacting the child in each of the outcome areas?
social relationships?
acquisition of skills and knowledge?
ability to communicate wants and needs?

53. Including Parents in the Discussion
No consensus around the country on whether parents should be included in deciding on the summary rating. Even parent groups don’t agree
Some states are including parents to help reach a rating
Others states are deciding on a rating without parents there

54. Including Parents in the Discussion
Parent input about the child’s functioning is critical
Family members see the child in situations that professionals do not
Need to ask family members about what the child does at home
Need a way to learn about what family members know about the child
No expectation that parents will be able to determine if what they are seeing is age appropriate

55. Explaining the Rating to Parents
If parents are included in deciding on a rating, professionals will need to be able to explain this process to parents
Even if parents are not included in deciding on a rating, professionals will need to be able to explain why the rating is being done and what it means
However, if your state develops materials before we do, please share them with us!!

56. ECO Resources for Child Outcomes Work
A variety of information and resources, including additional crosswalks, training slides, and examples of state developed materials for professionals and parents are posted on our web site
Usually somewhere in the day with the COSF training is a section delivered by the state helping individuals see how this will be specifically implemented in the state – describes state policies, how it fits with current activities (and tips to help it work and flow), guidelines for when and how to report ratings, what to count as entry and exit, how to handle children who leave program in one community and move to another community within the state, expectations about who will be involved (including parent involvement), when startup should begin and where, plans for ongoing quality assurance and opportunities to regroup and discuss what is working and what is challenging and what needs clarifying, how exit from Part C links (or doesn’t) to entry of Part B preschool, expectations about whether or not age for rating should include correction for prematurity (ECO Center recommends that it does not), how this fits with family outcomes data collection, and so on….If it is a train the trainer session, there may be time in the day to talk about tips and expectations for training others and implementing…
Suggest providing trainees with powerpoints, overview, introduction, and COSF forms, expanded detail about each of the 3 child outcomes (April, 2005 document from ECO on child outcomes, developed for recommendation of them to OSEP), decision tree, information about where to seek resources – information on the ECO center and information available from the ECO center website, resources on websites to review for more information on child development and appropriate age-expectations for children who are developing in typical patterns.

57. Group Sharing Questions? Needed clarifications?
Comments and reactions?
Recommendations to the state about efficient procedures for using the COSF?
What TA and/or other supports will you need?