Early Childhood Outcomes Center Measuring Progress: Improved Outcomes for Children and Families Kathy Hebbeler The Early Childhood Outcomes Center SRI International Virginia EI Conference Creating Connections: Strengthening Partnerships between Families and Providers Roanoke, VA, March 2007

Early Childhood Outcomes Center 2 Objectives Review history and rationale related to the selection of child indicators Describe what we have learned about implementing measurement systems Describe why child indicators are important for improved services

Early Childhood Outcomes Center 3 Goal of the ECO Center Promote the development and implementation of child and family outcome measures for infants, toddlers and preschoolers with disabilities that can be used in local, state, and national accountability systems

Early Childhood Outcomes Center 4 Being clear… For VA, the word “indicator” refers to the 3 OSEP outcomes to distinguish them from IFSP outcomes ECO Centers materials use the word “outcomes” to refer to both I will try to use “indicator” for the OSEP outcomes unless I mean all kinds of outcomes

Early Childhood Outcomes Center 5 What are Outcomes?

Early Childhood Outcomes Center 6 The Concept of Outcomes An outcome is the result of some action or set of actions Often expressed in the form of a statement Examples: All passengers will wait 10 minutes or less in the airport security line. All children will be fully immunized by 2.

Early Childhood Outcomes Center 7 The Concept of Outcomes Two parts: The expected result The action(s) that produces the result May not be stated but always present Outcomes are the “effect” in cause and effect

Early Childhood Outcomes Center 8 Outcomes Closer to Home EI programs provide high quality services and supports EI services are family centered Families help their child develop and learn Children take appropriate action to meet their needs

Early Childhood Outcomes Center 9 Different Kinds/Levels of Outcomes Program or systems outcomes EI programs provide high quality services and supports EI services are family centered EI services are coordinated What are the actions/activities that produce these results?

Early Childhood Outcomes Center 10 Different Kinds/Levels of Outcomes Child or family outcomes Families know their rights and advocate effectively for their children Children have positive social relationships What are the actions/activities that produce these results?

Supports Preservice training Inservice training Logic Model for Child and Family Outcomes Good Federal policies and programs Good State policies and programs High quality services and supports for children 0-5 and their families Good outcomes for children and families Good Local policies and programs Adequate funding Strong Leadership

Early Childhood Outcomes Center 12 Two Critical Outcome/Indicator Questions for Building High Quality Programs What is the intended result? What is the action/set of action that will produce the intended result?

Early Childhood Outcomes Center 13 How Looking at Child Indicators for EI Became Important

Early Childhood Outcomes Center 14 Focus on Accountability Followed an era of focusing on and measuring actions/activities Call for interest in the “ultimate result”: child and family outcomes/indicators Cut across both the public and private sectors Funders demanded data to determine whether a program was doing what it was supposed to do

Early Childhood Outcomes Center 15 Critical Events in EI Accountability 1992 –Osborne and Graebler, Reinventing Government 1993 – GPRA (Government Performance and Results Act) passed Intervening years… – PART finds there are no data on outcomes for Part C 2003 – OSEP begins to ask states for EI child outcome data (and funds the ECO Center!) 2005, 2006 – OSEP releases revisions to the reporting requirements

Early Childhood Outcomes Center 16 PART Review for Part C and Part B Preschool Results Not Demonstrated Part C “While the program has met its goal relating to the number of children served, it has not collected information on how well the program is doing to improve the educational and developmental outcomes of infants and toddlers served.” Part B Preschool “The Department has no performance information on preschool children with disabilities served by this program.” Read more at Expectmore.gov

Early Childhood Outcomes Center 17 Intervening Years Special Education – National study found poor outcomes for special education adolescents Results Push to include students with disabilities in statewide assessment systems Early Childhood Debate about whether child outcomes should be measured at all Discussion of all the problems in trying to measure outcomes for young children with disabilities

Early Childhood Outcomes Center 18 Measuring Child Indicators for EI The PART findings put an end to the debate about whether or not to do it Unfortunately, almost no progress had been made in the intervening years as to HOW to do it

Early Childhood Outcomes Center 19 Meanwhile: Re-thinking Assessment in Early Childhood Major changes in last 15 years in how assessment of young children is viewed Old position: Do not test little kids New position: Ongoing assessment is part of a high quality early childhood program

Early Childhood Outcomes Center 20 What changed New and different tools became available Curriculum-based assessments were developed General EC: Tools for 3-5 came first; 0-3 tools are coming now Interesting sidebar: Curriculum-based assessments for programs serving children 0-5 with disabilities have been around for years

Early Childhood Outcomes Center 21 What changed The purpose of assessment was redefined Not about: sorting, labeling, using to deny access (“the unready”) Now about: Getting a rich picture of what children can do and can’t do and using that information to help them acquire new skills “progress monitoring”

Early Childhood Outcomes Center 22 What changed Assessment had always been seen as a process with multiple purposes Distinctions were made between good and bad uses of assessment with young children Good uses are now promoted For more information: NAEYC web site (Position statement on Curriculum, Assessment and Evaluation)

Early Childhood Outcomes Center 23 Interesting Irony Even though the disability community had developed many curriculum-based assessment tools, currently **many** programs do not practice ongoing assessment Assess for eligibility only The push for ongoing assessment to monitor how a child is doing and plan for instruction/intervention is coming from the general education community

Early Childhood Outcomes Center 24 Where We Are Now Push for measuring outcomes for accountability (top down – but not just federal) Push for ongoing assessment of progress (monitoring outcomes) as best practice in EC programs Done well this could be a powerful blending of forces to improve outcomes for children

Early Childhood Outcomes Center 25 Why These 3 Indicators?

Early Childhood Outcomes Center 26 OSEP Reporting Requirements: Child Indicators Positive social emotional skills (including positive social relationships) Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) Use of appropriate behaviors to meet their needs

Early Childhood Outcomes Center 27 Origin of the Child Indicators ECO Center stakeholder meetings followed by public comment period (2004) First, collected themes and ideas Then, drafted and re-drafted indicator wording Made recommendation to OSEP (2005)

Early Childhood Outcomes Center 28 Themes from the Stakeholders: Child Indicators Consistent with IDEA and legislative intent Reflect what EI and ECSE are trying to do One set for birth to 5 Reflect what is known about development and learning

Early Childhood Outcomes Center 29 Themes from the Stakeholders Be compatible with best practice (esp. transdisciplinary service models, functional behaviors) Do not base them on domains Have potential to influence practice in a positive way Incorporate universal design Be readily understood

Early Childhood Outcomes Center 30 Make Outcomes/Indicators Functional Functional refers to things that are meaningful to the child in the context of everyday living Refers to an integrated series of behaviors or skills that allow the child to achieve the outcomes They are not a single behavior, nor are they the sum of a series of discrete behaviors

Early Childhood Outcomes Center 31 Functional What does a child typically do? Actual performance across settings and situations How child uses his/her skills to accomplish tasks Not the child’s capacity to function under unusual or ideal circumstances

Early Childhood Outcomes Center 32 Recommendations Decision: One set of indicators birth to 5 Decision: Functional outcomes Decision: Global, not specific

Early Childhood Outcomes Center 33 The need for an introductory statement Family and child outcomes are linked There are overarching goals for children and family that cut across the outcomes Helpful to frame the outcomes with these overarching goals

Early Childhood Outcomes Center 34 Concepts in the introduction: Goal for children The ultimate goal is for young children to be active and successful participants now and in the future in a variety of settings– in their homes, in their child care, preschool or school programs, and in the community

Early Childhood Outcomes Center 35 Concepts in the introduction: Goal for children Active and successful participants Now and in the future In variety of settings

Early Childhood Outcomes Center 36 Concepts in the introduction: Goal for families Enable families to provide appropriate care for their child Have resources they need to participate in community activities

Early Childhood Outcomes Center 37 Concepts in the introduction: Effective programs Support families in their quest for a satisfactory quality of life Provide needed supports and services in a timely fashion ***Remember: Outcomes are the result of actions***

Early Childhood Outcomes Center 38 Concepts in the preface: Outcomes and accountability Outcomes determined by variety of factors Not all families and children will achieve all outcomes BUT, system should still strive to achieve them

Early Childhood Outcomes Center 39 ECO Family Outcomes Understand their child’s strengths, abilities, and special needs Know their rights and advocate effectively for their children Help their children develop and learn Have support systems Access desired services, programs, activities in their community

Early Childhood Outcomes Center 40 The OSEP Reporting Categories

Early Childhood Outcomes Center 41 OSEP Reporting Categories Percentage of children who: a.Did not improve functioning b.Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c.Improved functioning to a level nearer to same-aged peers but did not reach it d.Improved functioning to reach a level comparable to same-aged peers e.Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers

Early Childhood Outcomes Center FunctioningFunctioning

Early Childhood Outcomes Center 43 Entry 14

Early Childhood Outcomes Center 44 EntryExit 15

Early Childhood Outcomes Center 45 EntryExit 16

Early Childhood Outcomes Center 46 CISF Numbers to OSEP Reporting Categories Each possible combination of numbers (and answer to the “b” question at entry and exit generates a line, i.e., a developmental trajectory) Each trajectory corresponds to an OSEP category Taking measurements more frequently than entry and exit generates more powerful information

Early Childhood Outcomes Center FunctioningFunctioning

Early Childhood Outcomes Center 48 OSEP Reporting Categories: Child Outcomes Percentage of children who: a.Did not improve functioning b.Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c.Improved functioning to a level nearer to same-aged peers but did not reach it d.Improved functioning to reach a level comparable to same-aged peers e.Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers

Early Childhood Outcomes Center 49 Comparing to Same Aged Peers Deficit model? Not individualized? Goal of EI: Active and successful participation now and in the future Kindergarten, school readiness, having friends, community participation Setting high expectations Moving from 1 to 2 = moving from 4 to 5 How much progress is enough?

Early Childhood Outcomes Center 50 COSF/CISF Implementation Issues

Early Childhood Outcomes Center 51 Overall Child indicators have been very well received Many states have embraced outcomes measurement at some level

Early Childhood Outcomes Center 53 Assessing functioning Process helps teams think about the child functionally across a variety of settings and situations Requires good assessment practices (and suggests need for improving current practice) Assessment only at entry. What is the role of assessment in EI? Moving from domains to functioning

Early Childhood Outcomes Center 54 Assessing functioning Providers need a way to have a conversation with families about child functioning across settings and situations Providers need training in explaining assessment results to parents “soft pedal”

Early Childhood Outcomes Center 55 Age Expectations Requires providers understand typical child development and how children of different ages are expected to function in all 3 indicators Reveals need for training in this area

Early Childhood Outcomes Center 56 Working with Families Changing the conversation between providers and families? Changing the nature of IFSPs? “Do we write IFSPs to these indicators?” Would that be such a bad thing?

Early Childhood Outcomes Center 57 Opportunity? Measurement of indicator data can encourage more frequent collection of assessment data Progress monitoring Improve practice… And provide outcome indicators

Early Childhood Outcomes Center 58 Validity and Reliability Validity is not a characteristic of a tool Validity is a characteristic of a set of scores produced by a tool CISF is judgment-based Substantial research to support that with defined criteria, providers and parents can reach reliable judgments

Early Childhood Outcomes Center 59 Validity and Reliability ECO working on procedures to help states assess validity of their data Good training and ongoing supervision will be critical Quality assurance will be critical

Early Childhood Outcomes Center 60 Some Concerns Being implemented very quickly Not all implementers are well trained Few states have procedures in place for assuring the process is high quality and reliable States doing the minimum will not have very useful data

Early Childhood Outcomes Center 61 How Will These Data Be Used? Hopefully not at all -- UNTIL checks are in place to insure they are of high quality Meaning needs to be attached to the findings by those closest to the program Craft your own message

Early Childhood Outcomes Center 62 Next steps for ECO More support and training materials More guidance on supervision and quality assurance More guidance on using data for program improvement

Early Childhood Outcomes Center 63 A Vision (What Does This Mean for You)

Supports Preservice training Inservice training Logic Model for Child and Family Outcomes Good Federal policies and programs Good State policies and programs High quality services and supports for children 0-5 and their families Good outcomes for children and families Good Local policies and programs Adequate funding Strong Leadership

Early Childhood Outcomes Center 65 Building a high quality system What if every piece of the system functioned as it was supposed to? What information is needed to know if this is happening? What information is needed to know how to identify and address weaknesses in the system?

Early Childhood Outcomes Center 66 Your contribution Where do you fit? What is your contribution to good outcomes? What actions do you take that produce good child and family outcomes? What information would you need to know if your actions were producing good outcomes?

Early Childhood Outcomes Center 67 The ideal outcome system gives persons at each level of the system the information to know if they are producing good outcomes -and if not, to change their practices/policies/procedures so eventually they will.

Early Childhood Outcomes Center 68 Remember the goal for children Active and successful participation now and in the future

Early Childhood Outcomes Center 69 And an overarching goal for families To enable families to provide care for their child and have the resources they need to participate in their own desired family and community activities

Early Childhood Outcomes Center 70 The Take Home Message Measuring outcomes is not primarily about data It is about doing good things for children and families And using data as a tool to help programs and providers to know whether what they are doing is making a difference

Early Childhood Outcomes Center 71 Finding out more Additional information about OSEP requirement and state activities New resources will be coming including training materials, resources on assessment and typical child development, and materials for parentswww.the-eco-center.org