Hawai’i January, 2008 Progress toward Measuring Goals in Early Intervention: What’s New from What Counts Kathy Hebbeler ECO at SRI International

Early Childhood Outcomes Center2 Objectives Review why data are being collected Describe national trends Identify and address challenges to good data Discuss some preliminary data from Hawai’i

Early Childhood Outcomes Center3 Why are we doing this?

Early Childhood Outcomes Center4 Keeping our eye on the prize: High quality services for children and families that will lead to good outcomes.

Early Childhood Outcomes Center5 High Quality Data on Outcomes Data are a piece of a system that helps to achieve overarching goals for children and families Data yield Findings that can be interpreted as having a particular meaning that should lead to specific actions to improve the system.

Prof’l Development Preservice Inservice System for Producing Good Child and Family Outcomes Good Federal policies and programs Good State policies and programs High quality services and supports for children 0-5 and their families Good outcomes for children and families Good Local policies and programs Adequate funding Strong Leadership

Early Childhood Outcomes Center7 The Vision: Using Data as a Tool for Program Improvement Hawai’i will have quality data available on an ongoing basis about multiple components of the system Goals for children and families Services provided Personnel (types, qualifications, etc.) Etc.

Early Childhood Outcomes Center8 Driving Force for Data on Child Goals Comes from the Federal Level Government Performance and Results Act (GPRA) Government Performance and Results Act (GPRA) Program Assessment Rating Tool (PART) Program Assessment Rating Tool (PART) Individuals with Disabilities Education Act (IDEA) Individuals with Disabilities Education Act (IDEA)

Early Childhood Outcomes Center9 Requires goals and indicators be established for IDEA Indicators and data collection further along for school age population than for EC Previously, for early childhood data had been collected on: Number of children served (Part C) Settings (both Part C and 619) Government Performance and Results Act (GPRA) passed in 1993

Early Childhood Outcomes Center programs examined in 2002; 50% programs had no performance data Programs looking at inputs, not results Part C and Section 619  No long-term child outcome goals or data  Need to develop a strategy to collect annual performance data in a timely manner OSEP: PART evaluation results (2002)

Early Childhood Outcomes Center14 Federal Funding for Early Intervention Total U.S.Hawai’i 2004$444,362,700$2,177, $440,808,096$2,160, $436,399,920$2,138,714

Early Childhood Outcomes Center15 SEC > MONITORING, TECHNICAL ASSISTANCE, AND ENFORCEMENT. ``(a) Federal and State Monitoring.- ….. ….. ``(2) Focused monitoring.--The primary focus of Federal and State monitoring activities described in paragraph (1) shall be on-- ``(A) improving educational results and functional outcomes for all children with disabilities; Individuals with Disabilities Education Act

Early Childhood Outcomes Center16 Where are we now: Federal reporting requirements

Early Childhood Outcomes Center17 OSEP Reporting Requirements: the Goals Positive social emotional skills (including positive social relationships) Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) Use of appropriate behaviors to meet their needs

Early Childhood Outcomes Center18 OSEP Reporting Categories Percentage of children who: a.Did not improve functioning b.Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c.Improved functioning to a level nearer to same- aged peers but did not reach it d.Improved functioning to reach a level comparable to same-aged peers e.Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 “measures” = 15 numbers

Early Childhood Outcomes Center19 Reporting Schedule Reported February 2007 Entry information: Age expected? Yes, No One time requirement Reported for children entering between July 1, 2005, and June 30, 2006 Due February 2008 Data in reporting categories at exit for all children who have been in the program for at least 6 months Must be reported for the year beginning July 1, 2006 Repeat with next year’s data in 2009, etc.

Early Childhood Outcomes Center20 Also States are required to Make public data reported to OSEP Analyze state data by program (i.e., compute a through e for each program) Make public the data by program

Age in Months

Early Childhood Outcomes Center25 Point of clarification “Why are we comparing children with delays and disabilities to typically developing children?”

Early Childhood Outcomes Center26 Point of clarification Process is NOT about comparing groups of children – it IS about asking how close children are to being able to do what is expected at their age Early learning guidelines Kindergarten and access to the general curriculum

Source: National Early Intervention Longitudinal Study

Early Childhood Outcomes Center28 Where are we now: State decisions and activities

Early Childhood Outcomes Center29 To respond to federal reporting requirements To meet provider/teacher, local and/or state need for outcome information and to respond to federal reporting requirements Purpose WHY?

Early Childhood Outcomes Center30 State approaches Most states have embraced outcomes measurement and are collecting outcomes data for their own purposes. Many states are building bigger systems than needed to produce the federal data. Go to for more information about what other states are doing

Early Childhood Outcomes Center31 How are states collecting child outcomes/goal data? Possible state approaches to collection of child data Child Outcomes Summary Form (COSF) [= Early Intervention Child Goals Summary Form in HI] Publisher’s online assessment system Single assessment statewide Other approaches

Early Childhood Outcomes Center32 State approaches to measurement for Part C child outcomes 40 states using the ECO Child Outcomes Summary Form (COSF) 40 states using the ECO Child Outcomes Summary Form (COSF) 8 states using 1 assessment tool statewide 8 states using 1 assessment tool statewide 3 states using on-line assessment systems with the capacity to report OSEP data reports 3 states using on-line assessment systems with the capacity to report OSEP data reports 5 states using other unique approaches 5 states using other unique approaches

Early Childhood Outcomes Center34 Variations across states in CGSF implementation Some states started early (HI); some did not start until mid to late 2007 Some states completing at IFSP; others at a separate meeting Some states including parents in the discussion; some are not

Early Childhood Outcomes Center35 Where states are now First data on 5 categories due to OSEP February 3 Many states do not have data on many children yet Many states focusing on improving the process of collecting the data

Early Childhood Outcomes Center36 What do we know so far: Positive impacts of the goals rating process

Early Childhood Outcomes Center37 Positive impacts reported by states Increases focus on functional outcomes on IFSPs Easier to write functional outcomes on IFSP Facilitates communication with parents

Early Childhood Outcomes Center38 Benefits of discussing the 3 goals “Requires us to talk & think in terms of functional behaviors, not test items Incorporates the parents as active and knowledgeable participants Looks at all settings and situations Bridges the gap between assessment tools and real life.” From presentation by Sandi Harrington, Norfolk, VA Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center39 Benefits “Is more meaningful to families Prepares the family for setting IFSP outcomes – thinking about the skills they want their child to have to function in their daily family life Guides us towards discipline-free contextualized goals.” From presentation by Sandi Harrington, Norfolk, VA Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center40 Benefits of INCLUDING families “Determining child progress requires we use the family’s expertise and knowledge of their child across setting and situations Our discussion becomes more inclusive with the family as an equal source of information for assessment purposes.” From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center41 Benefits of INCLUDING families “One of the biggest shifts in practice, for many systems, was the move to compare their children in Part C to their same age peers. Looking to children in the frame of same age peers allows us to have authentic, honest discussions with families about their child’s strengths and needs.” From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center42 Benefits of INCLUDING families “We need to be comfortable with reporting strengths AND areas of delay, while being family friendly.” From presentation by Sandi Harrington, Norfolk Infant Development Program, at the OSEP EC Meeting, December 2007

Early Childhood Outcomes Center43 What do we know so far: Challenges to getting good information

Early Childhood Outcomes Center44 Need for good data Encompasses all three levels: federal, state, local Depends on how well local programs are implementing procedures

Early Childhood Outcomes Center45 What we are learning nationally The process of training for child outcomes data collection has uncovered other areas of significant need related to professional development.

Early Childhood Outcomes Center46 Essential Knowledge for Completing the Child Goals Summary Form Between them, team members must: 1.Know about the child’s functioning across settings and situations 2.Understand age-expected child development 3.Understand the content of the three child outcomes 4.Know how to use the rating scale 5.Understand age expectations for child functioning within the child’s culture

Early Childhood Outcomes Center47 Important point It is not necessary that all team members be knowledgeable in all 5 areas Especially, no expectation that parents understand the rating scale or typical child development But the professionals have to!

Early Childhood Outcomes Center48 Providers need to know more about: Assessment How to gather assessment data to reflect functioning across settings and situations, especially how to gather child functioning information from families Understanding the results of the assessment Sharing assessment results sensitively and honestly with families

Early Childhood Outcomes Center49 Providers need to know more about: Functional outcomes What are they? How do they differ from outcomes organized around domains? What do they mean for how professionals from different disciplines operate as a team? Typical child development What are the functional expectations for children at different ages with regard to each of the 3 goal statements?

Early Childhood Outcomes Center50 Is this process too subjective to produce good data? Best practices in assessment requires looking at multiples sources of information Assessment as a tool vs. assessment as a process Research on judgment-based assessment indicates it is as good or better than traditional assessment

Early Childhood Outcomes Center51 What is informed opinion? Clinical judgment (informed opinion) – knowledgeable perceptions of caregivers and professionals about the elusive and subtle capabilities of children in different settings

Early Childhood Outcomes Center52 Clinical judgment provides good data when…. Operational definition of child characteristics to be judged Structured format for quantifying characteristics Information from multiple setting and individuals Training in methods that structure and quantify characteristics Decision making based on consensus From Bagnato, Smith-Jones, Matesa & McKeating- Esterle, 2006

Early Childhood Outcomes Center54 Ratings clarification Highest category (Completely, 7) = Child functions in an age appropriate manner across settings and situations Next highest (6) – Child functions in an age appropriate manner but there is a significant concern about some aspect of the child’s functioning

Early Childhood Outcomes Center55 Ratings clarification Somewhat (5) = Child shows a MIX of age appropriate and not age appropriate behaviors across settings and situations Between emerging and somewhat (4) – Child shows some age appropriate behavior but rarely

Early Childhood Outcomes Center56 Ratings clarification Emerging (3) – No age appropriate behavior yet. Shows immediate foundational skills in some to all settings and situations Between not yet and emerging (2) – No age appropriate behavior yet. Rarely uses immediate foundational skills (but does show some).

Early Childhood Outcomes Center57 Ratings clarification Not yet (1) – No age appropriate behavior yet. No immediate foundational skills yet.

Early Childhood Outcomes Center58 Should the rating be whatever parents want? No, the rating is a team consensus Need to think about what and how parents are being involved in this process Have the assessment results been thoroughly explained? Like so much in EI, the rating requires a partnership

Early Childhood Outcomes Center59 Thinking about relating to adults, relating to other children, and (for those older than 18 months) following rules related to groups or interacting with others. Δ How does the child relate to his/her parent(s)? Δ How does the child relate to other relatives or extended family and close family friends (e.g., grandparents, aunts, extended kin, etc.)? Do these interactions with people differ depending on the setting the child is in with these people? Δ How does the child interact with familiar caregivers (e.g., child care providers, babysitters)? Δ How does the child relate to strangers? At first? After a while? In different settings and using different approaches? Δ How does the child interact with/respond to people in community settings (e.g., park, library, church, grocery store, with neighbors on walks, at the bus stop, in restaurants, at playgroups or outings, etc.)? …. ECO Discussion Prompts: Child has positive social relationships (see ECO Tools)

Early Childhood Outcomes Center60 Obtaining good data Threats to good data Local providers do not understand the procedures Local providers do not follow the procedures And others….. Process requires good training procedures Initial Ongoing

Early Childhood Outcomes Center61 Many steps to ensuring quality data Before Information sharing Good data collection/Training Good data system During Ongoing supervision Feedback Refresher training After Monitoring Validity analyses

Early Childhood Outcomes Center62 Initial Data from Hawai’i

Early Childhood Outcomes Center63 These data are very, very preliminary.

Early Childhood Outcomes Center65 EIS Average Ratings at Initial IFSP Social- Emotional Skills Acquire and Use Knowledge and Skills Appropriate Action EIS (N=1774)

Early Childhood Outcomes Center67 HS Average Ratings at Initial IFSP Social- Emotional Skills Acquire and Use Appropriate Action HS (N=1311) 6.4

Appropriate Action Review Rating Initial_ 3B total Review Total

Another State: OSEP Categories for Goal 1 by Exit Status OSEP Categ ALL Exited a1% b5% c39% d44% e12% N=893

Another State: OSEP Categories for Outcome 1 by Exit Status OSEP Categ Exited Before 3 Exited at 3 ALL Exited a1% b2%6%5% c19%45%39% d57%39%44% e22%9%12% N=

Another State: OSEP Categories for Outcome 1 by Exit Status OSEP Categ Completed IFSP Exited Before 3 Exited at 3 ALL Exited a0%1% b 2%6%5% c12%19%45%39% d65%57%39%44% e22% 9%12% N=

Early Childhood Outcomes Center81 Questions to ask Do the data make sense? Am I surprised? Do I believe the data? Believe some of the data? All of the data? If the data are reasonable (or when they become reasonable), what might they tell us?

Early Childhood Outcomes Center82 Validity Validity refers to the use of the information Does evidence and theory support the interpretation of the data for the proposed use? Or Are you justified in reaching the conclusion you are reaching based on the data? Standards for Educational and Psychological Testing (1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education

Early Childhood Outcomes Center83 How will/might these data be used? Federal level Overall funding decisions (accountability) Resource allocation (e.g., what kind of TA to fund?) Decisions about effectiveness of program in individual states State level Program effectiveness?? Program improvement?? Local level Program improvement??

Early Childhood Outcomes Center84 What has ECO learned after 4+ years?

Early Childhood Outcomes Center85 ECO Message: Strong commitment States are committed to building good systems to collect data on how children are progressing Variations in how data are being collected Variations in how states plan to use the information Common thread: Widespread recogniton of the importance of the data

Early Childhood Outcomes Center86 ECO Message: Need to build state capacity Implement oversight procedures around data quality Examine data for validity Analyze and interpret data for program improvement Develop messages for policy-makers, public, media, families from the data

Early Childhood Outcomes Center87 ECO Message: Need to build provider capacity Assessment Functional outcomes Typical child development

Early Childhood Outcomes Center88 ECO Message: Need for better early childhood assessment tools Designed around the 3 functional outcomes for all children Designed to capture child functioning in a variety of setting and situations Designed to be used in accountability and program evaluation Current tools are antiquated: Need to incorporate latest research, recommended practices, psychometrics

Early Childhood Outcomes Center89 ECO Message: Need for more resources to ensure quality data National resources to support and coordinate across states: Training needs Analysis and use of data Support for states to continue to develop and validate their systems Investment in research to examine how outcomes data collection being carried out (impact on quality, local practice, etc.) Investment in new assessment tools

Early Childhood Outcomes Center90 Change can be difficult….

Early Childhood Outcomes Center91 For more information….

Early Childhood Outcomes Center93 health/eis/whatcounts.html

Early Childhood Outcomes Center95