Building Capacity to Use Child Outcomes Data to Improve Systems and Practices 2018 DEC Conference

Child Outcomes Percent of children who demonstrate improved: Positive social emotional skills (including positive social relationships) Acquisition and use of knowledge and skills (including early language/ communication [and early literacy]) Use of appropriate behaviors to meet their needs

Child Progress- the % of children who: a. did not improve functioning b. improved functioning but not sufficient to move nearer to functioning comparable to same-aged peers c. improved functioning to a level nearer to same-aged peers but did not reach it (no change in trajectory) d. improved functioning to reach a level comparable to same-aged peers e. maintained functioning at a level comparable to same-aged peers

Two Summary Statements per Outcome: Summary Statement 1 is the percentage of children who made greater than expected growth. Summary Statement 2 is the percentage of children who exited at or above age expectations.

Child Outcomes Data: National perspective 2018 DEC Conference

State Approaches to Measuring Child Outcomes: FFY 2016-17 Part C (N=56) Part B/619 (N=59) COS 42 (75%) 41 (69%) (decrease-2) One tool statewide 8 (14%) 10 (17%) (increase+2) Publisher's online system 3 (5%) 6 (10%) Other 2 (3%)

Child Outcomes Summary (COS) Process Team reviews and summarizes across multiple sources of information (professional observation, parent report, evaluation results, etc). Team uses a 7-point rating scale to indicate child’s functional status on each of the outcomes 1 = Child does not yet show functioning expected of a child his or her age in any situation. Child’s functioning does not yet include immediate foundational skills upon which to build age- appropriate functioning. 7 = Child shows functioning expected for his or her age in all or almost all everyday situations that are part of the child’s life.

Child Outcomes Summary (COS) Process For OSEP reporting, ratings must be collected when child enters and exits the program. Some states collect more often.

National Child Outcomes Data for Children Exiting in 2016-17 Part C Early Intervention Part B Preschool Summary Statement 1 Summary Statement 2 Social Relationships 67 58 80 60 Knowledge and Skills 73 49 81 56 Action to Meet Needs 75 57 65 National Child Outcomes Data for Children Exiting in 2016-17 Note: Data are based on 46 Part C states and 43 Part B Preschool states. Only states with high quality data were included.

Who’s Here from Washington State Susan Franck, Part C Data Manager, Early Support for Infants and Toddlers, Washington State Department of Children, Youth, and Families Debi Donelan, Early Support for Infants and Toddlers Lead, King County Developmental Disabilities Division www.dcyf.wa.gov

State Context State Lead Agency: transitioned July 1, 2018 to Department of Children, Youth, and Families 24 Local Lead Agencies (LLAs) Statewide: 17,658 eligible infants and toddlers (ages 0-3) and their families received services from July 1, 2017-June 30, 2018 King County: 5,036 eligible infants and toddlers and their families received services from July 1, 2017-June 30, 2018 13 Contracted Early Intervention Provider Agencies Statewide Improvement Plan focused on increasing social-emotional outcomes www.dcyf.wa.gov

Participants State Lead Agency Local Lead Agencies Families Local Lead Agencies State Lead Agency Early Intervention Providers www.dcyf.wa.gov

Why Prioritize Child Outcomes? Data analysis identified the following: Unexpected patterns related to entry ratings Inconsistent understanding of the Child Outcome Summary (COS) process Inconsistent involvement of families in the COS process www.dcyf.wa.gov

Wanted to Develop a Culture of Data Use Had data reports available through our web based system that locals were not using Did not have a system in place to support local data use Were not using COS data for program improvement www.dcyf.wa.gov

Systems to Support Data Use Data System Working on a revision to the data system to improve the quality of the data collected Personnel/Workforce Child Outcome Summary (COS) modules Technical assistance to Local Lead Agencies Training on engaging families in the COS process Developed statewide early intervention competencies www.dcyf.wa.gov

Systems to Support Data Use Accountability and Quality Improvement Required local lead agencies to complete the Local Child Outcomes Measurement Framework as part of their self-assessment. http://ectacenter.org/eco/assets/pdfs/L-COMS_Framework.pdf www.dcyf.wa.gov

Fun with Data: In-Person Kickoff www.dcyf.wa.gov

Developed an evaluation to see where each LLA was related to: Finding Reports in the Data System Understanding Report Content Interpreting & Analyzing Data Monitoring Data Quality Using Data for Program Improvement

Quarterly Call Topics October 2016 Reviewed quarterly call process Provided orientation to all COS reports Reviewed COS reports Progress codes and summary statements Entry by Exit report Progress codes by Race/Ethnicity www.dcyf.wa.gov

Quarterly Call Topics January 2017 Developed resource: https://www.dcyf.wa.gov/sites/default/files/pdf/esit/COS_Review_Sheet_0.pdf LLAs demonstrated understanding of the COS process and reports Data activity- LLAs charted their progress category percentages for each outcome for the year comparable to these data for the state www.dcyf.wa.gov

Quarterly Call Topics April 2017 Reviewed COS resource: guiding questions and activity template https://www.dcyf.wa.gov/sites/default/files/pdf/esit/Guiding_questions_for_data_analysis.pdf Data activity: walked through the guiding questions with COS reports https://www.dcyf.wa.gov/sites/default/files/pdf/esit/Data_activity_template.pdf www.dcyf.wa.gov

Sample Progress Code and Summary Statement-Statewide

Sample Progress Code and Summary Statement-Service Area

Questions What do you notice about the progress categories? What do you notice about the summary statement data? How do the local data compare to the state data? Is there anything you see in the data that concerns you? www.dcyf.wa.gov

Quarterly Call Topics July 2017 Sent self-guided data analysis worksheet for LLAs to complete prior to the call http://dcyf.wa.gov/sites/default/files/pdf/esit/DataActivity-SelfGuided.pdf During calls, reviewed each LLAs responses to the self-guided questions State team identified themes of types of questions from LLAs www.dcyf.wa.gov

Quarterly Call Topics May 2018 DCYF/ESIT data team ran query to pull COS entry scores for all children with an active IFSP between January 1, 2016-September 30, 2017 Reviewed COS entry scores using guiding questions www.dcyf.wa.gov

Entry Scores

Evaluating the Impact of System Improvements Challenges/lessons learned: Identifying a learning progression for data skills. There can be multiple people on a call who have different levels of comfort with data. Difficult to figure out how to estimate growth for the program. www.dcyf.wa.gov

Percentage of LLA Who Improved from Call 1 to Call 4

Next Steps Support locals in replicating data activities with their early intervention providers. Revise the data management system to streamline reporting and better meet the needs of the local administrators. www.dcyf.wa.gov

Reflection What is an your action step you can take to improve child outcome data quality and use? www.dcyf.wa.gov

Contact Information Susan.franck@dcyf.wa.gov ddonelan@kingcounty.gov www.dcyf.wa.gov

Thank You Department of Children, Youth, and Families www.dcyf.wa.gov