1. Using COS Data to Inform Program Improvement at All Levels Every day, we are honored to take action that inspires the world to discover, love and nurture the greatness in all children.

2. Overview of modules Module 1: Setting the Stage: Global Child Outcomes and the SFP Module 2: Learning More About the SFP and COS Rating Module 3: Essential Knowledge for Completing the SFP and COS Rating Module 4: Guidance for Completing the SFP and COS Rating Within the Team Process Module 5: Understanding Age-expected Child Development, Developmental Trajectories, and Progress Module 6: Using COS Data to Inform Program Improvement: Part 1–Essentials for Providers Part 2–Essentials for Administrators Kids' Potential, Our Purpose

3. Module 6 – Part I Using COS Data to Inform Program Improvement: Essentials for Providers

4. The focus of this module: Using COS data for program improvement Revisit how the COS data from the SFP fits into continuous program improvement cycle Describe questions and approaches for looking at the COS data that the SFP produces Examine different types of data and some basic reports in the DMS so we can consider how to use this information for: quality assurance and program improvement Kids' Potential, Our Purpose

5. Why gather child outcomes information? To provide information for families who as consumers have the right to know if their child’s program is effective To identify strengths and weaknesses of programs and use the information to improve programs To provide information for policymakers to justify future funding Because the federal government requires it Kids' Potential, Our Purpose

6. SFP Information SFP gathered at the individual level Process is valuable Quantifies the impact of early intervention services Use information mostly at aggregate level Review combined information from many children to inform the program, region, state, and national levels Developmental gains achieved by children at the Local Lead Agency level are reflected in the state data Kids' Potential, Our Purpose

7. Everyone has a role in using COS data Kids' Potential, Our Purpose

8. Uses for SFP Information Individual Program or LLA level: Look at patterns of child progress within a program or LLA Provides a sense of program or LLA’s effectiveness and strengths and weaknesses for improvement Across LLA’s Statewide: Look at patterns of progress across LLA’s May inform policy makers, technical assistance providers, and to justify future funding Overall Statewide: Describe overall pattern of progress for all children in ESIT in WA Provides required data about the difference ESIT is making for the federal government Kids' Potential, Our Purpose

9. Quality checks and quality assurance Information for program improvement Kids' Potential, Our Purpose Reports for state use COS Data from the SFP Informs

10. Continuous Program Improvement Plan (vision) What actions will improve the program and help it best support all children and families? Implement Check (Collect and analyze data ) Reflect Are we where we want to be? What do these data tell us? Is there a problem? Why is it happening? What should be done? Is it being done? Is it working?

11. Knowledge Check 1. True or False: Checking COS data quality across team members is enough because these data are mainly used by the child’s IFSP team and family for their reference. 2. True or False: Working with COS data is mainly something that administrators do.

12. Knowledge Check 1. True or False: Checking COS data quality across team members is enough because these data are mainly used by the child’s IFSP team and family for their reference. Ensuring high quality COS data matters because COS ratings are the best way to look at how children are functioning at entry and exit from ESIT within the whole group of children in particular programs, regions, or statewide in ESIT. 2. True or False: Working with COS data is mainly something that administrators do. Everyone has a role in working with COS data – from gathering it, to checking it for quality to asking questions, to interpreting the information produced and thinking through appropriate action steps for program improvement. What your role is depends on your involvement within the organization and your program’s needs and focus.

13. E vidence I nference A ction Using data for program improvement = EIA 13

14. Evidence Evidence refers to the numbers. For example, in the area of social-emotional skills: – 43% of children statewide entered ESIT fully at age expectations (with an entry rating of 7) – 58% of children exiting ESIT statewide showed substantial increases in their rate of growth after participating in the program. The numbers are not debatable 14

15. Inferences 15 How do you interpret the numbers? Is this good news? Bad news? Hard to interpret? What other information will help us understand these numbers? What can you conclude from these numbers? Why do the data look this way? What does it mean?

16. Inference is debatable Reasonable people can reach different conclusions Stakeholders can help with putting meaning on the numbers Inference 16

17. Discussing Inferences It is critical to engage stakeholders at all levels in discussions Examples of stakeholders and opportunities to incorporate data Kids' Potential, Our Purpose

18. Reflection: Discussing Inferences from Data Take a moment to jot some notes to yourself. How comfortable are you with reviewing data and making inferences about the data you have? What would you need to increase your comfort making and discussing inferences with others? Who else might be a part of these discussions? People in your program/LLA? Other stakeholders? What kinds of questions might you bring to these discussions? Kids' Potential, Our Purpose

19. Discussing Inferences Data-driven discussions can take many forms: – incorporate snipets of data as background in a bigger conversation, or – focus a whole meeting on in-depth review of data Questions and steps: – Have someone fully explain the data. Do you understand what the numbers are? – React: Is it surprising? Expected? Does it make sense? Why? Or, why not? – What could account for the data presented? – What else do you want to know?

20. Two kinds of inferences: – Early on, inference about the quality or consistency of the data – Later, inferences about root causes or conclusions showing how well a specific practice is being implemented or how effective it is with a group Kinds of Inferences 20

21. 21 Questions to ask about data Why does the data look this way? What caused this? Does it make sense? – Am I surprised? – Do I believe the data? – Believe some of the data? – All of the data? Are these data reasonable? What do these data tell us about the program? Do I need more information or to hear from other perspectives to figure out what these data mean?

22. Action Given the inference from the numbers…. What should be done? Identify specific recommendations or action steps Action can be debatable – and often is… Another role for stakeholders 22

23. Action Two kinds of actions: – Early on, focus on improving the quality of the data – Later, as the quality of data improves, actions focus on program improvement strategies 23

24. Take Home Message Kids' Potential, Our Purpose Evidence Inferences about program improvement issues Inferences about data quality issues Actions: Focus on quality assurance and improving data quality Actions: If data quality is believable, then target actions for improvement areas identified

25. Aside about Quality Assurance Kids' Potential, Our Purpose Before Good training and assessment Efficient data systems Timely and accurate data entry During Ongoing supervision of implementation Feedback to implementers Refresher training After Review sample of completed SFP’s Examine data input in system Pattern checking analysis

26. Starting point… Kids' Potential, Our Purpose

27. Looking at Data on Groups of Children

28. Kids' Potential, Our Purpose

31. Example Entry COS ratings for all children entering ESIT served by all agencies in the “Strong County” Service Area Kids' Potential, Our Purpose

32. Numbers = evidence How well do the numbers shown represent the group? What else do you want to know?

33. Kids' Potential, Our Purpose

34. Entry COS rating percentages “Strong County” Service Area Kids' Potential, Our Purpose COS Entry Rating Percentage of ChildrenTotal n 1234567 Outcome 14.715.511.315.018.86.128.6213 Outcome 29.417.49.416.022.16.119.7213 Outcome 35.218.39.415.526.34.221.1213

35. Entry COS Ratings Evidence… the numbers, the patterns Inference… the reason, the explanation Action…what to do or change given the inference Kids' Potential, Our Purpose

36. Exit COS Ratings for a Group Reactions? Surprising? Why do they look this way? What might be causing it? What do the data mean to you? What implications does this have for action? Kids' Potential, Our Purpose COS Exit rating Percentage of Children Total N 1234567 Outcome 14.28.53.38.916.48.050.7 213 Outcome 23.36.12.412.213.68.054.5 213 Outcome 32.87.02.85.616.08.557.3 213

37. Practice using evidence, inference, action steps Kids' Potential, Our Purpose

38. Examples of ways providers might use outcomes data… Review data in the DMS for children on your caseload. Does it match your expectations? Why or why not? Bring data to share at your next assessment team or IFSP team meeting. Spend a few moments reviewing results, reacting to the data, and inferring what it means. Bring data to your next supervision meeting and consider possible implications for new things to try or changes in practice. Kids' Potential, Our Purpose

39. Reflection What data are you most interested in beginning to use? What supports do you need to be able to use your data more effectively? Who can you talk with to access that support? Kids' Potential, Our Purpose

40. Knowledge Check 1.Multiple Choice: When thinking about evidence, inference, action steps for working with data, the evidence represents: a. Articles or literature supporting specific practices. b. The numbers themselves, the data being reviewed. c. The section of the law related to the issue. d. The formula used to compute the data. 2. True or False: Because inferences are based on evidence, only one inference is appropriate or valid.

41. Knowledge Check 1.Multiple Choice: When thinking about evidence, inference, action steps for working with data, the evidence represents: a. Articles or literature supporting specific practices. b. The numbers themselves, the data being reviewed. c. The section of the law related to the issue. d. The formula used to compute the data. 2. True or False: Because inferences are based on evidence, only one inference is appropriate or valid.

42. Resources and Support Check out the following websites: http://ectacenter.org/eco/pages/quality_assurance.asp http://ectacenter.org/eco/pages/training_resources.asp http://ectacenter.org/eco/assets/pdfs/TrainingandTAResource sMarch2014.pdf http://ectacenter.org/eco/pages/selflearning.asp http://ectacenter.org/eco/pages/training_activities.asp Kids' Potential, Our Purpose

43. Additional information – ESIT Website under “Information for Providers” link on the ESIT homepage Child and Family Outcomes heading at: http://www.del.wa.gov/development/esit/training.aspx Kids' Potential, Our Purpose

44. Overview of modules Module 1: Setting the Stage: Global Child Outcomes and the SFP Module 2: Learning More About the SFP and COS Rating Module 3: Essential Knowledge for Completing the SFP and COS Rating Module 4: Guidance for Completing the SFP and COS Rating Within the Team Process Module 5: Understanding Age-expected Child Development, Developmental Trajectories, and Progress Module 6: Using COS Data to Inform Program Improvement: Part 1–Essentials for Providers Part 2–Essentials for Administrators Kids' Potential, Our Purpose

45. For more information contact your LLA or Program Consultant http://www.del.wa.gov/publications/esit/docs/ContactsDirectory.pdf