1 We Changed the COSF The Flip The Skills The Decision-Tree Questions

2 The Flip We flipped source info and overall skill rating. I.e., the source info (assessment, parent, teacher observations, etc.) is at the top of the page; skill rating at the bottom.

3 Reason for The Flip We want to make sure that providers base their ratings on the data they’ve collected. We want providers to list their data, i.e., the relevant source information, and then arrive at a rating. We wanted to build the decision-making process that providers should follow onto the form itself.

4 The Skills We added a section that asks the provider to list the highest-level of Age-Appropriate (AA), Immediate Foundational (IF), or Foundational (F) skills the child demonstrates.

5 Reason for The Skills We want to make clear that the overall rating should relate to whether the child is demonstrating AA, IF, or F skills. The overall rating should not be solely tied (necessarily) to percentile scores and age-equivalent scores (for example) that might be listed under “Summary of Test Results.” Once again, we wanted to build the decision-making process that providers should follow onto the form itself.

6 The Decision-Tree Questions We added two questions that reflect the ECO Center Decision Tree to the COSF. 3. Think of the positive social-emotional age- appropriate skills the child demonstrated as indicated in the tables above. Did the child demonstrate these age-appropriate skills across all or almost all everyday situations (multiple settings)? (Circle one response) a Yes  7 or 6 b Somewhat  5c Rarely  4d Child demonstrated no age-appropriate skills (go to Q4)

7 The Decision-Tree Questions 4. Think of the positive social-emotional immediate foundational skills the child demonstrated as indicated in the tables above. Did the child demonstrate these immediate foundational skills across all or almost all everyday situations (multiple settings)? (Circle one response; If you answered a, b, or c for question 3, no need to answer) a Yes  3 b Somewhat  2 c No  1 d Child demonstrated no immediate foundational skills  1

8 Reason for the Decision-Tree Questions We want to make clear that the decision tree must be used to arrive at the rating, i.e., providers must consider the degree of AA, IF, F skills, and the extent to which they are demonstrating these skills across multiple settings. By building into the form, we’re not relying on providers to remember to use the decision tree or to locate a copy of the tree.

9 Additional reasons for COSF Changes We wanted to collect the skill info and the decision tree question info so that we could use this info to validate the ratings.

10 Additional Reasons continued... If a provider says (on the decision tree question) that a child is demonstrating age-appropriate skills some of the time, but gives the child a rating of “2” – then we know there’s a problem somewhere! If a provider says (on the decision tree question) that a child is demonstrating app-appropriate skills some of the time, and gives the child a rating of “5” (so far so good), BUT does not list any age-appropriate skills in the skills boxes – we know there’s a problem somewhere!

11 To Sum Up Re the Form Changes We wanted to incorporate the decision- making process into the form. We wanted to collect info on the form that would help us validate the ratings.

12 The COSF Data Submission Process Providers complete on paper and mail hard- copy to the DDD. September 2007: a Word document will be provided that will allow providers to type in information. Future: creating an online form as part of the DDD Early Childhood database system.

13 Are We Doing a Good Job? Are providers following the correct process in gathering data on child outcomes? Are providers correctly completing the COSF? Are providers making valid ratings?

14 Gathering Data - Are providers following the correct process in gathering data on child outcomes? Providers are encouraged to collect relevant data from parents. And they are. Providers are told to use one of several core assessments. And they are. Providers are told to use multiple sources of data. And they are.

15 Gathering Data Percent Collecting Relevant Info from Parents Part CPart B Positive Social-Emotional Skills89%90% Acquiring and Using Knowledge and Skills 77%80% Taking Appropriate Action to Meet Needs 83%85%

16 Gathering Data Percent Using a State-Approved Core Assessment Part CPart B Positive Social-Emotional Skills96%81% Acquiring and Using Knowledge and Skills 79% Taking Appropriate Action to Meet Needs 94%84%

17 Gathering Data Percent Using Multiple Sources of Information Part CPart B Positive Social-Emotional Skills96% Acquiring and Using Knowledge and Skills 89%90% Taking Appropriate Action to Meet Needs 94%91%

18 Completing the Form -- Are providers correctly completing the COSF? Providers are to give summary of actual test results. And they are. Providers are to list AA, IF, F skills. And they are. Providers are supposed to answer every question. And they are. For the most part! Most of the issues we have with incomplete forms is readily fixed by offering training on the COSF and by providing sample completed COSFs.

19 Percent who did not answer key questions: Decision tree questions: about 3.5% for each skill area for each of Part C and Part B Overall ratings: About 3% of Social Skills and Acquiring Knowledge and Skills; about 5.5% for Taking Appropriate Action Did child make progress questions (the question asked on exit form and used to determine improvement category) – about 40% blank for each skill area!

20 Tackling “Not Answering Questions” “Leaving questions blank” was addressed in training – Some providers thought the decision tree questions and improvement questions were optional. – Some providers thought that overall ratings were necessary for only the outcome area the in which the child had a problem.

21 Making Valid Ratings -- Are providers making valid ratings? The decision-tree questions are supposed to guide the decision-making process. The listed skills are supposed to guide the decision-making process. The summary of actual test scores is supposed to guide the decision-making process.

22 Making Valid Ratings The percent whose decision-tree questions matched the overall rating: about 67% on each skill area for each of Part B and Part C – More training is needed!

23 Making Valid Ratings Content Analysis: We have produced a document of the listed skills and their corresponding rating by age group so that a content analysis can be done to determine if providers are making an appropriate overall rating given the listed skills a child is demonstrating. We will also produce a document from this content analysis that will illustrate behaviors representative of different scale points at different age groups.

24 Making Valid Ratings Score Analysis: We will be correlating the actual test scores on the core assessments to the overall ratings to determine if this tells us anything valuable. Even though providers are supposed to use multiple sources of information and not rely solely on the test score, we would expect at least some relationship between test scores and overall ratings. We want to determine what the relationship is and if this has implications for technical assistance.

25 Bottom Line We want to make sure that providers are making valid ratings AND We want to make sure that the ratings are consistent across regions.