1. Involving Families
Early Childhood Outcomes Center

2. Informing Families
What are you doing to inform families about the data collection?
Why it is occurring
What it involves
What it means for them and their child
Early Childhood Outcomes Center

3. www.the-eco-center.org Federal and State Activities State Activities
Early Childhood Outcomes Center

4. Early Childhood Outcomes Center

5. Preparing Families
Helping families be active participants in the discussion
What is working?
What is not working?
General principle: Families need to know what to expect
Early Childhood Outcomes Center

6. What Do We Expect from Families
Yes - That they will be able to provide rich information about their child’s functioning across settings and situation
Maybe but not necessarily – That they will know whether their child is showing age appropriate behavior
Early Childhood Outcomes Center

7. Involving Families in a Conversation about Their Child
Avoid jargon
Avoid questions that can be answered with a yes or no
“Does Anthony finger feed himself?”
Ask questions that allow parents to tell you what they have seen
“Tell me about how Anthony eats”
Early Childhood Outcomes Center

8. Involving Families in a Conversation about Their Child
What is working?
What is not working?
Early Childhood Outcomes Center

9. Involving Families in the Rating Discussion
What % of families are participating?
How are you doing this?
What is working?
What is not working?
Early Childhood Outcomes Center

10. Families’ Right to COSF Information
All families have a right to know what ratings have given to their child -- and to the records containing the information.
Early Childhood Outcomes Center

11. Quality Review of Discussion
Activity
Including Families in the Rating Discussion
Early Childhood Outcomes Center

12. Looking for Quality Data
I know it is in here
somewhere
Early Childhood Outcomes Center

13. Activity What factors work to improve the quality of your data?
What factors work to lessen the quality of your data?
How to address these factors?
Early Childhood Outcomes Center

14. Take Home Message
If you conclude the data are not (yet) valid, they cannot be used for program effectiveness, program improvement or anything else.
What do you if the data are not as good as they should be?
Answer: Continue to improve data collection through ongoing quality assurance
Early Childhood Outcomes Center

15. Many steps to ensuring quality data
Before
Good data collection/Training
Good data system and data entry procedures
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review of COSF records
Data analyses for validity checks
Representativeness of the responses.
Early Childhood Outcomes Center

16. Many steps to ensuring quality data
Before
Good data collection/Training
Good data system and data entry procedures
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review of COSF records
Data analyses for validity checks
Representativeness of the responses.
Early Childhood Outcomes Center

17. Promoting quality data
Training and support before and during data collection
Analysis of the data after data collection
Data system and verification after data collection
Early Childhood Outcomes Center

18. Promoting Quality Data
Through training and communication related to:
Assessment
Understanding the COSF process
Age expectations
Data entry
Early Childhood Outcomes Center

19. Promoting Quality Data
Through training materials, such as
Video team and child examples
Written child examples
“Quizzes” for ensuring learning
Refresher trainings –
Beware of Drift!!
Early Childhood Outcomes Center

20. Promoting Quality Data
Through data systems and verification, such as
Data system error checks
Good data entry procedures
Early Childhood Outcomes Center

21. Many steps to ensuring quality data
Before
Good data collection/Training
Good data system and data entry procedures
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review of COSF records
Data analyses for validity checks
Representativeness of the responses.
Early Childhood Outcomes Center

22. Ongoing supervision Review of the process Methods
Is the process high quality?
Are teams reaching the correct rating?
Methods
Observation
Videos
Early Childhood Outcomes Center

23. Quality Review of COSF Team Discussion
1. Do all team members participate in the discussion?
2. Is parent input considered in the rating? Give examples.
3. Is the team documenting the rating discussion? Give examples.
4. Does the team discuss multiple assessment sources? What are they?
Early Childhood Outcomes Center

24. Quality Review of COSF Team Discussion
5. Does the team describe the child’s functioning, rather than just test scores? Give examples.
6. Does the discussion include the child’s full range of functioning, including skills and behaviors that are age appropriate, immediate foundational, and leading to immediate foundational? Give examples.
Early Childhood Outcomes Center

25. Quality review through process checks
Provider surveys
Self assessment of competence
Knowledge checks
Process descriptions (who participates?)
Identification of barriers
Kansas survey
Alaska’s survey
Early Childhood Outcomes Center

26. Questions from Alaska’s Survey
3. How would you rate your own level of proficiency with the COSF process? (please select only one)
I am confident I know how to do it, and I do it well
I know how to do it, but I need some more practice and assistance
I understand it to a point, but I need more training
I do not know how to do this yet
4. Some cases are different from others, but of the choices below, which process seems to be the most typical in your experience? (please select only one)
I gather information and determine COSF ratings on my own
I gather information and consult with another provider to determine COSF ratings
I gather information and consult with the family to determine COSF ratings
I gather information, discuss it with a team and the team determines the COSF ratings

27. Ongoing Supervision Feedback to teams is critical Refresher training
Beware of
Auto pilot
Drift
Early Childhood Outcomes Center

28. Quality Review of COSF Team Discussion
Activity
Observe team video
Evaluate quality
Early Childhood Outcomes Center

29. Many steps to ensuring quality data
Before
Good data collection/Training
Good data system and data entry procedures
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review of COSF records
Data analyses for validity checks
Representativeness of the responses.
Early Childhood Outcomes Center

30. Quality Review of Completed COSFs
Is the COSF complete?
Is there adequate evidence for the basis for the rating?
Does the evidence match the appropriate outcome area?
Is the evidence based on functional behaviors?
Early Childhood Outcomes Center

31. Quality Review of Completed COSFs
Is there evidence that the child’s functioning across settings and situations considered?
Are the ratings consistent with the evidence?
Early Childhood Outcomes Center

32. Review completed COSF with errors
Quality Review of COSF
Activity
Review completed COSF with errors
Early Childhood Outcomes Center

33. Promoting quality data through data analysis
Examine the data for inconsistencies
If/when you find something strange, look for other data that might help explain it.
Is the variation caused by something other than bad data?
Early Childhood Outcomes Center

34. The validity of your data is questionable if…
The overall pattern in the data looks ‘strange’
Compared to what you expect
Compared to other data
Compared to similar states/regions/school districts
Early Childhood Outcomes Center

35. COSF Ratings - Fall Rating Class 1 Class 2 Class 3 Class 4 1 3 2 4 5 67
Early Childhood Outcomes Center

36. Outcome 3: Appropriate Action
Spring
Fall 1 2 3 4 5 6 7
total 9 26 15 14 27 19 83 21 39 28 12
108 71 86 48
232 63
136 18 23 56 99
Review Total 13 38 60
185
207
186
691

37. Outcome 3: Appropriate Action
Spring
Fall 1 2 3 4 5 6 7
total 9 26 15 14 27 19 83 21 39 28 12
108 71 86 48
232 63
136 18 23 56 99
Review Total 13 38 60
185
207
186
691

38. Outcome 3: Appropriate Action
Spring
Fall 1 2 3 4 5 6 7
total 9 26 15 14 27 19 83 21 39 28 12
108 71 86 48
232 63
136 18 23 56 99
Review Total 13 38 60
185
207
186
691

39. Outcome 3: Appropriate Action
Spring
Fall 1 2 3 4 5 6 7
total 9 26 15 14 27 19 83 21 39 28 12
108 71 86 48
232 63
136 18 23 56 99
Review Total 13 38 60
185
207
186
691

40. OSEP Categories OSEP Categories (%) 23 16 24 15 13 32 34 37 28 21 25 2
Class 1 (%)
Class 2
(%)
Class 3 (%)
e. Maintained Age Appro Trajectory 23 16 24
d. Changed Traj – Age Appro 15 13
c. Changed Traj – Closer to Age Appropriate 32 34 37
b. Same Trajectory -Progress 28 21 25
a. Flat Trajectory – No Prog. 2 6 1
Early Childhood Outcomes Center

41. Questions to ask Do the data make sense?
Am I surprised? Do I believe the data? Believe some of the data? All of the data?
If the data are reasonable (or when they become reasonable), what might they tell us?
Can’t use data for program improvement until you believe them.
Early Childhood Outcomes Center

42. Validity Or Validity refers to the use of the information
Does evidence and theory support the interpretation of the data for the proposed use? Or
Are you justified in reaching the inference you are reaching based on the data?
Standards for Educational and Psychological Testing (1999) by American Educational Research Association, American Psychological Association, National Council on Measurement in Education
Early Childhood Outcomes Center

43. The validity of your data is questionable if:?
Early Childhood Outcomes Center

44. ….not all providers are not knowledgeable about in the COSF process
…not all providers are careful with the COSF process
…the data look “strange”
…etc.
Early Childhood Outcomes Center

45. Many steps to ensuring quality data
Before
Good data collection/Training
Good data system and data entry procedures
During
Ongoing supervision of implementation
Feedback to implementers
Refresher training
After
Review of COSF records
Data analyses for validity checks
Representativeness of the responses.
Early Childhood Outcomes Center