1. Are your C4 data reflective of the families you serve? Joy Markowitz, Director Jean Dauphinee, TA Specialist Measuring Child and Family Outcomes Conference, August 28, 2008

2. 2 Purpose of Collecting C4 data To figure out how to serve our families better specifically related to Indicator C4: Percent of families participating in Part C who report that early intervention services have helped the family: – A Know their rights; – B. Effectively communicate their children's needs; and – C. Help their children develop and learn.

3. Measuring Child and Family Outcomes Conference, August 28, 2008 3 Why do we care who responds to our surveys? We don’t want to invest time and money in improvement activities if data do not reflect the opinions and/or experiences of the families/children we serve. This is an issue if you collect data from all families (census) or a sample of families because not all families who receive a survey respond.

4. Measuring Child and Family Outcomes Conference, August 28, 2008 4 Population Population may be defined by the state and could be (for example): all Part C families all Part C families exiting in a given year, or all Part C families in program more than 1 year.

5. Measuring Child and Family Outcomes Conference, August 28, 2008 5 Other terms used Population Target group Target population Respondents Response Pool Respondent Pool Respondent Group

6. Measuring Child and Family Outcomes Conference, August 28, 2008 6 Nonresponse Bias Representativeness is a word we have been using. It’s not a statistical term. Correct term –nonresponse bias. Meaning: Are the families who did not respond different from the families that did?

7. Measuring Child and Family Outcomes Conference, August 28, 2008 7 What is a Sample? A sample is a subset of the population that you define. However, this sample must be derived using an approved sampling plan. OSEP collaborates with DAC to review and approve submitted sampling plans.

8. Measuring Child and Family Outcomes Conference, August 28, 2008 8 Questions about Response Rates What is high enough? How do you determine if your respondents are reflective of the population you defined? Whether using census or sampling, your return or response rate is unlikely to be as high as you want it to be; therefore, you must address nonresponse bias.

9. Measuring Child and Family Outcomes Conference, August 28, 2008 9 Example 1 Families served Black 30 White 50 AI/AN 20 Total100 Survey respondents Black 24 White 44 AI/AN 2 Total 70 Do these data reflect the population? Families served in 2006-2007 (n=100):

10. Measuring Child and Family Outcomes Conference, August 28, 2008 10 Example 2 Families served English 75 Other language 25 Total 100 Survey respondents English65 Other language 5 Total70 Do these data reflect the population? Families exiting in 2006-2007 (n=100):

11. Measuring Child and Family Outcomes Conference, August 28, 2008 11 Example 3 Families served by agencies Agency 1100 Agency 2100 Agency 3100 Agency 4100 Total400 Survey respondents Agency 1 49 Agency 2 63 Agency 3 66 Agency 4 21 Total 199 Do these data reflect the population? Families served in programs for at least one year (n=100):

12. Measuring Child and Family Outcomes Conference, August 28, 2008 12 State Data Collection Methods/Strategies Methods Census (n=41 states) Sample (n= 15 states) – oversampling populations that are known to be hard to reach. Analysis strategies Frequencies and percentages Weighting Sampling among respondents

13. Measuring Child and Family Outcomes Conference, August 28, 2008 13 Summary of C4 Findings (Based on analysis of 54 state 2006-2007 APRs) 67% (n= 36 states) reported nonresponse bias, 26% (n=14 states) did not. Among the 67% who reported nonresponse bias, most reported by one variable (e.g. race/ethnicity, region, gender, or child’s age)

14. Measuring Child and Family Outcomes Conference, August 28, 2008 14 Summary of C4 Findings (Based on analysis of 54 state 2006-2007 APRs) cont. Most states measured nonresponse bias by: Race/ethnicity or Race/ethnicity and other factors such as: a.length of time in program, b.region of the state, c.child’s age, or d.gender.

15. Measuring Child and Family Outcomes Conference, August 28, 2008 15 From SPP/APR Instruction Sheet States are allowed to use sampling when so indicated on the Part C Indicator Measurement Table. When sampling is used, a description of the sampling methodology outlining how the design will yield valid and reliable estimates must be submitted to OSEP. The description must describe the: (a) sampling procedures followed (e.g., random/stratified, forms validation); and (b) the similarity or differences of the sample to the population of children with disabilities in the early intervention program (e.g., how all aspects of the population such as disability category, race, age, gender, etc. will be represented).

16. Measuring Child and Family Outcomes Conference, August 28, 2008 16 From SPP/APR Instruction Sheet (cont.) The description must also include how the Lead Agency addresses any problems with: (1) response rates; (2) missing data; and (3) selection bias. Samples from EIS programs must be representative of each of the EIS programs sampled, considering such variables as eligibility definition (diagnosed condition or developmental delay), age, race, and gender.

17. Measuring Child and Family Outcomes Conference, August 28, 2008 17 From SPP/APR Instruction Sheet (cont.) In reporting on the performance of small EIS programs, the Lead Agency shall not report to the public or the Secretary any information on performance that would result in the disclosure of personally identifiable information about individual children or where the available data is insufficient to yield statistically reliable information, i.e., numbers are too small. Source: Part C State Performance Plan (SPP) and Annual Performance Report (APR) Instruction Sheet dated 10/19/2007.

18. Measuring Child and Family Outcomes Conference, August 28, 2008 18 Examples from States How have states tackled these issues? State representatives: Wendy Whipple, NV Sue Campbell and Rosanne Griff- Cabelli, DE Alice Ridgeway, CT

19. Measuring Child and Family Outcomes Conference, August 28, 2008 19 Questions ? What questions are you getting from OSEP that you need help to answer? What guidance do you need from OSEP? What are realistic expectations of states for reporting nonresponse bias?