Coverage error Survey Research and Design Spring 2006 Class #3

Survey Research and Design (Umbach) 2 Today’s objectives Understand coverage error and how to handle it or reduce it Explain application exercise #1 (handout) Work in groups to develop constructs, target population, sampling frame, and to discuss possible issues of coverage

Survey Research and Design (Umbach) 3 Survey Process Define Research Objectives Choose Mode of Collection Construct and Pretest Questionnaire Recruit and Measure Sample Code and Edit Data Make Postsurvey Adjustments Perform Analysis Choose Sampling Frame Design and Select Sample

Survey Research and Design (Umbach) 4 Quality perspective (Groves, et al., p. 48) Construct Measurement Response Edited Response Measurement Target Population Sampling Frame Sample Respondents Representation Postsurvey Adjustments Survey Statistic Validity Measurement Error Processing Error Coverage Error Sampling Error Nonresponse Error Adjustment Error YiYi yiyi y ip  _ y prw _ y rw _yr_yr _yc_yc _ys_ys _Y_Y

Survey Research and Design (Umbach) 5 What is coverage error? Gap between the target population and the sampling frame The result of not allowing all members of the survey population to have an equal or nonzero chance of being sampled for participation People that can never be sampled (telephone survey/no telephone) Exists before the sample is drawn

Survey Research and Design (Umbach) 6 Discussion question What do you do when your sampling frame misses your target population entirely or partially?  Redefine target population to fit frame better  Admit the possibility of coverage error in statistics describing original target population What should you do? What are the tradeoffs?

Survey Research and Design (Umbach) 7 Coverage properties of frames Four possible coverage problems:  Undercoverage  Ineligible units/overcoverage/foreign elements  Duplication  Clustering

Survey Research and Design (Umbach) 8 Example of coverage error The CIRP Freshman Survey  Survey overview Survey overview  Administration Administration  Instrument Instrument

Survey Research and Design (Umbach) 9 Exercise A day care center is conducting a survey of parents. They placed the notice and survey in each child’s folder. (Each child has a folder for communications from the center to the parents – parents are supposed to check the folder daily.) A school refuses to give me contact information for students. Using the current semester’s course catalog, I randomly select courses that list an instructor’s name, contact the instructors for permission to administer my survey when their course meets, and then administer the survey in each course where the instructor agrees. Please discuss with your partner any coverage error that the two surveys might have.

Survey Research and Design (Umbach) 10 Target populations and their frame issues Households and persons Customers, employees or members of an organization Organizations

Survey Research and Design (Umbach) 11 Dealing with coverage error Remember, coverage error affects your statistics, so it depends on  The proportion of population not covered  Difference in statistic between covered and not covered elements Strategies:  Thoroughly understand list generation Dillman’s five questions

Survey Research and Design (Umbach) 12 Practical questions to ask about lists (Dillman, 2000) Does the list contain everyone in the population? Does the list include names of people who are not in the study population? How is the list maintained and updated? Are the same sample units included on the list more than once? Does the list contain other information that can be used to improve the survey?

Survey Research and Design (Umbach) 13 Dealing with coverage error Multiple frame/mixed-mode design Filter questions CIRP Freshman Survey example Benefits of reducing coverage error have to be weighed against their costs

Survey Research and Design (Umbach) 14 Group project What is the objective of your survey? Describe your constructs. What is your target population? What is your sampling frame? What types of coverage issues might you expect?

Survey Research and Design (Umbach) 15 For next week… Sample design and sampling error Readings  Groves et al. Ch. 4  Urdan (psych reserves)  NO Lohr reading

Survey Research and Design (Umbach) 16 One-minute paper What was the most important thing you learned during this class What important question remains unanswered?