1. Inference and Error in Surveys
Global Development
26. Maj 2014
Inference and Error in Surveys
Monday 26 September 2016, 10:
Henrik Hansen
Department of Economics
Based on:
Groves et al. (2009),
Survey Methodology, 2nd ed.
Chapter 2 + 3

2. Agenda The Total Survey Error Framework
Global Development
Global Development
26. Maj 2014
Agenda
The Total Survey Error Framework
The individual steps and errors explained
The Sampling Frame and Coverage Error
Advanced Research Methods, Fall 2016

3. The Total Survey Error Framework
Global Development
Global Development
26. Maj 2014
The Total Survey Error Framework
A good survey designer seeks to minimize error in survey statistics by making design and estimation choices that minimizes the gap between two successive stages in the survey process.
Note:
Each error component has verbal descriptions and statistical formulations
The error components are properties of individual survey statistics, not of whole surveys
Advanced Research Methods, Fall 2016

4. From Construct to Measurement
Global Development
Global Development
26. Maj 2014
From Construct to Measurement
Constructs are the elements of information we seek
Values, perceptions, wellbeing
Poverty, financial status, employment, education
Measurements are our ways to gather information about the constructs
Physical measures (height, weight)
Questions posed to respondents (see Chapter 7)
Construct validity is the extent to which construct and measure are related
Advanced Research Methods, Fall 2016

5. Construct Validity in Psychometric Measurement Theory
Global Development
Global Development
26. Maj 2014
Construct Validity in Psychometric Measurement Theory
Subscript i indicates an individual
Subscript t indicates a trial:
For each individual we imagine we can repeat the measurement
Eit is the expectation (average) over all trials and all individuals
The correlation between construct and measurement is a simple measure of construct validity
In most cases this is purely theoretical
Advanced Research Methods, Fall 2016

6. From Measurement to Response
Global Development
Global Development
26. Maj 2014
From Measurement to Response
Measurements are our ways to gather information about the constructs
Physical measures (height, weight)
Questions posed to respondents
Response is the actual measure provided by the respondent
The measured height
The response
Measurement error is the departure from the true value of the measurement as applied to a sample unit and the value provided
Advanced Research Methods, Fall 2016

7. From Response to Edited Response
Global Development
Global Development
26. Maj 2014
From Response to Edited Response
Response is the actual measure provided by the respondent
The measured height
The response
Edited Response the recorded response in the cleaned data. Data cleaning “corrects” atypical patterns
The measured height may be corrected for decimal error or other mistyping
The response may be edited or recoded (say, to missing)
Processing effect is the difference between the response as provided and the response as edited
The edited responses are the actual data you are given from a survey producer. (The numbers in your spreadsheet)
In many cases you will perform further data cleaning (i.e., more editing)
Advanced Research Methods, Fall 2016

8. Edited Responses from the European Social Survey (ESS7)
Global Development
Global Development
26. Maj 2014
Edited Responses from the European Social Survey (ESS7)
Advanced Research Methods, Fall 2016

9. The Measurement Process in Practice
Global Development
Global Development
26. Maj 2014
The Measurement Process in Practice
You will conduct the Copenhagen Infrastructure Study (CIS 2016)
Today you will move from construct to measures as you design a survey questionnaire
The background is your reading of Chapter 7, Huppert et al. (2009).
You can get more inspiration about wellbeing questions from
Advanced Research Methods, Fall 2016

10. Back to the top: From Target Population to Sampling Frame
Global Development
Global Development
26. Maj 2014
Back to the top: From Target Population to Sampling Frame
The Target Population is the set of units to be studied
The population in Tanzania
Manufacturing firms in Mozambique
HIV infected individuals in Vietnam
The adult population in Copenhagen,
The Sampling Frame (The Frame Population) is the set of target population members that have a chance to be selected into the survey sample
Coverage error is the difference in the statistic of interest of the target population and the statistic of interest in the covered population
Advanced Research Methods, Fall 2016

11. From Frame Population to Sample
Global Development
Global Development
26. Maj 2014
From Frame Population to Sample
The interesting problem is not the sampling error but the computation of the relevant statistic
The Sampling Frame (The Frame Population) is the set of target population members that have a chance to be selected into the survey sample
The Sample is the group of elements (individuals, firms) actually selected for measurement. The sample is much smaller than the frame population
Sampling error is the difference between the sample statistic and the population statistic. This is the only error we add deliberately (to save money)
Advanced Research Methods, Fall 2016

12. From Sample to Respondents
Global Development
Global Development
26. Maj 2014
From Sample to Respondents
The Sample is the group of elements (individuals, firms) actually selected for measurement. The sample is much smaller than the frame population
The respondents are the elements actually (partially) providing measures. (We are now close to the left leg of the flow as the responses are given by the respondents)
Non-response error is the difference between the statistic of interest when based on the sample and on the responses
Advanced Research Methods, Fall 2016

13. From Respondents to Post Survey Adjustments
Global Development
Global Development
26. Maj 2014
From Respondents to Post Survey Adjustments
The respondents are the elements actually (partially) providing measures. (We are now close to the left leg of the flow as the responses are given by the respondents)
The Postsurvey Adjustments are adjustments of the computations by reweighting or imputation. We either rescale or fill our missing items
Adjustment error is the difference between the statistic of interest when based on the responses and the adjustments
Advanced Research Methods, Fall 2016

14. The Long Road to Results
Global Development
Global Development
26. Maj 2014
The Long Road to Results
Psychology+Anthropology+Sociology+Economics +HRM
HRM+Theory
Theory + context
The individual data records in our database are (hopefully) related to our concept.
The summaries and other statistics of the data records are (hopefully) related to summaries and statistics in our population of interest
We need to be aware of the sources of error in order to minimize them
Theory + Statistics
Theory + Statistics
Statistics
Theory + Context
Advanced Research Methods, Fall 2016

15. Target Populations, Sampling Frames, and Coverage Error
Global Development
Global Development
26. Maj 2014
Target Populations, Sampling Frames, and Coverage Error
Advanced Research Methods, Fall 2016

16. From Target Population to Sampling Frame
Global Development
Global Development
26. Maj 2014
From Target Population to Sampling Frame
The Target Population: the set of units to be studied
The Sampling Frame: the set of target population members that have a chance to be selected into the survey sample
Undercoverage: Elements of the target population that do not appear in the sampling frame.
Ineligible units: Elements in the sampling frame that are not part of the target population
Clustering: Multiple mappings of frame to population
Duplication: Multiple mapping of population to frame
Coverage error: the difference in the statistic of interest of the target population and the statistic of interest in the covered population
Advanced Research Methods, Fall 2016

17. The Coverage Error Math
Global Development
Global Development
26. Maj 2014
The Coverage Error Math
Advanced Research Methods, Fall 2016

18. Example: Minority Poverty in Vietnam
Global Development
Global Development
26. Maj 2014
Example: Minority Poverty in Vietnam
Source:
Advanced Research Methods, Fall 2016

19. Example 2: People with disabilities in Tanzania
Global Development
Global Development
18 April 2018
Example 2: People with disabilities in Tanzania
The survey was conducted in Mbeya, Tanga and Lindi, three geographically dispersed districts in Tanzania where CHF was functional, between August and September 2014.
District
Population
(Census 2012)
Mbeya
2,707,410
Tanga
2,045,205
Lindi
864,652
Tanzania, Mainland
44,928,923
U/N
0,875
Advanced Research Methods, Fall 2016

20. Therefore, we often work with flexible sampling frames
Global Development
Global Development
26. Maj 2014
If the Sampling Frame is wrong, there is no way to “correct” the survey results down the line.
Therefore, we often work with flexible sampling frames
Area Sampling with updating of frames during data collection
Multiplicity Sampling
Multiple Frame Designs
Advanced Research Methods, Fall 2016

21. The Population and Sampling Frame for CIS 2016
Global Development
Global Development
26. Maj 2014
The Population and Sampling Frame for CIS 2016
We are to conduct studies to assess the perception of the metro construction project’s influence on urban livelihood and the quality of life in the vicinity of the new metro stations? We can start by looking at it at We have data on the population in Copenhagen in data at Q: What is the Target Population?
Advanced Research Methods, Fall 2016

22. Global Development
Global Development
26. Maj 2014
The Seminar today
Today’s task is to start formulating a questionnaire.
Each group specifies research questions, constructs and measurements
After 1 hour, groups are gathered in pairs (or 3 groups) to discuss and select the better constructs and measurements
After 30 minutes (+ a break) the discussion is broadened to the classes
After (at most) 1 hour each class must have
Specific ideas for core constructs and measurements
Specific ideas for interesting background information about the target population that must be obtained
Each class must select two people to be members of the Survey Management Team (SMT) for CIS The SMT will have the overall responsibility for the final questionnaire and the time schedule (with responsibility follows power)
Advanced Research Methods, Fall 2016

23. Global Development
Global Development
26. Maj 2014
Next Week
Sampling designs and design based estimators Lecture: Sampling frames and design based estimators The lecture introduces Sampling frames and the sampling error. The focus is on understanding best practices, when they exist. The sampling error is given special treatment as this is directly related to statistical analyses of survey data, including statistical analyses of secondary data. The close relation between the survey design and the optimal estimators of means and ratios is stressed. Groves et al. (2009), Survey Methodology, 2nd ed. Chapter 4 Seminar: Students finalize the CIS questionnaire and do pilot testing.
Advanced Research Methods, Fall 2016