POPULATION RESEARCH SEMINAR SERIES Sponsored by the Statistics and Survey Methods Core of the U54 Partnership Post Survey Adjustments Lee Hargraves, Ph.D. & Lois Biener, Ph.D. Center for Survey Research University of Massachusetts Boston

Statistical Areas of Concern Nonresponse is a problem if there are systematic differences in who responds to a survey, especially if it is related to key measures

Weight Adjustments: Post-stratification Need to know or estimate characteristics of the population you want to generalize to. For example, gender, race, age, Hispanic origin, education level, income level. Such estimates may come from Census data or ACS (American Community Survey) data. For health care surveys, know characteristics may come from administrative data, claims, or electronic health record data

Weight Adjustments: Post Stratification You need to measure these same characteristics in your survey. You can compare the distributions from your survey to the distributions from the “best” source (or “controls”).

Example 1: Statewide Medicaid Survey Race% among Respondents% among Sample Black White Other Source: A 2012 survey among members of Medicaid health plans, conducted by mail (3 contacts) and telephone. Spanish Language% among Respondents% among Sample Yes No Gender% among Respondents% among Sample Female Male

Example 1: Statewide Medicaid Survey Age% among Respondents% among Sample 18 – – – – – – or older Source: A 2012 survey among members of Medicaid health plans, conducted by mail (3 contacts) and telephone.

Example 1: Statewide Medicaid Survey Age% among Respondents% among Sample 18 – – – – – – or older Source: A 2012 survey among members of Medicaid health plans, conducted by mail (3 contacts) and telephone

Weight Adjustments: Post Stratification If they differ, can consider adjusting the weights to “make them match.” This is what is called post-stratification weight adjustment.

What is a Survey Weight? A value assigned to each case in the data file. Typically weights are used to make statistics computed from the data more representative of the population. Each case has a non-zero value that indicates how much it will count in statistical procedures. Examples: – A weight of 1 means that the case only counts as one case in the dataset. – A weight of 10 means that the case counts in the dataset as ten identical cases. – Weights can be a fraction, but are always positive and non- zero.

Example 2: Men may be less likely to respond to surveys Source: A statewide survey of 4608 adults (age 18 +) using random digit dialing (RDD) of landlines with a supplement of mobile phone surveys. To ensure that the sample represented the population in terms of age, gender, and ethnicity, data were weighted to match Census data. (2012, a state in New England) More accurate representation of the state’s men and women.

Example 2: Effect of differential non-response on key measures Source: A statewide survey of adults (age 18 +) using random digit dialing (RDD) of landlines with a supplement of mobile phone surveys. To ensure that the sample represented the population in terms of age, gender, and ethnicity, data were weighted to match Census data. (2012, a state in New England) More accurate representation of people with access problems.

Example 2: Effect of differential non-response on key measures Source: A statewide survey of adults (age 18 +) using random digit dialing (RDD) of landlines with a supplement of mobile phone surveys. To ensure that the sample represented the population in terms of age, gender, and ethnicity, data were weighted to match Census data. (2012, a state in New England) More accurate representation of the percent of adults who are obese.

How are weights calculated? A topic for a longer session – Post-stratification are one potential adjustment. – Non-response adjustments are another. – Both depend on the sample design. – Both are affected by the response rate. Best advice for non-statisticians: – Take a course on survey sampling. – Work with a statistician!

Conclusions Always beware of what you don’t know!! Post stratified weight adjustments do NOT always make estimates better. They can even make them worse. Ask questions, dig into data, evaluate, evaluate, evaluate !!!! Still, they frequently can improve estimates, but you must decide (with help from your friendly statistician) Leisure Activities Example

Working with survey organization You should ask those who collected your data: – Did they post-stratify weight adjust? – If so, what factors did they use in this adjustment? – Can they provide information about the size of the weight adjustment factors? – Better yet, can they place on the final data file, values of weights before and after the adjustment?

POPULATION RESEARCH SEMINAR SERIES Sponsored by the Statistics and Survey Methods Core of the U54 Partnership Questions? Comments? Send us an !