Successful Student Surveys: Lessons from the Field Merle Feldbaum Rackham Institutional Research November 8, 2015

Lessons to Share Stratification and probability sampling as a key to obtaining a representative sample. Incentives and communication strategies to get students to participate. Response analyses are important to learn more about who responds and understand any biases in the sample.

Define the population of interest Sampling Define the population of interest Consider the important subgroups and characteristics of the population Use stratified sampling to insure subgroup representation

Stratification for Representation of Groups Stratification will vary the rates of selection for important population subgroups to insure that the sample will be representative of key characteristics of the population If a group is 10% of the population, the group will be 10% of the sample. Take information you know about the population and put it to work for you

Example – PhD Experience Survey Stratification - Rackham Division (Biological & Health Sciences,Physical Sciences & Engineering,Social Sciences, Humanities & Arts) - Department - Candidacy Status - Gender - Citizenship Proc SurveySelect. Set sample size. Use random seed. Data is sorted by strata and then selection by systematic sample

Method for Sampling Always use a probability method so each case has a known probability of selection The process can be straightforward. In PhD experience survey, we used systematic sampling with stratification. Note about setting up survey - need to use panels, SSO, or verification so that there is some respondent identification captured. Important to be able to analyze who responds and able to manage reminders.

Use of Verification, SSO and/or Panels Importance of knowing who responds -For survey administration -Ability to use institutional data to merge on demographics, program, milestones -Surveys take less time However, may involve IRB approval Note about setting up survey - need to use panels, SSO, or verification so that there is some respondent identification captured. Important to be able to analyze who responds and able to manage reminders.

Getting Students to Participate Research on incentives show: -Some form of incentive is better than no incentive -Prepaid incentives are better than promised incentives -Cash is better than an in-kind incentive -Giving everyone a small incentive is better than giving everyone a chance at a larger amount (lottery) (The Science of Web Surveys, 2013)

Communication Strategy Pre-notification of the survey will encourage participation. However, our experience is that the communications should come from a relevant institutional leader to be effective. Personalization of the communication always encourages response.

Communications to be Sent Pre-notification Survey invitation with link to survey Reminders to only those who have not completed Reminders should be 3-4 days apart.

Response Analysis – PhD Experience Survey The logit results show that having a discontinued status as of 2010 decreases the likelihood of response. Those who have been in their program longer, are less likely to respond. In contrast, females and students in the humanities are associated with increases in the probability of response.

Predicted Probabilities of Response The qualities of the “median” respondent: Male, 5th year doctoral student, Non-Humanities Predicted probability of response: 72% - Female: +9% - Humanities +10% Predicted probabilities of “median” respondent varying only year in program: 3rd year: 80% 7th year (75th percentile): 67 %