1. Successful Student Surveys: Lessons from the Field
Merle Feldbaum Rackham Institutional Research
November 8, 2015

2. Lessons to Share
Stratification and probability sampling as a key to obtaining a representative sample.
Incentives and communication strategies to get students to participate.
Response analyses are important to learn more about who responds and understand any biases in the sample.

3. Define the population of interest
Sampling
Define the population of interest
Consider the important subgroups and characteristics of the population
Use stratified sampling to insure subgroup representation

4. Stratification for Representation of Groups
Stratification will vary the rates of selection for important population subgroups to insure that the sample will be representative of key characteristics of the population
If a group is 10% of the population, the group will be 10% of the sample.
Take information you know about the population and put it to work for you

5. Example – PhD Experience Survey
Stratification
- Rackham Division (Biological & Health Sciences,Physical Sciences & Engineering,Social Sciences, Humanities & Arts)
- Department
- Candidacy Status
- Gender
- Citizenship
Proc SurveySelect. Set sample size. Use random seed. Data is sorted by strata and then selection by systematic sample

6. Method for Sampling
Always use a probability method so each case has a known probability of selection
The process can be straightforward. In PhD experience survey, we used systematic sampling with stratification.
Note about setting up survey - need to use panels, SSO, or verification so that there is some respondent identification captured.
Important to be able to analyze who responds and able to manage reminders.

7. Use of Verification, SSO and/or Panels
Importance of knowing who responds -For survey administration -Ability to use institutional data to merge on demographics, program, milestones -Surveys take less time However, may involve IRB approval
Note about setting up survey - need to use panels, SSO, or verification so that there is some respondent identification captured.
Important to be able to analyze who responds and able to manage reminders.

8. Getting Students to Participate
Research on incentives show:
-Some form of incentive is better than no incentive
-Prepaid incentives are better than promised incentives
-Cash is better than an in-kind incentive
-Giving everyone a small incentive is better than giving
everyone a chance at a larger amount (lottery)
(The Science of Web Surveys, 2013)

9. Communication Strategy
Pre-notification of the survey will encourage participation. However, our experience is that the communications should come from a relevant institutional leader to be effective. Personalization of the communication always encourages response.

10. Communications to be Sent
Pre-notification
Survey invitation with link to survey
Reminders to only those who have not completed
Reminders should be 3-4 days apart.

11. Response Analysis – PhD Experience Survey
The logit results show that having a discontinued status as of 2010 decreases the likelihood of response. Those who have been in their program longer, are less likely to respond. In contrast, females and students in the humanities are associated with increases in the probability of response.

12. Predicted Probabilities of Response
The qualities of the “median” respondent: Male, 5th year doctoral student, Non-Humanities Predicted probability of response: 72% - Female: +9% - Humanities +10% Predicted probabilities of “median” respondent varying only year in program: 3rd year: 80% 7th year (75th percentile): 67 %