1. The Social Science Research Unit (SSRU) Stephanie Kane, Project Manager Barbara Foltz, Unit Manager J.D. Wulfhorst, Director

2. Evolution of the SSRU History as a “Service Unit” History as a “Service Unit” Started from the ground up (’89 – ’99) Started from the ground up (’89 – ’99) Era of re-development (’00 – ‘05) Era of re-development (’00 – ‘05) Modernization of technology Modernization of technology Trained professional staff and management Trained professional staff and management Expertise in multiple methodologies Expertise in multiple methodologies Self-sustaining for over 80% of our operations Self-sustaining for over 80% of our operations New tier of projects beginning in 2006 New tier of projects beginning in 2006

3. SSRU Outcomes & Benefits Scientifically rigorous survey methodology intended to yield high response rates and statistically valid results Scientifically rigorous survey methodology intended to yield high response rates and statistically valid results SSRU assists clients in SSRU assists clients in study design, sampling methodologies, and writing survey instruments; study design, sampling methodologies, and writing survey instruments; conducting all or components of studies; and conducting all or components of studies; and analyzing and summarizing the results of projects analyzing and summarizing the results of projects

4. SSRU Facilities & Laboratory 1,200 sq ft survey laboratory 1,200 sq ft survey laboratory Phone bank with 14 CATI stations using SPSS Data Entry Software Phone bank with 14 CATI stations using SPSS Data Entry Software Automated Voice Recording System with a dedicated server Automated Voice Recording System with a dedicated server Data analysis using SPSS and SAS statistical software packages Data analysis using SPSS and SAS statistical software packages Interview staff primarily UI students Interview staff primarily UI students

5. Project Methodologies Telephone and Mail Surveys Telephone and Mail Surveys Dillman method for survey research Dillman method for survey research Random sample of households Random sample of households Typically 50% - 65% response rates Typically 50% - 65% response rates Person-to-Person Surveys Person-to-Person Surveys Internet Surveys Internet Surveys Focus Groups—Qualitative research Focus Groups—Qualitative research Also used in conjunction with mail or phone surveys for mixed methodologies Also used in conjunction with mail or phone surveys for mixed methodologies

6. What is the Dillman Method? Scientifically tested method for survey research Scientifically tested method for survey research Involves introductory hand-signed letter (mail) or postcard (phone) with real stamps Involves introductory hand-signed letter (mail) or postcard (phone) with real stamps Multiple follow-ups: several mailings (mail) or phone calls (phone) Multiple follow-ups: several mailings (mail) or phone calls (phone) Can involve incentives Can involve incentives

7. Terminology Population: The entire group the researcher is interested in learning something about (e.g. Idaho residents) Population: The entire group the researcher is interested in learning something about (e.g. Idaho residents) Sample: The smaller group of units that are actually measured Sample: The smaller group of units that are actually measured Parameter: The summary measure for an entire population (the “true value”) Parameter: The summary measure for an entire population (the “true value”) Statistic: The summary measure computed from sample data (the estimate) Statistic: The summary measure computed from sample data (the estimate)

8. Types of Experiments Randomized experiments Randomized experiments Subjects randomly assigned to treatment Subjects randomly assigned to treatment Typical in medical, biological, chemical, engineering, physical fields Typical in medical, biological, chemical, engineering, physical fields Observational studies Observational studies Subjects randomly selected from population Subjects randomly selected from population Typical in social science and marketing research Typical in social science and marketing research

9. Types of Sampling Simple random sample—every unit in the population has equal chance of being in the sample Simple random sample—every unit in the population has equal chance of being in the sample Stratified sample—the population is separated into non-overlapping groups, and a simple random sample is drawn from each group Stratified sample—the population is separated into non-overlapping groups, and a simple random sample is drawn from each group Cluster sample—each sampling unit is a cluster of elements Cluster sample—each sampling unit is a cluster of elements

10. Simple Random Sample

11. Stratified Random Sampling

12. Cluster Sampling

13. Sources of Error in Survey Research Errors of non-observation Errors of non-observation 1. Sampling error 2. Coverage bias 3. Non-response bias Errors of observation (measurement error) Errors of observation (measurement error)

14. Errors of Non-Observation Sampling error: deviations of sample statistic from population parameter. Can be reduced by good sampling and appropriate sample size. Can be measured by confidence intervals. Sampling error: deviations of sample statistic from population parameter. Can be reduced by good sampling and appropriate sample size. Can be measured by confidence intervals. Coverage bias: selected sample is not representative of targeted population. Can be reduced by good sampling. Coverage bias: selected sample is not representative of targeted population. Can be reduced by good sampling. Non-response bias: error resulting from non-response of members of the sample. Can be reduced or controlled for by follow-ups and sub-sampling of non- respondents Non-response bias: error resulting from non-response of members of the sample. Can be reduced or controlled for by follow-ups and sub-sampling of non- respondents

15. Sampling Error Sampling error a result of not taking a complete census (which is often unpractical or impossible) Sampling error a result of not taking a complete census (which is often unpractical or impossible) Statistics are used to place bounds on the error—we can be reasonably confident that true population parameter falls in a certain range Statistics are used to place bounds on the error—we can be reasonably confident that true population parameter falls in a certain range National studies usually +/- 3% National studies usually +/- 3%

16. Coverage Bias Sample does not adequately reflect population Sample does not adequately reflect population Often a result of an incomplete frame (sometimes unavoidable) Often a result of an incomplete frame (sometimes unavoidable) Equine survey case study Equine survey case study

17. Calculating Response Rate C = Completed survey PC = Partially completed survey (has unit non-response) R = Refusals NC = Non-contacts U = Unknown eligibility ** Individuals who are ineligible for a study (because they are deceased, or not in the population of interest) are removed from the sample

18. Non-Response Bias Unit non-response Unit non-response Unable to reach respondent Unable to reach respondent Cell-phone use case study Cell-phone use case study Respondent refused survey Respondent refused survey Item non-response Item non-response Respondent refused question (usually sensitive question) Respondent refused question (usually sensitive question)

19. Errors of Observation Interviewer effects: intonation, staying on script, gender effects Interviewer effects: intonation, staying on script, gender effects Respondent effects: “social desirability”--not understanding a question or not wanting to tell the interviewer they don’t know Respondent effects: “social desirability”--not understanding a question or not wanting to tell the interviewer they don’t know Question effects: question order, question wording, option order Question effects: question order, question wording, option order Method of data collection, transcription or data entry errors Method of data collection, transcription or data entry errors

20. Reducing Survey Error Improving coverage: Improving coverage: Multi-frame studies (e.g. landline and cell phone lists) Multi-frame studies (e.g. landline and cell phone lists) Multi-mode studies (e.g. phone and mail) Multi-mode studies (e.g. phone and mail) RDD RDD Improving response Improving response Increased number of call backs or mailings Increased number of call backs or mailings Incentives or rewards Incentives or rewards Pre-post card Pre-post card

21. Testing for Non-Response Bias Call a subset of the non-respondents and see if they differ on key questions from respondents Call a subset of the non-respondents and see if they differ on key questions from respondents Compare sample demographics to census (population) demographics Compare sample demographics to census (population) demographics Compare “hard to reach” respondents to “easy to reach” respondents Compare “hard to reach” respondents to “easy to reach” respondents

22. Accounting for Non-Response Weighting of observations Weighting of observations “Hot Deck” method: random replacement of item non-response from respondents (could be done by strata or demographic variables) “Hot Deck” method: random replacement of item non-response from respondents (could be done by strata or demographic variables) Maximum likelihood Maximum likelihood

23. Reducing Measurement Error Well trained interviewers Well trained interviewers Data checks and proofs Data checks and proofs Pre-testing survey to test for question bias, order bias, or confusing questions Pre-testing survey to test for question bias, order bias, or confusing questions Rule-of-thumb: Less than 5% should respond “I don’t know” Rule-of-thumb: Less than 5% should respond “I don’t know”

24. Project Examples Crime Victimization (Idaho State Police) Crime Victimization (Idaho State Police) Equine Census in Idaho (ID Horse Council/Horse Board) Equine Census in Idaho (ID Horse Council/Horse Board) Youth 4-H (Extension) Youth 4-H (Extension) Green Industry of Idaho (INLA, economic impact study) Green Industry of Idaho (INLA, economic impact study) Food Thermometer Use (UI – Food & Consumer Science) Food Thermometer Use (UI – Food & Consumer Science) School Facilities (Moscow School District) School Facilities (Moscow School District) Farm Experiential Learning (UI/WSU Extension & Rural Roots, Inc.) Farm Experiential Learning (UI/WSU Extension & Rural Roots, Inc.) Students with Disabilities (UI – Human Resource services) Students with Disabilities (UI – Human Resource services) Ext. Critical issues: dairy wastes, forestry management training, mint production Ext. Critical issues: dairy wastes, forestry management training, mint production

25. Suggested Resources Utts, J.M. and R.F. Heckard. 2004. Mind on Statistics. Thompson Learning, Inc. Belmont, CA Utts, J.M. and R.F. Heckard. 2004. Mind on Statistics. Thompson Learning, Inc. Belmont, CA Dillman, D.A. 1978. Mail and Telephone Surveys: The Total Design Method. John Wiley & Sons, Inc. NY, NY Dillman, D.A. 1978. Mail and Telephone Surveys: The Total Design Method. John Wiley & Sons, Inc. NY, NY Scheaffer, R.L., W. Mendenhall III, and R.L. Ott. 1996. Elementary Survey Sampling, 5 th Ed. Duxbury Press, NY, NY. Scheaffer, R.L., W. Mendenhall III, and R.L. Ott. 1996. Elementary Survey Sampling, 5 th Ed. Duxbury Press, NY, NY. Levy, P.S. and S. Lemeshow. 1991. Sampling of Populations: Methods and Applications. John Wiley & Sons, Inc. NY, NY. Levy, P.S. and S. Lemeshow. 1991. Sampling of Populations: Methods and Applications. John Wiley & Sons, Inc. NY, NY.