Pre-Conference Workshop ELI06 January 29, Student and Faculty Surveys Paul R. Hagner Associate Program Director EDUCAUSE Learning Initiative Copyright Paul R. Hagner, This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Pre-Conference Workshop ELI06 January 29, Agenda Why Use Surveys? Types of Surveys Types of Samples Response Bias Question Design Quasi-Experimental Designs Ethics

Pre-Conference Workshop ELI06 January 29, What do you want to know?

Pre-Conference Workshop ELI06 January 29, Finding out what you need to know Unobtrusive Measures

Pre-Conference Workshop ELI06 January 29,

6

7

8 Finding out what you need to know Unobtrusive Measures Obtrusive Measures Error

Pre-Conference Workshop ELI06 January 29, Surveys Done Right…. Good snapshot Benchmarking Accreditation Good basis for policy making

Pre-Conference Workshop ELI06 January 29, Surveys Done Wrong Time well wasted Inaccurate results Easy to dismiss Dangerous to base policy decisions upon

Pre-Conference Workshop ELI06 January 29, Types of Survey Delivery

Pre-Conference Workshop ELI06 January 29, Paper Surveys

Pre-Conference Workshop ELI06 January 29, In-Person Surveys

Pre-Conference Workshop ELI06 January 29, Mail Surveys

Pre-Conference Workshop ELI06 January 29, Phone Surveys

Pre-Conference Workshop ELI06 January 29, Surveys

Pre-Conference Workshop ELI06 January 29, Web-Based Surveys

Pre-Conference Workshop ELI06 January 29, Types of Samples Non-Probability Probability Population

Pre-Conference Workshop ELI06 January 29, Non-Probability Available Subjects Purposive Snowball

Pre-Conference Workshop ELI06 January 29, Non-Probability Surveys Advantages Relatively easy Relatively cheap Disadvantages Not representative Not generalizable

Pre-Conference Workshop ELI06 January 29, Probability Samples

Pre-Conference Workshop ELI06 January 29, Probability Samples Advantages Can be representative Generalizable Allows targeted follow-up Allows for non- response analyses Disadvantages Lack of anonymity Time intensive Response rate

Pre-Conference Workshop ELI06 January 29, Population Web Surveys Advantages Can protect anonymity Inclusive Can be representative Not time intensive Non-Respondent analyses are possible Disadvantages No targeted follow-up Response rate Sample bias

Pre-Conference Workshop ELI06 January 29, Response Rate: Destroyer of Survey Results First thing to understand: Numbers don’t matter A randomly selected survey of 400 can have less sample bias than a non-random one of 10,000

Pre-Conference Workshop ELI06 January 29, Bias: Random v Systematic

Pre-Conference Workshop ELI06 January 29, Me & My Scales Scale 1  Day 1200  Day 2195  Day 3205  Day 4197  Day 5203  Average200 Scale 2  Day 1190  Day 2190  Day 3190  Day 4190  Day 5190  Average190

Pre-Conference Workshop ELI06 January 29, Sampling Error vs Sample Bias Sampling error is estimated based on the number of respondents in your sample –The bigger your sample the lower your sampling error

Pre-Conference Workshop ELI06 January 29,

Pre-Conference Workshop ELI06 January 29, Sampling Error vs Sample Bias Sampling error is estimated based on the number of respondents in your sample –The bigger your sample the lower your sampling error –But!!! It assumes that the sample you have drawn is a random sample!!!

Pre-Conference Workshop ELI06 January 29, Sample Bias When there is a non-random reason why some are in your sample and some are not If this bias is significant, the size of the sample won’t solve the problem What matters is the response rate The higher the response rate, the less chance of having a high sample bias

Pre-Conference Workshop ELI06 January 29, Why this matters Sample generalizability If the reason people do not respond to the survey is related to the topic of the survey itself, you will get an unrepresentative inclusion or exclusion.

Pre-Conference Workshop ELI06 January 29, Exercise 2 Part 1 You are developing a student and faculty survey instruments that measure 1) What instructional technology students expect to be included in their coursework and 2) what instructional technology faculty think students want in their coursework. Q: For students and faculty, what would be some reasons why they would and would not reply to this survey?

Pre-Conference Workshop ELI06 January 29, Exercise 2 Part 2 Your student survey has a response rate of 60% Your faculty survey has a response rate of 50%. Q: What can you do to estimate how biased these samples are?

Pre-Conference Workshop ELI06 January 29, How High a Response Rate Do You Need? Anything under 50% challenges claims that the survey results represent your target population. National Center for Educational Statistics

Pre-Conference Workshop ELI06 January 29, Type of Survey Stage-Specific Design Response Rates ScreenerSchoolAll Other Universe Cross-sectional Longitudinal Assessment Random-Digit Dial> Household    STANDARD 2-2-3: NCES sample survey data collections must be designed to meet a target item response rate of at least 90 percent for each key item.

Pre-Conference Workshop ELI06 January 29, How High a Response Rate Do You Need? Anything under 50% challenges claims that the survey results represent your target population. National Center for Educational Statistics This is why random samples are better than population samples

Pre-Conference Workshop ELI06 January 29, How to Improve Your Response Rates Be brief! Make response opportunities easily understood Make the survey look professional Do a pre-notification: A survey is coming and it will demand very little of your time

Pre-Conference Workshop ELI06 January 29, How to Improve Your Response Rates Have the survey endorsed by a well- regarded (and highly placed) official Follow up: The first, no longer than a week after deployment Make them believe that this survey is important Assure them of confidentiality

Pre-Conference Workshop ELI06 January 29, Asking the Right Questions The return of random and systematic bias (Remember the bathroom scale?) When we’re talking about measurements, we are talking about measurement error

Pre-Conference Workshop ELI06 January 29,

Pre-Conference Workshop ELI06 January 29, Reliability & Validity Reliability –How accurate is our measurement? Validity –Are we measuring what we think we are measuring?

Pre-Conference Workshop ELI06 January 29, Exercise 3 We’re trying to measure a student’s information literacy. Here’s the question: What is the frequency that you go on the Internet? __Often __From time to time __Seldom

Pre-Conference Workshop ELI06 January 29,

Pre-Conference Workshop ELI06 January 29,

Pre-Conference Workshop ELI06 January 29,

Pre-Conference Workshop ELI06 January 29, Measurement Error Reliability is a quantifiable problem Validity is a theoretical problem

Pre-Conference Workshop ELI06 January 29, Reliability Fixes Use established measures Test/retest Use multiple indicators –A note about Response Set, however Split half techniques

Pre-Conference Workshop ELI06 January 29, Exercise 4 Suggest some multiple indicators for the concept: Information literacy

Pre-Conference Workshop ELI06 January 29, Validity Checks Face Validity Criterion Validity Content Validity

Pre-Conference Workshop ELI06 January 29, Exercise 5 Taking the multiple-indicator measure of information literacy developed in the last exercise, suggest ways that you can validate it.

Pre-Conference Workshop ELI06 January 29, A Word about Opinions Beliefs Attitudes Opinions The first two are internal Opinions are external Pseudo-opinions

Pre-Conference Workshop ELI06 January 29, Surveys: Summary Pre-test!! Publicize!! Randomize!! Put it on the Web! Follow-up!

Pre-Conference Workshop ELI06 January 29, The ELI Student & Faculty Survey

Pre-Conference Workshop ELI06 January 29, Quasi-Experimental Designs What are they used for? Why ‘Quasi’ ? Experimental group Control group

Pre-Conference Workshop ELI06 January 29, Quasi-Experimental Designs

Pre-Conference Workshop ELI06 January 29, Why Use a Control Group? Maturation effects History Selection Bias Experimental mortality

Pre-Conference Workshop ELI06 January 29, What Should the Control Group Look Like? Ideally: The only difference should be that one gets the “treatment” and the other doesn’t Matching Problems for measuring the impact of new technologies on teaching and learning.

Pre-Conference Workshop ELI06 January 29, Ethical Issues in the Use of Surveys Babbie) Voluntary participation Harm to participants Anonymity Confidentiality Deception Institutional Review Boards (Human Subjects Committees)

Pre-Conference Workshop ELI06 January 29, Final Thoughts Make friends with someone at your institution who understands survey methodology Staff development opportunities Don’t do it, unless you can do it right “Having some data is better than having none at all” NOT!!!

Pre-Conference Workshop ELI06 January 29,