1. QUANTITATIVE RESEARCH METHODS Irina Shklovski

2. Quantitative Research Methods  Include a wide variety of laboratory and non- laboratory procedures  Involve measurement…

5. Quantitative Research Methods  Measurement  Populations and Sampling  Random Assignment  Generalizability

6. Quantitative Research Methods  Measurement  Populations and Sampling  Random Assignment  Generalizability  Time  Cross-sectional studies & single experiments  Longitudinal studies & repeated measures

7. Quantitative Research Methods  Method  Experiments & Quasi-experiments  Behavioral Measures  Questionnaires & Surveys  Social Network Analysis  Archival and Meta-Analysis

8. What we will talk about today  Measurement  Population & Sampling  Random Assignment  Generalizability  Method  Experiments & Quasi-experiments  Questionnaires & Surveys

9. Measurement – Sampling  Specify your population of concern  Sampling  Selecting respondents from population of concern  Random sampling  Systematic selection  Stratified sampling  Convenience sampling  Snowball sampling

10. Sampling Biases  Non-response bias  Be persistent  Offer incentives and rewards  Make it look important  Volunteer bias  Some people volunteer reliably more than others for a variety of tasks

11. Random assignment  Different from random sampling  Mostly used for experiments or quazi-experiments  Protects against unsuspected sources of bias  Does NOT guarantee to balance out the differences between participants  Chance is LUMPY

12. Generalizability  How do you know that what you found in your research study is, in fact, a general trend?  Does A really, always cause B?  If A happens, is B really as likely to happen as you claim? Always? Under certain conditions?

13. Association vs. Causality Thanks to Sara Kiesler for these graphs!

14. Experiments & Quasi-experiments  ex·per·i·ment  Pronunciation: \ik- ˈ sper- ə -m ə nt also - ˈ spir-\  Function: noun  Etymology: Middle English, from Anglo-French esperiment, from Latin experimentum, from experiri  Date: 14th century  An operation or procedure carried out under controlled conditions to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law

15. Experiments & Quasi-experiments  Key feature common to all experiments:  To deliberately vary something in order to discover what happens to something else later  To seek the effects of presumed causes

16. An Experiment is  A controlled empirical test of a hypothesis.  Hypotheses include:  A causes B  A is bigger, faster, better than B  A changes more than B when we do X  Two requirements:  Independent variable that can be manipulated  Dependent variable that can be measured

17. Experiments in Research  Comparing one design or process to another  Deciding on the importance of a particular feature in a user interface  Evaluating a technology or a social intervention in a controlled environment  Finding out what really causes an effect  Finding out if an effect really exists

18. Remember  Experiments explore the effects of things that can be MANIPULATED  (but there is a caveat)

19. Types of Experiments  Randomized – units/participants assigned to receive treatment or alternative condition randomly  Quazi – no random assignment  Natural – contrasting a naturally occurring event (i.e. disaster) with a comparison condition

20. If your study involves experiments  Experimental design: Shadish W.R., Cook T.D. & Campbell P.T. (2002) Experimental and Quasi-Experimental Design for Generalized Causal Inference. Boston, Mass: Houghton Mifflin  Experimental data analysis: Bruning, J. L. & Kintz, B. L. (1997). Computational handbook of statistics (4th ed.). New York: Longman.

21. Questionnaires & Surveys  Self-report measures  Questionnaires & surveys  Interviews  Diaries  Types  Structured  Open-ended

22. Questionnaires & Surveys  Advantages  Sample large populations (cheap on materials & effort)  Efficiently ask a lot of questions  Disadvantages  Self-report is fallible  Response biases are unavoidable

23. Response biases  Relying on people’s memory of events & behaviors  Emotional states can “prime” memory  Recency effects  Routines are deceiving  Social desirability  Solution: none that are simple  Yea-saying  Solution: vary the direction of response alternatives

24. General Survey Biases  Sampling – are respondents representative of population of interest? How were they selected?  Coverage – do all persons in the population have an equal change of getting selected?  Measurement – question wording & ordering can obstruct interpretation  Non-response – people who respond differ from those that do not

25. Design is KEY  Format – booklet, printed vertical, one-sided  Question ordering – earlier questions can prime answers to later questions  Page layout – group similar items & use consistent fonts and response categories  Pre-testing – conduct think-alouds with volunteers demographically similar to expected participants

26. Common Problems  Avoid complicated & double-barrel questions  Complexity increases errors & non-response  Navigation is paramount – make sure the survey is EASY to follow  Open-ended questions  The size of the field allotted will determine the number of words  Incentive is key  BUT amount differences have little impact

27. If your study involves surveys  Designing surveys: Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009). Internet, mail, and mixed-mode surveys : the tailored design method (3rd ed.). Hoboken, N.J.: Wiley & Sons. Fowler, F. J. (1995). Improving survey questions : design and evaluation. Thousand Oaks: Sage Publications.  Analyzing data: Cohen, J., Cohen, P., West, S., & Aiken, L. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

28. So… what?  Difference between quantitative methods is in the questions they can answer  There are a LOT of methods and even more statistical techniques  Regardless of the method, if it’s not an experiment, you CAN NOT prove causation

29. Things we did NOT talk about  Reliability assessments  Validity assessments  Statistical analysis of data  Interpretation of results