1. Types of research design – experiments Chapter 8 in Babbie & Mouton (2001) Chapter 8 in Babbie & Mouton (2001) Introduction to all research designs Introduction to all research designs All research designs have specific objectives they strive for All research designs have specific objectives they strive for Have different strengths and limitations Have different strengths and limitations Have validity considerations Have validity considerations SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

2. Validity considerations When we say that a knowledge claim (or proposition) is valid, we make a JUDGEMENT about the extent to which relevant evidence supports that claim to be true When we say that a knowledge claim (or proposition) is valid, we make a JUDGEMENT about the extent to which relevant evidence supports that claim to be true Is the interpretation of the evidence given the only possible one, or are there other plausible ones? Is the interpretation of the evidence given the only possible one, or are there other plausible ones? "Plausible rival hypotheses" = potential alternative explanations/claims "Plausible rival hypotheses" = potential alternative explanations/claims e.g. New York City's "zero tolerance" crime fighting strategy in the 1980s and 1990s - the reverse of the "broken windows" effect e.g. New York City's "zero tolerance" crime fighting strategy in the 1980s and 1990s - the reverse of the "broken windows" effect SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

3. The logic of causal social research in the controlled experiment Explanatory rather than descriptive Explanatory rather than descriptive Different from correlational research - one variable is manipulated (IV) and the effect of that manipulation observed on a second variable (DV) Different from correlational research - one variable is manipulated (IV) and the effect of that manipulation observed on a second variable (DV) If … then …. If … then …. E.g. E.g. "Animals respond aggressively to crowding" (causal) "Animals respond aggressively to crowding" (causal) "People with premarital sexual experience have more stable marriages" (noncausal) "People with premarital sexual experience have more stable marriages" (noncausal) SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

4. Three pairs of components: Independent and dependent variables Independent and dependent variables Pre-testing and post-testing Pre-testing and post-testing Experimental and control groups Experimental and control groups SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

5. Components Variables Variables Dependent (DV) Dependent (DV) Independent (IV) Independent (IV) Pre-testing and post-testing Pre-testing and post-testing OX O OX O Experimental and control groups Experimental and control groups To off-set the effects of the experiment itself; to detect effects of the experiment itself To off-set the effects of the experiment itself; to detect effects of the experiment itself SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

6. Types of design - experiments 6 The generic experimental design: RO 1 XO 2 RO 1 XO 2 RO 3 O 4 The IV is an active variable; it is manipulated The IV is an active variable; it is manipulated The participants who receive one level of the IV are equivalent in all ways to those who receive other levels of the IV The participants who receive one level of the IV are equivalent in all ways to those who receive other levels of the IV SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

7. Types of design - experiments 7 Sampling 1. Selecting subjects to participate in the research 1. Selecting subjects to participate in the research Careful sampling to ensure that results can be generalized from sample to population Careful sampling to ensure that results can be generalized from sample to population The relationship found might only exist in the sample; need to ensure that it exists in the population The relationship found might only exist in the sample; need to ensure that it exists in the population Probability sampling techniques Probability sampling techniques SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

8. Types of design - experiments 8 Sampling Sampling 2. How the sample is divided into two or more groups is important 2. How the sample is divided into two or more groups is important to make the groups similar when they start off to make the groups similar when they start off randomization - equal chance randomization - equal chance matching - similar to quota sampling procedures matching - similar to quota sampling procedures match the groups in terms of the most relevant variables; e.g. age, sex, and race match the groups in terms of the most relevant variables; e.g. age, sex, and race SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

9. Types of design - experiments 9 Variations on the standard experimental design One-shot case study One-shot case study XO No real comparison No real comparison SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

10. Types of design - experiments 10 A famous one-group posttest-only design Milgram's study on obedience Milgram's study on obedience Obedience to authority Obedience to authority The willingness of subjects to follow E's orders to give painful electrical shocks to another subject The willingness of subjects to follow E's orders to give painful electrical shocks to another subject A real, important issue here: how could "ordinary" citizens, like many Germans during the Nazi period, do these incredibly cruel and brutal things? A real, important issue here: how could "ordinary" citizens, like many Germans during the Nazi period, do these incredibly cruel and brutal things? If a person is under allegiance to a legitimate authority, under what conditions will the person defy the authority if s/he is asked to carry out actions clearly incompatible with basic moral standards? If a person is under allegiance to a legitimate authority, under what conditions will the person defy the authority if s/he is asked to carry out actions clearly incompatible with basic moral standards? SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

11. Types of design - experiments 11 One-group pre-test post-test design O 1 XO 2 O 1 XO 2 SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

12. Types of design - experiments 12 Example We want to find out whether a family literacy programme enhances the cognitive development of preschool-age children. We want to find out whether a family literacy programme enhances the cognitive development of preschool-age children. Find 20 families with a 4-year old child, enrol the family in a high-quality family literacy programme Find 20 families with a 4-year old child, enrol the family in a high-quality family literacy programme Administer a pretest to the 20 children - they score a mean of say 50 on the cognitive test Administer a pretest to the 20 children - they score a mean of say 50 on the cognitive test The family participates in the programme for twelve months The family participates in the programme for twelve months Administer a post-test to the 20 children; now they score 75 on the test - a gain of 25 Administer a post-test to the 20 children; now they score 75 on the test - a gain of 25 SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

13. Types of design - experiments 13 Two claims/conclusions: 1 The children gained 25 points on average in terms of their cognitive performance 1 The children gained 25 points on average in terms of their cognitive performance 2 the family literacy programme caused the gain in scores 2 the family literacy programme caused the gain in scores VALIDITY - rival explanations VALIDITY - rival explanations SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

14. Types of design - experiments 14 Static-group comparison XO O SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

15. Types of design - experiments 15 Evaluating research (experiments) We know the structure of research We know the structure of research We understand designs We understand designs We know the requirements of "good" research We know the requirements of "good" research Then we can evaluate a study Then we can evaluate a study Is it good? Can we believe its conclusions? Is it good? Can we believe its conclusions? Back to plausible rival hypotheses Back to plausible rival hypotheses SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

16. Types of design - experiments 16 Validity in designs If the design is not valid, then the conclusions drawn are not supported; it is like not doing research at all If the design is not valid, then the conclusions drawn are not supported; it is like not doing research at all Validity of designs come in two parts: Validity of designs come in two parts: Internal validity Internal validity can the design sustain the conclusions? can the design sustain the conclusions? External validity External validity can the conclusions be generalized to the population? can the conclusions be generalized to the population? SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

17. Types of design - experiments 17 Internal validity Each design is only capable of supporting certain types of conclusions Each design is only capable of supporting certain types of conclusions e.g. only experiments can support conclusions about causality e.g. only experiments can support conclusions about causality Says nothing about if the results can be applied to the real world (generalization) Says nothing about if the results can be applied to the real world (generalization) Generally, the more controlled the situation, the higher the internal validity Generally, the more controlled the situation, the higher the internal validity The conclusions drawn from experimental results may not accurately reflect hat has gone on in the experiment itself The conclusions drawn from experimental results may not accurately reflect hat has gone on in the experiment itself SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

18. Types of design - experiments 18 Sources of internal invalidity These sources often discussed as part of experiments, but can be applied to all designs (e.g. see reactivity) These sources often discussed as part of experiments, but can be applied to all designs (e.g. see reactivity) History History Historical events may occur that will be confounded with the IV Historical events may occur that will be confounded with the IV Especially in field research (compare the control in a laboratory, e.g. nonsense syllables in memory studies Especially in field research (compare the control in a laboratory, e.g. nonsense syllables in memory studies SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

19. Types of design - experiments 19 Maturation Changes over time can be caused by a natural learning process Changes over time can be caused by a natural learning process People naturally grow older, tired, bored, over time People naturally grow older, tired, bored, over time SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

20. Types of design - experiments 20 Testing (reactivity) People realize they are being studied, and respond the way they think is appropriate People realize they are being studied, and respond the way they think is appropriate The very act of studying something may change it In qualitative research, the "on stage" effects In qualitative research, the "on stage" effects SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

21. Types of design - experiments 21 The Hawthorne studies Improved performance because of the researcher's presence - people became aware that they were in an experiment, or that they were given special treatment Improved performance because of the researcher's presence - people became aware that they were in an experiment, or that they were given special treatment Especially for people who lack social contacts, e.g. residents of nursing homes, chronic mental patients Especially for people who lack social contacts, e.g. residents of nursing homes, chronic mental patients SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

22. Placebo effect When a person expects a treatment or experience to change her/him, the person changes, even when the "treatment" is know to be inert or ineffective When a person expects a treatment or experience to change her/him, the person changes, even when the "treatment" is know to be inert or ineffective Medical research Medical research "The bedside manner", or the power of suggestion "The bedside manner", or the power of suggestion SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

23. Experimenter expectancy Pygmalion effect - self-fulfilling prophecies of e.g. teachers' expectancies about student achievement Pygmalion effect - self-fulfilling prophecies of e.g. teachers' expectancies about student achievement Experimenters may prejudge their results - experimenter bias Experimenters may prejudge their results - experimenter bias Double blind experiments: Double blind experiments: Both the researcher and the research participant are "blind" to the purpose of the study. Both the researcher and the research participant are "blind" to the purpose of the study. They don't know what treatment the participant is getting They don't know what treatment the participant is getting SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

24. Instrumentation Instruments with low reliability lead to inaccurate findings/missing phenomena Instruments with low reliability lead to inaccurate findings/missing phenomena e.g. human observers become more skilled over time (from pretest to posttest) and so report more accurate scores at later time points e.g. human observers become more skilled over time (from pretest to posttest) and so report more accurate scores at later time points SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

25. Statistical regression to the mean Studying extreme scores can lead to inflated differences, which would not occur in moderate scorers Studying extreme scores can lead to inflated differences, which would not occur in moderate scorers SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

26. Selection biases Selection subjects for the study, and assigning them to E-group and C-group Selection subjects for the study, and assigning them to E-group and C-group Look out for studies using volunteers Look out for studies using volunteers SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

27. Types of design - experiments 27 Attrition Sometimes called experimental (or subject) mortality Sometimes called experimental (or subject) mortality If subjects drop out, it creates a bias to those who did not If subjects drop out, it creates a bias to those who did not e.g. comparing the effectiveness of family therapy with discussion groups for treatment of drug addiction e.g. comparing the effectiveness of family therapy with discussion groups for treatment of drug addiction addicts with the worst prognosis more likely to drop out of the discussion group addicts with the worst prognosis more likely to drop out of the discussion group will make it look like family therapy does less well than discussion groups, because the "worst cases" were still in the family therapy group will make it look like family therapy does less well than discussion groups, because the "worst cases" were still in the family therapy group SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

28. Types of design - experiments 28 Diffusion or imitation of treatments When subject can communicate to each other, pass on some information about the treatment (IV) When subject can communicate to each other, pass on some information about the treatment (IV) SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

29. Types of design - experiments 29 Compensation In real life, people may feel sorry for C- group who does not get "the treatment" - try to give them something extra In real life, people may feel sorry for C- group who does not get "the treatment" - try to give them something extra e.g. compare usual day care for street children with an enhanced day treatment condition e.g. compare usual day care for street children with an enhanced day treatment condition service providers may very well complain about inequity, and provide some enhanced service to the children receiving usual care service providers may very well complain about inequity, and provide some enhanced service to the children receiving usual care SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

30. Compensatory rivalry C-group may "work harder" to compete better with the E-group C-group may "work harder" to compete better with the E-group SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

31. Demoralization Opposite to compensatory rivalry Opposite to compensatory rivalry May feel deprived, and give up May feel deprived, and give up e.g. giving unemployed high school dropouts a second chance at completing matric via a special education programme e.g. giving unemployed high school dropouts a second chance at completing matric via a special education programme if we assign some of them to a control group, who receive "no treatment", they may very well become profoundly demoralized if we assign some of them to a control group, who receive "no treatment", they may very well become profoundly demoralized SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

32. Types of design - experiments 32 External validity Can the findings of the study be generalized? Can the findings of the study be generalized? Do they speak only of our sample, or of a wider group? Do they speak only of our sample, or of a wider group? To what populations, settings, treatment variables (IV's), and measurement variables can the finding be generalized? To what populations, settings, treatment variables (IV's), and measurement variables can the finding be generalized? SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

33. External validity Mainly questions about three aspects: Mainly questions about three aspects: Research participants Research participants Independent variables, or manipulations Independent variables, or manipulations Dependent variables, or outcomes Dependent variables, or outcomes Says nothing about the truth of the result that we are generalizing Says nothing about the truth of the result that we are generalizing External validity only has meaning once the internal validity of a study has been established External validity only has meaning once the internal validity of a study has been established Internal validity is the basic minimum without which an experiment is uninterpretable Internal validity is the basic minimum without which an experiment is uninterpretable SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

34. External validity Our interest in answering research questions is rarely restricted to the specific situation studied - our interest is in the variables, not the specific details of a piece of research Our interest in answering research questions is rarely restricted to the specific situation studied - our interest is in the variables, not the specific details of a piece of research But studies differ in many ways, even if they study the same variables: But studies differ in many ways, even if they study the same variables: operational definitions of the variables operational definitions of the variables subject population studied subject population studied procedural details procedural details observers observers settings settings Generally bigger samples with valid measures lead to better external validity Generally bigger samples with valid measures lead to better external validity SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

35. Sources of external invalidity Subject selection - Selecting a sample which does not represent the population well, will prevent generalization Subject selection - Selecting a sample which does not represent the population well, will prevent generalization Interaction between the testing situation and the experimental stimulus Interaction between the testing situation and the experimental stimulus When people have been sensitized to the issues by the pre-test When people have been sensitized to the issues by the pre-test Respond differently to the questionnaires the second time (post-test) Respond differently to the questionnaires the second time (post-test) Operationalization Operationalization SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

36. Types of design - experiments 36 Operationalization We take a variable with wide scope and operationalize it in a narrow fashion We take a variable with wide scope and operationalize it in a narrow fashion Will we find the same results with a different operationalization of the same variable? Will we find the same results with a different operationalization of the same variable? SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

37. Field experiments "natural" - e.g. disaster research "natural" - e.g. disaster research Static-group comparison type Static-group comparison type Non-equivalent experimental and control groups Non-equivalent experimental and control groups SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎

38. Strengths and weaknesses Strengths Strengths Control Control Manipulating the IV Manipulating the IV Sorting out extraneous variables Sorting out extraneous variables Weaknesses Weaknesses Articifiality - a generalization problem Articifiality - a generalization problem Expense Expense Limited range of questions Limited range of questions SUMBER: web.uct.ac.za/.../Types%20of%20research%20d...‎