Copyright © Allyn & Bacon (2007) Controls to Reduce Threats to Validity Graziano and Raulin Research Methods: Chapter 9 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2007) Threats to Validity Covered in Chapter 8 Covered in Chapter 8 Validity can be threatened in many ways Validity can be threatened in many ways –Presence of confounding variables –Unrepresentative samples –Inappropriate statistical tests or violations of statistics assumptions –Subject and experimenter effects All these threats can be controlled All these threats can be controlled

Copyright © Allyn & Bacon (2007) Control Procedures General control procedures (applicable to virtually all research) General control procedures (applicable to virtually all research) Control over subject and experimenter effects Control over subject and experimenter effects Control through the selection and assignment of participants Control through the selection and assignment of participants Control through experimental design Control through experimental design

Copyright © Allyn & Bacon (2007) General Control Procedures Preparation of the setting Preparation of the setting –Free of distractions that might interfere –A natural setting increases external validity Response Measurement Response Measurement –Use reliable and valid measures Replication Replication –Demonstrates that findings are consistent and robust

Copyright © Allyn & Bacon (2007) Types of Replication Exact Replication Exact Replication –Repeating a study using identical procedures to the original Systematic Replication Systematic Replication –Using a theoretical or procedural change Conceptual Replication Conceptual Replication –Varying the operational definitions of the variables to get new research hypotheses

Copyright © Allyn & Bacon (2007) Subject and Experimenter Effects Blind procedures Blind procedures –Best control for expectancy effects –Single-blind: The experimenter does not know what condition the participant is in –Double-blind: Neither the experimenter nor the participant knows what condition the participant is in

Copyright © Allyn & Bacon (2007) Subject and Experimenter Effects Automation Automation –Reduces contact between participants and the experimenter –Gives the experimenter less opportunity to affect participants Using objective measures Using objective measures –Objective measure require less judgment –Provides less opportunity for subtle experimenter biases to affect the data

Copyright © Allyn & Bacon (2007) Subject and Experimenter Effects Multiple observers Multiple observers –Reduces bias because it challenges observers to be as objective as possible –Can measure amount of observer agreement (percent agreement or Kappa) Using deception Using deception –Hides purpose of the study from participants –Balanced placebo design is a good example

Copyright © Allyn & Bacon (2007) Balanced Placebo Design Separates the pharmacological effects from the expectancy effects of alcohol Separates the pharmacological effects from the expectancy effects of alcohol A two-factor design A two-factor design –Factor 1 is whether the person drinks alcohol –Factor 2 is whether the person thinks he or she is drinking alcohol

Copyright © Allyn & Bacon (2007) Balanced Placebo Design This design crosses the consumption of alcohol with the belief that alcohol is being consumed People Led to Believe Drinking Alcohol Not Drinking Alcohol Actual Situation Drinking Alcohol Not Drinking Alcohol

Copyright © Allyn & Bacon (2007) Participant Selection Can generalize only if your sample is representative Can generalize only if your sample is representative Populations and samples Populations and samples –General population: all potential participants –Target population: those participants you are interested in –Accessible population: portion of target population that is available to the researcher –Sample: drawn from the accessible population

Copyright © Allyn & Bacon (2007) Populations and Sampling This figure shows the relationship between the various populations This figure shows the relationship between the various populations –General Population –Target Population –Accessible Population –Sample

Copyright © Allyn & Bacon (2007) Sampling Procedures Random sampling Random sampling –Every participant has an equal chance of being sampled Stratified random sampling Stratified random sampling –Random sampling within strata (subgroups) Ad hoc samples Ad hoc samples –Random sample from accessible population –Must generalize cautiously –Should describe sample to help define limits of generalization

Copyright © Allyn & Bacon (2007) Participant Assignment Critical to internal validity Critical to internal validity Assignment Procedures Assignment Procedures –Free random assignment Random assignment of participants to groups Random assignment of participants to groups –Matched random assignment Random assignment of participants in matched sets to groups Random assignment of participants in matched sets to groups –Other matching procedures e.g., match groups on key characteristics e.g., match groups on key characteristics

Copyright © Allyn & Bacon (2007) Matched Random Assignment Match on relevant variables Match on relevant variables –Variables likely to affect the dependent measure –Variables that show the largest variability in the population Procedures Procedures –Match in sets on the relevant variable Set size is the number of groups in the study Set size is the number of groups in the study –Randomly assign participants from the set, one to each group –Keep track of matching data for the statistical analysis

Copyright © Allyn & Bacon (2007) Experimental Design Main focus of Chapters 10 through 13 Main focus of Chapters 10 through 13 Experimental design maximizes validity Experimental design maximizes validity –Need to also include the other control procedures covered in this chapter Key elements of experiments Key elements of experiments –One or more control groups –Random assignment of participants to groups

Copyright © Allyn & Bacon (2007) Summary Most threats to validity can be minimized with proper use of control procedures Most threats to validity can be minimized with proper use of control procedures Broad classes of control procedures Broad classes of control procedures –General control procedures –Control over subject and experimenter effects –Control through participant selection and assignment –Control through specific experimental design