Experimental Research Validity and Confounds

What is it? Systematic inquiry that is characterized by: Systematic inquiry that is characterized by: An investigator’s direct manipulation (variation) of some factor or factors (IV or IV’s) thought to be causally related to some outcome or outcomes (DV or DV’s) which are observed as data while holding all other factors constant. An investigator’s direct manipulation (variation) of some factor or factors (IV or IV’s) thought to be causally related to some outcome or outcomes (DV or DV’s) which are observed as data while holding all other factors constant.

Independent Variable The independent variable: conditions in the experiment The independent variable: conditions in the experiment Different situations (rooms, experiences, etc) Different situations (rooms, experiences, etc) Different instructions Different instructions Different tasks Different tasks Different treatments Different treatments Experimental vs. control groups Experimental vs. control groups Bystander study: Manipulate the number of bystanders Bystander study: Manipulate the number of bystanders

Essential Features of Experiments Independent Variable Independent Variable Types? Types? Control group Control group Dependent variable Dependent variable

Manipulated vs. Subject Variable Subject variables are characteristics that you cannot manipulate: gender, choosing subjects with high or low anxiety. Subject variables are characteristics that you cannot manipulate: gender, choosing subjects with high or low anxiety. Subject variable studies also called “natural groups studies” Subject variable studies also called “natural groups studies” Manipulated variables  true experimental designs Manipulated variables  true experimental designs Create groups where you place/force subjects to be in one group or another Create groups where you place/force subjects to be in one group or another Create anxiety for ½ of the group. Create anxiety for ½ of the group. Limitation with a natural groups study  can’t draw causal conclusions because you can’t rule out other possible factors that might be the “true” causal factor. Limitation with a natural groups study  can’t draw causal conclusions because you can’t rule out other possible factors that might be the “true” causal factor. Example: gender Example: gender

Extraneous variables vs. Confounds Extraneous Variable: uncontrolled, unsystematic factors or variance. Can be ANY variable that is related to the IV Extraneous Variable: uncontrolled, unsystematic factors or variance. Can be ANY variable that is related to the IV Creates “noise” and may affect the outcome variable, just not systematically. Creates “noise” and may affect the outcome variable, just not systematically. Confounding variable: uncontrolled, systematic variance Confounding variable: uncontrolled, systematic variance Affects the different levels of the IV differently. Affects the different levels of the IV differently. 2 things to control for confounds: 2 things to control for confounds: Random assignment Random assignment Within Subjects designs Within Subjects designs

Internal vs External Validity External validity: the extent to which the results of ones study can be generalized to subjects and/or conditions outside of the study. Often done by replication studies External validity: the extent to which the results of ones study can be generalized to subjects and/or conditions outside of the study. Often done by replication studies Internal validity: found when results are characterized by the ABSENCE of competing explanations. You are confident that changes in the IV did indeed lead to changes in the DV. Internal validity: found when results are characterized by the ABSENCE of competing explanations. You are confident that changes in the IV did indeed lead to changes in the DV. Experimental control is the key to internal validity Experimental control is the key to internal validity An absence of confounds and extraneous variables An absence of confounds and extraneous variables Control groups help establish internal validity Control groups help establish internal validity

Threats to Internal Validity Post test study Post test study Treatment  observation Treatment  observation Ritalin and ADHD Ritalin and ADHD Pre test-post test study Pre test-post test study Observation  treatment  observation- Offers a baseline measurement from which we can compare Observation  treatment  observation- Offers a baseline measurement from which we can compare

Threats to Internal Validity In pre test-post test studies we find.. History and maturation History and maturation History: Something external happens and affects changes in your independent variable History: Something external happens and affects changes in your independent variable maturation: external to your study, but internal to your subjects. Development or experience maturation: external to your study, but internal to your subjects. Development or experience Regression to the mean Regression to the mean When people are chosen because they are ‘extreme’ in some way and you measure their change over time in that extreme dimension When people are chosen because they are ‘extreme’ in some way and you measure their change over time in that extreme dimension you’re likely to see a change in the score closer to the mean. you’re likely to see a change in the score closer to the mean. Instrumentation and testing Instrumentation and testing Change in the researchers (or subjects) instrumentation or measuring over time. Change in the researchers (or subjects) instrumentation or measuring over time. Testing: when actually taking the test affects the behavior or helps “practice”. Testing: when actually taking the test affects the behavior or helps “practice”. IDU study Instrumentation: changes in the instrument used to measure from pre to post. Instrumentation: changes in the instrument used to measure from pre to post.

Controlling Against TtIV Use a Control Group Use a Control Group Observation  Treatment  Observation Observation  Treatment  ObservationAND Observation  Observation Observation  ObservationOR Observation  Placebo---  Observation Observation  Placebo---  Observation But, what if these threats don’t affect both groups equally? But, what if these threats don’t affect both groups equally?

Example Want to know the sexual assault opinions of UWT students Want to know the sexual assault opinions of UWT students Group 1: Wild life video Group 1: Wild life video 2 weeks later… Group 2: Assault video Group 2: Assault video But during those 2 weeks there was a string of sexual assaults on campus What kind of threat? What kind of threat? Solution? Solution?

Other Threats Other threats come from HOW a subject is chosen for the study and how comparison groups are formed Other threats come from HOW a subject is chosen for the study and how comparison groups are formed Subject selection- if subjects chosen for one group are different than those in another group before the study takes place. Subject selection- if subjects chosen for one group are different than those in another group before the study takes place. Solution: Random assignment of subjects into groups Solution: Random assignment of subjects into groups Attrition/subject mortality- When subjects drop out or refuse to take part in your experiment. Attrition/subject mortality- When subjects drop out or refuse to take part in your experiment. Possible solution: try to interview the people who dropped out to find out why and see if that is related to the IV Possible solution: try to interview the people who dropped out to find out why and see if that is related to the IV