Week 2 Internal and External Validity

Internal Validity: Evaluating Your Experiment from the Inside Internal Validity The concept of internal validity revolves around the question of whether your IV actually caused any change that you observe in your DV.

Internal Validity: Evaluating Your Experiment from the Inside Internal Validity The concept of internal validity revolves around the question of whether your IV actually caused any change that you observe in your DV. If you use adequate control techniques, your experiment should be free from confounding and you can, indeed, conclude that your IV caused the change in your DV.

Threats to Internal Validity Threats to Internal Validity History History refers to events that occur between the DV measurements in a repeated measures design.

Threats to Internal Validity Threats to Internal Validity History Maturation Maturation refers to changes in participants that occur over time during an experiment.

Threats to Internal Validity Threats to Internal Validity History Maturation Maturation refers to changes in participants that occur over time during an experiment. These changes could include actual physical maturation or tiredness, boredom, hunger, and so on.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Testing is a threat to internal validity that occurs because measuring the DV causes a change in the DV.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Testing is a threat to internal validity that occurs because measuring the DV causes a change in the DV. Campbell (1957) noted that if you take the same test more than once, scores on the second test may vary systematically from the first scores simply because you took the test a second time.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect A practice effect is a beneficial effect on a DV measurement caused by previous experience with the DV.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Reactive measures are DV measurements that actually change the DV being measured.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Reactive measures are DV measurements that actually change the DV being measured. Many attitude questionnaires are reactive measures. If we ask you a number of questions about how you feel about people of different racial groups, or about women’s rights, or about the president’s job performance, you can probably figure out that your attitude is being measured.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Instrumentation (Instrument Decay) Instrumentation is a threat to internal validity that occurs if the equipment or human measuring the DV changes the measuring criterion over time.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Instrumentation (Instrument Decay) Statistical Regression Statistical regression occurs when low scorers improve or high scorers fall on a second administration of a test due solely to statistical reasons.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Instrumentation (Instrument Decay) Statistical Regression Selection If we choose participants in such a way that our groups are not equal before the experiment, we cannot be certain that our IV caused any difference we observe after the experiment.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Instrumentation (Instrument Decay) Statistical Regression Selection Mortality Mortality can occur if experimental participants from different groups drop out of the experiment at different rates.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Instrumentation (Instrument Decay) Statistical Regression Selection Mortality Interactions with Selection Interactions with selection can occur when the groups we have selected show differences on another variable (i.e., maturation, history, or instrumentation) that vary systematically by groups.

Threats to Internal Validity Threats to Internal Validity History Maturation Testing Practice Effect Reactive Measures Instrumentation (Instrument Decay) Statistical Regression Selection Mortality Interactions with Selection Diffusion or Imitation of Treatments Diffusion or imitation of treatments can occur if participants in one treatment group become familiar with the treatment of another group and copy that treatment.

Protecting Internal Validity Protecting Internal Validity Two Approaches You can (and should) implement various control procedures.

Protecting Internal Validity Protecting Internal Validity Two Approaches You can (and should) implement the various control procedures. Use a standard procedure Experimenters used standard procedures called experimental designs to help ensure internal validity.

Protecting Internal Validity Protecting Internal Validity How Important is Internal Validity?

Protecting Internal Validity Protecting Internal Validity How Important is Internal Validity? It is the most important property of any experiment. If you do not concern yourself with the internal validity of your experiment, you are wasting your time.

External Validity: Generalizing Your Experiment to the Outside The second type of evaluation that you must make of your experiment involves external validity.

External Validity: Generalizing Your Experiment to the Outside The second type of evaluation that you must make of your experiment involves external validity. When you consider external validity, you are asking a question about generalization.

External Validity: Generalizing Your Experiment to the Outside External Validity A type of evaluation of your experiment that asks whether your experimental results apply to populations and situations that are different from those of your experiment.

External Validity: Generalizing Your Experiment to the Outside External Validity A type of evaluation of your experiment that asks whether your experimental results apply to populations and situations that are different from those of your experiment. Generalization Applying the results from an experiment to a different situation or population. In essence, we would like to take our results beyond the narrow confines of our specific experiment. Generalization is an important aspect for any science.

External Validity: Generalizing Your Experiment to the Outside There are three customary types of generalization in which we are interested: Population generalization Environmental generalization Temporal generalization

External Validity: Generalizing Your Experiment to the Outside There are three customary types of generalization in which we are interested: Population generalization Applying the results from an experiment to a group of participants that is different and more encompassing than those used in the original experiment. Environmental generalization Temporal generalization

External Validity: Generalizing Your Experiment to the Outside There are three customary types of generalization in which we are interested: Population generalization Environmental generalization Applying the results from an experiment to a situation or environment that differs from that of the original experiment. Temporal generalization

External Validity: Generalizing Your Experiment to the Outside There are three customary types of generalization in which we are interested: Population generalization Environmental generalization Temporal generalization Applying the results from an experiment to a time that is different from that when the original experiment was conducted.

Threats to External Validity (Based on Methods) Here is a summary of factors relating to external validity from Campbell and Stanley (1966):

Threats to External Validity (Based on Methods) Here is a summary of factors relating to external validity from Campbell and Stanley (1966): Interaction of Testing and Treatment A threat to external validity that occurs when a pretest sensitizes participants to the treatment yet to come. Occurs for the pretest-posttest control group design Because of a pretest, your participants’ reaction to the treatment will be different.

Threats to External Validity (Based on Methods) Here is a summary of factors relating to external validity from Campbell and Stanley (1966): Interaction of Testing and Treatment Interaction of Selection and Treatment A threat to external validity that can occur when a treatment effect is found only for a specific sample of participants. Occurs when the effects that you demonstrate hold true only for the particular groups that you selected for your experiment. Treatment interaction becomes greater as it becomes more difficult to find participants for your experiment (Campbell & Stanley, 1966).

Threats to External Validity (Based on Methods) Here is a summary of factors relating to external validity from Campbell and Stanley (1966): Interaction of Testing and Treatment Interaction of Selection and Treatment Reactive Arrangements A threat to external validity caused by an experimental situation that alters participants’ behavior, regardless of the IV involved. We cannot be sure that the behaviors we observe in the experiment will generalize outside that setting because the artificial conditions of the experiment do not exist in the real world.

Threats to External Validity (Based on Methods) Here is a summary of factors relating to external validity from Campbell and Stanley (1966): Interaction of Testing and Treatment Interaction of Selection and Treatment Reactive Arrangements Demand Characteristics Features from the experiment that inadvertently lead participants to respond in a particular manner. Demand characteristics make generalizations difficult because it is not clear from a set of research findings whether the participants are responding to an experiment’s IV, its demand characteristics, or both.

Threats to External Validity (Based on Methods) Here is a summary of factors relating to external validity from Campbell and Stanley (1966): Interaction of Testing and Treatment Interaction of Selection and Treatment Reactive Arrangements Demand Characteristics Multiple-Treatment Interference A threat to external validity that occurs when a set of findings results only when participants experience multiple treatments in the same experiment (repeated measures designs).

Threats to External Validity (Based on Our Participants) The Infamous White Rat

Threats to External Validity (Based on Our Participants) The Infamous White Rat If you are interested in the behavior of subhumans, generalizing from rats (and pigeons) to all other animals may be a stretch. If you are interested in generalizing from animal to human behavior, there are certainly closer approximations to humans (and pigeons) in the animal kingdom.

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student Researchers who want to conduct human research turn to a ready, convenient source of human participants – students in introductory courses (a technique referred to as convenience sampling ).

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student Researchers who want to conduct human research turn to a ready, convenient source of human participants – students in introductory courses (a technique referred to as convenience sampling ). Convenience Sampling A researcher’s sampling of participants based on ease of locating the participants; often does not involve true random selection.

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex All four of these derogatory labels have been applied to women at various points in time.

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex All four of these derogatory labels have been applied to women at various points in time. The supposed inferiority of women has carried over into some psychological theories. Freud’s theories Erikson’s theory of psychosocial crises (“Eight Stages of Man”)

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex Carol Tavris’s (1992) thesis is that “despite women’s gains in many fields in the last twenty years, the fundamental belief in the normalcy of men, and the corresponding abnormality of women, has remained virtually untouched” (p. 17)

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex Even the Rats and Students Were White

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex Even the Rats and Students Were White Just as history has failed to record the accomplishments of many women throughout time, it has largely ignored the accomplishments of African Americans and other minority groups. When we conduct research and make generalizations, we should be cautious that we do not exclude minority groups from our considerations.

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex Even the Rats and Students Were White Even the Rats, Students, Women, and Minorities Were American

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex Even the Rats and Students Were White Even the Rats, Students, Women, and Minorities Were American

Threats to External Validity (Based on Our Participants) The Infamous White Rat The Ubiquitous College Student The “Opposite” or “Weaker” or “Inferior” or “Second” Sex Even the Rats and Students Were White Even the Rats, Students, Women, and Minorities Were American In the mid-1960’s, social researchers started taking culture and ethnicity more seriously. The field of cross-cultural ______(fill in the blank—nursing, etc.) has evolved from those changes that began in the 1960s.

Threats to External Validity (Based on Our Participants) Cross-Cultural Research A branch of research whose goal is to determine the universality of research results. Ethnocentricity Other cultures are viewed as an extension of one’s own.

The Devil’s Advocate: Is External Validity Always Necessary? Mook (1983) pointed out four alternative goals of research that do not stress external validity:

The Devil’s Advocate: Is External Validity Always Necessary? Mook (1983) pointed out four alternative goals of research that do not stress external validity: We may merely want to find out if something can happen (not whether it actually happens).

The Devil’s Advocate: Is External Validity Always Necessary? Mook (1983) pointed out four alternative goals of research that do not stress external validity: We may merely want to find out if something can happen (not whether it actually happens). We may be predicting from the real world to the lab – seeing a phenomenon in the real world, we think it will operate in a certain manner in the lab.

The Devil’s Advocate: Is External Validity Always Necessary? Mook (1983) pointed out four alternative goals of research that do not stress external validity: We may merely want to find out if something can happen (not whether it actually happens). We may be predicting from the real world to the lab – seeing a phenomenon in the real world, we think it will operate in a certain manner in the lab. If we can demonstrate that a phenomenon occurs in a lab’s unnatural setting, the validity of the phenomenon may actually be strengthened.

The Devil’s Advocate: Is External Validity Always Necessary? Mook (1983) pointed out four alternative goals of research that do not stress external validity: We may merely want to find out if something can happen (not whether it actually happens). We may be predicting from the real world to the lab – seeing a phenomenon in the real world, we think it will operate in a certain manner in the lab. If we can demonstrate that a phenomenon occurs in a lab’s unnatural setting, the validity of the phenomenon may actually be strengthened. We may study phenomena in the lab that don’t even have a real- world analogy.

The Devil’s Advocate: Is External Validity Always Necessary? Replication An additional scientific study that is conducted in exactly the same manner as the original research project. When we replicate an experimental finding, we are able to place more confidence in that result.

The Devil’s Advocate: Is External Validity Always Necessary? Replication An additional scientific study that is conducted in exactly the same manner as the original research project. When we replicate an experimental finding, we are able to place more confidence in that result. Replication With Extension An experiment that seeks to confirm (replicate) a previous finding but does so in a different setting or with different participants or under different conditions.

Reliability Reliability Refers to the extent that the test or inventory is consistent in its evaluation of the same individuals over repeated administrations. The greater the similarity between scores produced by the same individuals on repeated administrations, the greater the reliability of the test or inventory.

Procedures Reliability (assessment) Test-retest procedure

Procedures Reliability (assessment) Test-retest procedure The test is simply given a second time and scores from the two tests are compared; the greater the similarity, the higher the reliability

Procedures Reliability (assessment) Test-retest procedure Split-Half technique

Procedures Reliability (assessment) Test-retest procedure Split-Half technique Involves dividing a test or inventory into two halves or subtests and then administering them to the same individuals on different occasions or administering the entire test and then splitting it into two halves.