Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program. Copyright © Allyn & Bacon (2007) Graziano & Raulin (1997)

Testing Hypotheses Research tests specific hypotheses Generated from the initial research idea through a series of steps A research idea can generate dozens of research hypotheses depending on how it is translated into a statement of the problem the variables are operationally defined Copyright © Allyn & Bacon (2007)

The Initial Idea The initial idea is the starting point Often vague or general It requires refining before research hypotheses can be generated Refinement of the initial idea is based on (1) a search of relevant research literature (2) initial observations of the phenomenon Narrow and formalize the initial idea into a statement of the problem Copyright © Allyn & Bacon (2007)

Statement of the Problem In the form of a question Clearly indicates an expected relationship Nature of the question dictates the required level of constraint of a study Causal questions will require experimental research Questions about relationships can be answered with lower-constraint research Convert into research hypotheses by operationally defining the variables Copyright © Allyn & Bacon (2007)

Generating Research Hypotheses Ideas lead to observations library research Statement of problem Problem statements become research hypotheses when constructs are operationalized Copyright © Allyn & Bacon (2007)

Operational Definitions The procedures used to measure and/or manipulate variables Most variables can be operationally defined in many different ways Copyright © Allyn & Bacon (2007)

Research Hypotheses State clearly the expected relationship between the variables The form is a declarative statement, but it is a tentative statement to be tested in research Variables in research hypotheses are stated in operational definition terms Copyright © Allyn & Bacon (2007)

The Role of Theory to the research Hypothesis Theory guides all research planning Often the primary source of the research hypothesis Guides the selection of variables Guides the operational definitions of variables Most research is based on multiple, overlapping, and interacting theories Copyright © Allyn & Bacon (2007)

Testing Research Hypotheses Actually testing three hypotheses The null or statistical hypothesis The confounding variable hypothesis The causal hypothesis Accept causal hypothesis only if you reject null hypothesis (statistical analysis) rule out each potential confounding variable hypothesis (based on appropriate controls) Copyright © Allyn & Bacon (2007)

Evaluating Hypotheses Copyright © Allyn & Bacon (2007)

Types of Validity Statistical Validity Construct Validity External Validity Internal Validity Copyright © Allyn & Bacon (2007)

Statistical Validity Are the statistical tests accurate? Threatened by Unreliable measures Violations of statistical assumptions Strengthened by Using well validated measures Having approximately equal sample sizes in each group (More details about choosing the right sample will be discussed later). Copyright © Allyn & Bacon (2007)

Construct Validity Is our theory the best explanation for the results? Threatened by Any alternative explanation for the results Strengthened by Using well-validated constructs to build the theoretical predictions for the study Copyright © Allyn & Bacon (2007)

External Validity Do the results apply to the broader population? Threatened by Unrepresentative samples Generalizing beyond the limits of the sample Strengthened by Gathering a representative sample (if possible) Clearly describing sample, so that other researchers will know the limits of generalization Copyright © Allyn & Bacon (2007)

Internal Validity Is the independent variable responsible for the observed changes in the dependent variable? Threatened by Confounding variables Strengthened by Adding adequate controls to reduce or eliminate confounding Copyright © Allyn & Bacon (2007)

Avoiding Confounding Confounding and internal validity Many sources for confounding (covered next) With proper controls, confounding can be virtually eliminated Confounding and construct validity Make sure that you have considered alternative theoretical explanations for the anticipated phenomenon Copyright © Allyn & Bacon (2007)

Confounding Variables 1 Maturation Changes due to growth or predictable changes History Changes due to an event that occurs during the study Testing Changes due to the effects of previous testing Copyright © Allyn & Bacon (2007)

Confounding Variables 2 Instrumentation Any change in the calibration of the measuring instrument over the course of the study Regression to the Mean Tendency for participants selected because of extreme scores to be less extreme on a retest Selection Any factor that creates groups that are not equal at the start of the study Copyright © Allyn & Bacon (2007)

Confounding Variables 3 Attrition Loss of participants during a study; are the participants who drop out different from those who continue? Diffusion of treatment Changes in participants’ behavior due to information they obtained about other conditions Sequence Effects Effects on performance in one condition due to experience with previous conditions Copyright © Allyn & Bacon (2007)

Subject Effects Participants are not passive They try to understand the study to help them to know what they “should do” (termed subject effects) Respond to subtle cues about what is expected (termed demand characteristics) Placebo effect: treatment effect due to expectations that the treatment will work Copyright © Allyn & Bacon (2007)

Experimenter Effect Based on the expectations of the researcher Affects the outcome of studies if not controlled May be due to the experimenter providing demand characteristics to the participant Not the same as scientific fraud (which is deliberate) Copyright © Allyn & Bacon (2007)

Validity, Control, and Constraint Three closely-tied concepts Validity The accuracy of the study or procedure Increased by using appropriate control procedures The more controls we employ, the higher the level of constraint Copyright © Allyn & Bacon (2007)

Summary Start by building a research hypothesis Testing the research hypothesis is actually testing three hypotheses (1) null; (2) confounding-variable; (3) causal Several types of validity Many potential confounding variables Subject and experimenter effects can also affect the outcome of the study Copyright © Allyn & Bacon (2007)