1. Hypothesis Testing, Validity, and Threats to Validity
Graziano and Raulin
Research Methods: Chapter 8
This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.
Copyright © Allyn & Bacon (2007)
Graziano & Raulin (1997)

2. Testing Hypotheses Research tests specific hypotheses
Generated from the initial research idea through a series of steps
A research idea can generate dozens of research hypotheses depending on how
it is translated into a statement of the problem
the variables are operationally defined
Copyright © Allyn & Bacon (2007)

3. The Initial Idea The initial idea is the starting point
Often vague or general
It requires refining before research hypotheses can be generated
Refinement of the initial idea is based on
(1) a search of relevant research literature
(2) initial observations of the phenomenon
Narrow and formalize the initial idea into a statement of the problem
Copyright © Allyn & Bacon (2007)

4. Statement of the Problem
In the form of a question
Clearly indicates an expected relationship
Nature of the question dictates the required level of constraint of a study
Causal questions will require experimental research
Questions about relationships can be answered with lower-constraint research
Convert into research hypotheses by operationally defining the variables
Copyright © Allyn & Bacon (2007)

5. Generating Research Hypotheses
Ideas lead to
observations
library research
Statement of problem
Problem statements become research hypotheses when constructs are operationalized
Copyright © Allyn & Bacon (2007)

6. Operational Definitions
The procedures used to measure and/or manipulate variables
Most variables can be operationally defined in many different ways
Copyright © Allyn & Bacon (2007)

7. Research Hypotheses
State clearly the expected relationship between the variables
The form is a declarative statement, but it is a tentative statement to be tested in research
Variables in research hypotheses are stated in operational definition terms
Copyright © Allyn & Bacon (2007)

8. The Role of Theory to the research Hypothesis
Theory guides all research planning
Often the primary source of the research hypothesis
Guides the selection of variables
Guides the operational definitions of variables
Most research is based on multiple, overlapping, and interacting theories
Copyright © Allyn & Bacon (2007)

9. Testing Research Hypotheses
Actually testing three hypotheses
The null or statistical hypothesis
The confounding variable hypothesis
The causal hypothesis
Accept causal hypothesis only if you
reject null hypothesis (statistical analysis)
rule out each potential confounding variable hypothesis (based on appropriate controls)
Copyright © Allyn & Bacon (2007)

10. Evaluating Hypotheses
Copyright © Allyn & Bacon (2007)

11. Types of Validity Statistical Validity Construct Validity
External Validity
Internal Validity
Copyright © Allyn & Bacon (2007)

12. Statistical Validity Are the statistical tests accurate? Threatened by
Unreliable measures
Violations of statistical assumptions
Strengthened by
Using well validated measures
Having approximately equal sample sizes in each group (More details about choosing the right sample will be discussed later).
Copyright © Allyn & Bacon (2007)

13. Construct Validity Is our theory the best explanation for the results?
Threatened by
Any alternative explanation for the results
Strengthened by
Using well-validated constructs to build the theoretical predictions for the study
Copyright © Allyn & Bacon (2007)

14. External Validity Do the results apply to the broader population?
Threatened by
Unrepresentative samples
Generalizing beyond the limits of the sample
Strengthened by
Gathering a representative sample (if possible)
Clearly describing sample, so that other researchers will know the limits of generalization
Copyright © Allyn & Bacon (2007)

15. Internal Validity
Is the independent variable responsible for the observed changes in the dependent variable?
Threatened by
Confounding variables
Strengthened by
Adding adequate controls to reduce or eliminate confounding
Copyright © Allyn & Bacon (2007)

16. Avoiding Confounding Confounding and internal validity
Many sources for confounding (covered next)
With proper controls, confounding can be virtually eliminated
Confounding and construct validity
Make sure that you have considered alternative theoretical explanations for the anticipated phenomenon
Copyright © Allyn & Bacon (2007)

17. Confounding Variables 1
Maturation
Changes due to growth or predictable changes
History
Changes due to an event that occurs during the study
Testing
Changes due to the effects of previous testing
Copyright © Allyn & Bacon (2007)

18. Confounding Variables 2
Instrumentation
Any change in the calibration of the measuring instrument over the course of the study
Regression to the Mean
Tendency for participants selected because of extreme scores to be less extreme on a retest
Selection
Any factor that creates groups that are not equal at the start of the study
Copyright © Allyn & Bacon (2007)

19. Confounding Variables 3
Attrition
Loss of participants during a study; are the participants who drop out different from those who continue?
Diffusion of treatment
Changes in participants’ behavior due to information they obtained about other conditions
Sequence Effects
Effects on performance in one condition due to experience with previous conditions
Copyright © Allyn & Bacon (2007)

20. Subject Effects Participants are not passive
They try to understand the study to help them to know what they “should do” (termed subject effects)
Respond to subtle cues about what is expected (termed demand characteristics)
Placebo effect: treatment effect due to expectations that the treatment will work
Copyright © Allyn & Bacon (2007)

21. Experimenter Effect Based on the expectations of the researcher
Affects the outcome of studies if not controlled
May be due to the experimenter providing demand characteristics to the participant
Not the same as scientific fraud (which is deliberate)
Copyright © Allyn & Bacon (2007)

22. Validity, Control, and Constraint
Three closely-tied concepts
Validity
The accuracy of the study or procedure
Increased by using appropriate control procedures
The more controls we employ, the higher the level of constraint
Copyright © Allyn & Bacon (2007)

23. Summary Start by building a research hypothesis
Testing the research hypothesis is actually testing three hypotheses
(1) null; (2) confounding-variable; (3) causal
Several types of validity
Many potential confounding variables
Subject and experimenter effects can also affect the outcome of the study
Copyright © Allyn & Bacon (2007)