1. Copyright © Allyn & Bacon (2007) Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8 This multimedia product and its contents are protected under copyright law. The following are prohibited by law: (1) Any public performance or display, including transmission of any image over a network; (2) Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental, lease, or lending of the program.

2. Copyright © Allyn & Bacon (2007) Testing Hypotheses Research tests specific hypotheses Research tests specific hypotheses –Generated from the initial research idea through a series of steps A research idea can generate dozens of research hypotheses depending on how A research idea can generate dozens of research hypotheses depending on how –it is translated into a statement of the problem –the variables are operationally defined

3. Copyright © Allyn & Bacon (2007) The Initial Idea The initial idea is the starting point The initial idea is the starting point –Often vague or general –It requires refining before research hypotheses can be generated Refinement of the initial idea is based on Refinement of the initial idea is based on (1) a search of relevant research literature (2) initial observations of the phenomenon Narrow and formalize the initial idea into a statement of the problem Narrow and formalize the initial idea into a statement of the problem

4. Copyright © Allyn & Bacon (2007) Statement of the Problem In the form of a question In the form of a question –Clearly indicates an expected relationship –Nature of the question dictates the required level of constraint of a study Causal questions will require experimental research Causal questions will require experimental research Questions about relationships can be answered with lower-constraint research Questions about relationships can be answered with lower-constraint research Convert into research hypotheses by operationally defining the variables Convert into research hypotheses by operationally defining the variables

5. Copyright © Allyn & Bacon (2007) Generating Research Hypotheses Ideas lead to Ideas lead to –observations –library research Statement of problem Statement of problem Problem statements become research hypotheses when constructs are operationalized Problem statements become research hypotheses when constructs are operationalized

6. Copyright © Allyn & Bacon (2007) Operational Definitions Initially covered in Chapter 3 Initially covered in Chapter 3 The procedures used to measure and/or manipulate variables The procedures used to measure and/or manipulate variables Most variables can be operationally defined in many different ways Most variables can be operationally defined in many different ways

7. Copyright © Allyn & Bacon (2007) Research Hypotheses State clearly the expected relationship between the variables State clearly the expected relationship between the variables The form is a declarative statement, but it is a tentative statement to be tested in research The form is a declarative statement, but it is a tentative statement to be tested in research Variables in research hypotheses are stated in operational definition terms Variables in research hypotheses are stated in operational definition terms

8. Copyright © Allyn & Bacon (2007) The Role of Theory Theory guides all research planning Theory guides all research planning –Often the primary source of the research hypothesis –Guides the selection of variables –Guides the operational definitions of variables Most research is based on multiple, overlapping, and interacting theories Most research is based on multiple, overlapping, and interacting theories

9. Copyright © Allyn & Bacon (2007) Testing Research Hypotheses Actually testing three hypotheses Actually testing three hypotheses –The null hypothesis –The confounding variable hypothesis –The causal hypothesis Accept causal hypothesis only if you Accept causal hypothesis only if you –reject null hypothesis (statistical analysis) –rule out each potential confounding variable hypothesis (based on appropriate controls)

10. Copyright © Allyn & Bacon (2007) Evaluating Hypotheses

11. Copyright © Allyn & Bacon (2007) Types of Validity Statistical Validity Statistical Validity Construct Validity Construct Validity External Validity External Validity Internal Validity Internal Validity

12. Copyright © Allyn & Bacon (2007) Statistical Validity Are the statistical tests accurate? Are the statistical tests accurate? Threatened by Threatened by –Unreliable measures –Violations of statistical assumptions Strengthened by Strengthened by –Using well validated measures –Having approximately equal sample sizes in each group (covered in Chapter 14)

13. Copyright © Allyn & Bacon (2007) Construct Validity Is our theory the best explanation for the results? Is our theory the best explanation for the results? Threatened by Threatened by –Any alternative explanation for the results Strengthened by Strengthened by –Using well-validated constructs to build the theoretical predictions for the study

14. Copyright © Allyn & Bacon (2007) External Validity Do the results apply to the broader population? Do the results apply to the broader population? Threatened by Threatened by –Unrepresentative samples –Generalizing beyond the limits of the sample Strengthened by Strengthened by –Gathering a representative sample (if possible) –Clearly describing sample, so that other researchers will know the limits of generalization

15. Copyright © Allyn & Bacon (2007) Internal Validity Is the independent variable responsible for the observed changes in the dependent variable ? Is the independent variable responsible for the observed changes in the dependent variable ? Threatened by Threatened by –Confounding variables Strengthened by Strengthened by –Adding adequate controls to reduce or eliminate confounding

16. Copyright © Allyn & Bacon (2007) Avoiding Confounding Confounding and internal validity Confounding and internal validity –Many sources for confounding (covered next) –With proper controls, confounding can be virtually eliminated (see Chapter 9) Confounding and construct validity Confounding and construct validity –Make sure that you have considered alternative theoretical explanations for the anticipated phenomenon

17. Copyright © Allyn & Bacon (2007) Confounding Variables 1 Maturation Maturation –Changes due to growth or predictable changes History History –Changes due to an event that occurs during the study Testing Testing –Changes due to the effects of previous testing

18. Copyright © Allyn & Bacon (2007) Confounding Variables 2 Instrumentation Instrumentation –Any change in the calibration of the measuring instrument over the course of the study Regression to the Mean Regression to the Mean –Tendency for participants selected because of extreme scores to be less extreme on a retest Selection Selection –Any factor that creates groups that are not equal at the start of the study

19. Copyright © Allyn & Bacon (2007) Confounding Variables 3 Attrition Attrition –Loss of participants during a study; are the participants who drop out different from those who continue? Diffusion of treatment Diffusion of treatment –Changes in participants’ behavior due to information they obtained about other conditions Sequence Effects Sequence Effects –Effects on performance in one condition due to experience with previous conditions

20. Copyright © Allyn & Bacon (2007) Subject Effects Participants are not passive Participants are not passive –They try to understand the study to help them to know what they “should do” (termed subject effects) –Respond to subtle cues about what is expected (termed demand characteristics) Placebo effect: treatment effect due to expectations that the treatment will work Placebo effect: treatment effect due to expectations that the treatment will work

21. Copyright © Allyn & Bacon (2007) Experimenter Effect Based on the expectations of the researcher Based on the expectations of the researcher Affects the outcome of studies if not controlled Affects the outcome of studies if not controlled May be due to the experimenter providing demand characteristics to the participant May be due to the experimenter providing demand characteristics to the participant Not the same as scientific fraud (which is deliberate) Not the same as scientific fraud (which is deliberate)

22. Copyright © Allyn & Bacon (2007) Validity, Control, and Constraint Three closely-tied concepts Three closely-tied concepts Validity Validity –The accuracy of the study or procedure –Increased by using appropriate control procedures The more controls we employ, the higher the level of constraint The more controls we employ, the higher the level of constraint

23. Copyright © Allyn & Bacon (2007) Summary Start by building a research hypothesis Start by building a research hypothesis Testing the research hypothesis is actually testing three hypotheses Testing the research hypothesis is actually testing three hypotheses –(1) null; (2) confounding-variable; (3) causal Several types of validity Several types of validity Many potential confounding variables Many potential confounding variables Subject and experimenter effects can also affect the outcome of the study Subject and experimenter effects can also affect the outcome of the study