Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 11 Enhancing Rigor in Quantitative Research

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Types of Validity Statistical conclusion validity Internal validity Construct validity External validity

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Controlling Intrinsic Source of Extraneous Variability Randomization Crossover Homogeneity Blocking/Stratification Matching (pair matching) Statistical control

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Benefits and Limitations of Control Methods: Randomization Benefits: Controls all extraneous variables Does not require advance knowledge of which variables to control Limitations: Ethical and practical constraints on manipulation Possible artificiality of conditions

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Benefits and Limitations of Control Methods: Crossover (Repeated Measures) Benefits: If done with randomization, strongest possible approach Reduces sample size requirements Limitations: Cannot be used if there are possible carryover effects from one condition to another

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Benefits and Limitations of Control Methods: Homogeneity Benefits: Easy to achieve Enhances interpretability of relationships Limitations: Limits generalizability Requires knowledge of which variables to control

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Benefits and Limitations of Control Methods: Blocking Benefits: Enhances interpretability of relationships Offers possibility of examining blocking variable as an independent variable Limitations: Manageable only with a few blocking variables Requires knowledge of which variables to control

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Benefits and Limitations of Control Methods: Matching Benefits: Enhances interpretability of relationships Easy to do if there is a large pool of available comparison group subjects Limitations: Manageable only with a few matching variables Requires knowledge of which variables to control May be hard to find comparison group matches

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Benefits and Limitations of Control Methods: Statistical Studies Benefits: Enhances interpretability of relationships Easy and economical Can be used with a large number of extraneous variables Limitations: Requires knowledge of which variables to control Requires statistical sophistication

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Threats to Statistical Conclusion Validity Low statistical power Low precision Factors that undermine a strong operationalization of the independent variables Inadequate participation in treatment conditions

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Threats to Internal Validity Temporal ambiguity Selection History Maturation Mortality/Attrition Testing and Instrumentation

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Research Designs and Threats to Internal Validity

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Construct Validity Concerns inferences from the particular exemplars of a study to the higher order constructs that they are intended to represent

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Threats to Construct Validity Reactivity to the study situations Researcher expectancies Novelty effects Compensatory effects Treatment diffusion or contamination

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins External Validity Concerns inferences about the extent to which relationships observed in a study hold true over variations in people, conditions, and setting, as well as over variations in treatments and outcomes

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Threats to External Validity Interaction between relationship and people Interaction between causal effects and treatment variation

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Accessible vs. Target Population Accessible population: The population available for a particular study Target population: The total group of people in whom a researcher is interested and to whom results could be generalized

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Threats to External Population Inadequate sampling Expectancy effects Novelty effects Interaction of history and treatment effects Experimenter effects Measurement effects