EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012.

Slides:



Advertisements
Similar presentations
Andrea M. Landis, PhD, RN UW LEAH
Advertisements

Tests of Hypotheses Based on a Single Sample
Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity Threats to Validity.
Chapter 8: Construct and External Validity in Experimental Research
Experimental Research Designs
Statistical Issues in Research Planning and Evaluation
EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013.
Chapter 8 Construct and External Validity in Experimental Research ♣ ♣ Construct Validity   External Validity   Cautions in Evaluating the External.
1. Estimation ESTIMATION.
Review: What influences confidence intervals?
Research Methods in MIS
Behavioural Science II Week 1, Semester 2, 2002
Experimental Design, Statistical Analysis CSCI 4800/6800 University of Georgia Spring 2007 Eileen Kraemer.
MARE 250 Dr. Jason Turner Hypothesis Testing II. To ASSUME is to make an… Four assumptions for t-test hypothesis testing:
Evaluating Hypotheses Chapter 9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics.
Evaluating Hypotheses Chapter 9 Homework: 1-9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics ~
Statistics Micro Mini Threats to Your Experiment!
PSY 1950 Confidence and Power December, Requisite Quote “The picturing of data allows us to be sensitive not only to the multiple hypotheses that.
EVAL 6970: Experimental and Quasi-Experimental Designs
Today Concepts underlying inferential statistics
Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.
Chapter 14 Inferential Data Analysis
Richard M. Jacobs, OSA, Ph.D.
Descriptive Statistics
Inferential Statistics
Copyright (c) 2004 Brooks/Cole, a division of Thomson Learning, Inc. Chapter 8 Tests of Hypotheses Based on a Single Sample.
Chapter 12 Inferential Statistics Gay, Mills, and Airasian
Thomas Songer, PhD with acknowledgment to several slides provided by M Rahbar and Moataza Mahmoud Abdel Wahab Introduction to Research Methods In the Internet.
Fig Theory construction. A good theory will generate a host of testable hypotheses. In a typical study, only one or a few of these hypotheses can.
Statistics 11 Hypothesis Testing Discover the relationships that exist between events/things Accomplished by: Asking questions Getting answers In accord.
Chapter 4 Hypothesis Testing, Power, and Control: A Review of the Basics.
Testing Hypotheses I Lesson 9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics n Inferential Statistics.
HYPOTHESIS TESTING Dr. Aidah Abu Elsoud Alkaissi
Chapter 8 Introduction to Hypothesis Testing
Construct Validity And its Threats Jill Hoxmeier H615: Advanced Research Design October 10, 2013 “Boy, a few more like that and I’ll be ready for Gamblers.
Chapter 8 Introduction to Hypothesis Testing
Statistical Fundamentals: Using Microsoft Excel for Univariate and Bivariate Analysis Alfred P. Rovai Hypothesis Testing PowerPoint Prepared by Alfred.
 Internal Validity  Construct Validity  External Validity * In the context of a research study, i.e., not measurement validity.
Chapter 9 Power. Decisions A null hypothesis significance test tells us the probability of obtaining our results when the null hypothesis is true p(Results|H.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Techniques of research control: -Extraneous variables (confounding) are: The variables which could have an unwanted effect on the dependent variable under.
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 11 Enhancing Rigor in Quantitative Research.
Research Process Parts of the research study Parts of the research study Aim: purpose of the study Aim: purpose of the study Target population: group whose.
Inference and Inferential Statistics Methods of Educational Research EDU 660.
Education 793 Class Notes Decisions, Error and Power Presentation 8.
Issues concerning the interpretation of statistical significance tests.
CHAPTER 2 Research Methods in Industrial/Organizational Psychology
Research Design ED 592A Fall Research Concepts 1. Quantitative vs. Qualitative & Mixed Methods 2. Sampling 3. Instrumentation 4. Validity and Reliability.
Review I A student researcher obtains a random sample of UMD students and finds that 55% report using an illegally obtained stimulant to study in the past.
URBDP 591 I Lecture 4: Research Question Objectives How do we define a research question? What is a testable hypothesis? How do we test an hypothesis?
European Patients’ Academy on Therapeutic Innovation The Purpose and Fundamentals of Statistics in Clinical Trials.
CJ490: Research Methods in Criminal Justice UNIT #4 SEMINAR Professor Jeffrey Hauck.
Chapter 13 Understanding research results: statistical inference.
Construct validity s.net/kb/consthre.htm.
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
Inferential Statistics Psych 231: Research Methods in Psychology.
Chapter 9 Introduction to the t Statistic
Some Terminology experiment vs. correlational study IV vs. DV descriptive vs. inferential statistics sample vs. population statistic vs. parameter H 0.
Dr.MUSTAQUE AHMED MBBS,MD(COMMUNITY MEDICINE), FELLOWSHIP IN HIV/AIDS
Understanding Results
CHAPTER 2 Research Methods in Industrial/Organizational Psychology
2 independent Groups Graziano & Raulin (1997).
Review: What influences confidence intervals?
Group Experimental Design
Psych 231: Research Methods in Psychology
Psych 231: Research Methods in Psychology
Interpreting Epidemiologic Results.
Psych 231: Research Methods in Psychology
Testing Hypotheses I Lesson 9.
Statistical Power.
Presentation transcript:

EVAL 6970: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Dr. Anne Cullen Spring 2012

Agenda Statistical power/design sensitivity Statistical conclusion validity and internal validity

Statistical Power/Design Sensitivity

Types of Hypotheses General forms: – Superiority Nondirectional or directional – Equivalence and noninferiority Within a prespecified bound

Accept-Reject Dichotomy H 0 trueH 0 false Fail to Accept Correct decision 1 – α Type II error β Fail to Reject Type I error α Correct decision 1 – β

Type I Error Type I error (sometimes referred to as a false-positive) is the conditional prior probability of rejecting H 0 when it is true, where this probability is typically expressed as alpha (α) Alpha is a prior probability because it is specified prior to data gathering, and it is a conditional probability because H 0 is assumed to be true and can be expressed as α = p (Reject H 0 | H 0 true)

Type II Error Power is the conditional prior probability of making the correct decision to reject H 0 when it is actually false, where Power = p (Reject H 0 | H 0 false) Type II error (often referred to as a false- negative) occurs when the sample result leads to the failure to reject H 0 when it is actually false, and it also is a conditional prior probability, where β = p (Fail to reject H 0 | H 0 false)

Type II Error Because power and β are complimentary Power + β = 1.00 Whatever increases power decreases the probability of a Type II error and vice versa Several factors affect statistical power, including α levels, sample size, score reliability, design elements (e.g., within- subject designs, covariates), and the magnitude of an effect in a population

Determinants of Power Four primary factors (there are others) that affect design sensitivity/statistical power – Sample size – Alpha level – Statistical tests – Effect size By lowering α, for example, statistical power is lost, thus reducing the likelihood of a Type I error, which simultaneously increases the probability of a Type II error

Sample Size Statistical significance testing is concerned with sampling error, the discrepancy between sample values and population parameters Sampling error is smaller for larger samples and therefore less likely to obscure real differences and increase statistical power

Alpha Alpha levels influence the likelihood of statistical significance Larger alpha levels make significance easier to attain than smaller levels When the null hypothesis is false, statistical power increases as alpha increases

Statistical Tests Tests of statistical significance are made within the framework of particular statistical tests The test itself is one of the factors affecting statistical power Some tests are more sensitive than others (e.g., analysis of covariance)

Effect Size The larger the true effect, the greater the probability of statistical significance and the greater the statistical power

Basic Approaches to Power 1.Power determination approach (post hoc) – Begins with an assumption about an effect size – The aim is to determine the power to detect an effect size with a given sample size 2.Effect size approach (a priori) – Begins with a desired level of power to estimate a minimum detectable effect size (MDES) at a prespecified level of power

Working with Power and Precision 2.0 and 3.0

Construct Validity & External Validity

Construct Validity

The degree to which inferences are warranted from the observed persons, settings, treatments, and outcome (cause-effect) operations sampled within a study to the constructs that these samples represent

Construct Validity Most constructs of interest do not have a natural units of measurement Nearly all empirical studies are studies of specific instances of persons, settings, treatments, and outcomes and require inferences to the higher order constructs represented by sampled instances

Why Construct Inferences are a Problem Names reflect category memberships that have implications about relationships to other concepts, theories, and uses (i.e., nomonological network) In the social sciences it is nearly impossible to establish a one-to-one relationship between the operations of a study and corresponding constructs

Why Construct Inferences are a Problem Construct validity is fostered by: 1.Clear explication of person, treatment, setting, and outcome constructs of interest 2.Careful selection of instances that match constructs 3.Assessment of match between instances and constructs 4.Revision of construct descriptions (if necessary)

Assessment of Sampling Particulars All sampled instances of persons, settings, treatments, and outcomes should be carefully assessed using whatever methods (i.e., quantitative, qualitative, etc.) necessary to assure a match between higher order constructs and sampled instances (i.e., careful explication)

A Note about “Operations” To operationalize is to define a concept or variable in such a way that it can be measured or defined (i.e., operated on) A operational definition is a description of the way a variable will be observed and measured – It specifies the actions [operations] that will be taken to measure a variable

Threats to Construct Validity 1.Inadequate explication of constructs. Failure to adequately explicate a construct may lead to incorrect inferences about the relationship between operation and construct 2.Construct confounding. Operations usually involve more than one construct, and failure to describe all constructs may result in incomplete construct inferences 3.Mono-operation bias. Any one operationalization of a construct both underrepresents the construct of interest and measure irrelevant constructs, complicating inferences 4.Mono-method bias. When all operationalizations use the same method (e.g., self-report), that method is part of the construct actually studied 5.Confounding construct with levels of constructs. Inferences about the constructs that best represent study operations may fail to describe the limited levels of the construct studied

Threats to Construct Validity 6.Treatment sensitive factorial structure. The structure of a measure may change as a result of treatment, change that may be hidden if the same scoring is always used 7.Reactive self-report changes. Self-reports can be affected by participants motivation to be in a treatment condition, motivation that can change after assignment has been made 8.Reactivity to experimental situation. Participant responses reflect not just treatments and measures but also participants’ perceptions of the experimental situation, and those perceptions are actually part of the treatment construct 9.Experimenter expectancies. The experimenter can influence participant responses by conveying expectations about desirable responses, and those responses are part of the treatment construct 10.Novelty and disruption effects. Participants may respond unusually well to a novel innovation or unusually poorly to one that disrupts their routine, a response that must then be included as part of the treatment construct definition

Threats to Construct Validity 11.Compensatory equalization. When treatment provides desirable goods or services, administrators, staff, or constituents may provide compensatory goods or services to those not receiving treatment, and this action must be included as part of the treatment construct description 12.Compensatory rivalry. Participants not receiving treatment may be motivated to show they can do as well as those receiving treatment, and this must be included as part of the treatment construct 13.Resentful demoralization. Participants not receiving a desirable treatment may be so resentful or demoralized that they respond more negatively than otherwise, and this must be included as part of the treatment construct 14.Treatment diffusion. Participants may receive services from a condition to which they were not assigned, making construct definitions of both conditions difficult

External Validity

The degree to which inferences about the extent to which a causal relationship holds over variations in persons, settings, treatments, and outcomes

External Validity Inferences to (1) those who were in an experiment or (2) those who were not Narrow to broad Broad to narrow At a similar level To a similar or different kind Random sample to population members

Threats to External Validity 1.Interaction of the causal relationship with units. An effect found when certain kinds of units might not hold if other types of units had been studied 2.Interaction of the causal relationship over treatment variations. An effect found with one treatment variation might not hold with other variations of the treatment, or when that treatment is combined with other treatments, or when only part of a treatment is used 3.Interaction of the causal relationship with outcomes. An effect found on one kind of outcome observation may not hold if other outcome observations were used 4.Interaction of the causal relationship with settings. An effect found in one kind of setting may not holds in other settings 5.Context-dependent mediation. An explanatory mediator of a causal relationship in one context may not mediate in another

Constancy of Effect Size versus Constancy of Causal Direction Arguably, few causal relationships in the social world have consistent effect sizes A better method of generalization is constancy of causal direction

Random Sampling and External Validity Random sampling has benefits for external validity, but poses practical limitations in experiments Random samples of persons not common in experiments, but sometimes feasible Random samples of settings are rare, but increasing with the advent of place- based experiments Random samples of treatments and outcomes are even more rare

The Relationship Between Construct Validity and External Validity Both are generalizations Valid knowledge of constructs can provide valuable knowledge about external validity They differ in the kinds of inferences being made – Construct validity to sampled instances – External validity to whether the size or direction of a causal relationship holds over variations in persons, settings, treatments, and outcomes Can be right about one and not the other