Regression Discontinuity Design Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides.

Slides:



Advertisements
Similar presentations
Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity Threats to Validity.
Advertisements

Ch 8: Experimental Design Ch 9: Conducting Experiments
EMR 6550: Experimental and Quasi- Experimental Designs Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013.
Experimental Design True Experimental Designs n Random assignment n Two comparison groups n Controls threats to internal validity n Strongest evidence.
GROUP-LEVEL DESIGNS Chapter 9.
Experimental Research Designs
Analyzing Regression-Discontinuity Designs with Multiple Assignment Variables: A Comparative Study of Four Estimation Methods Vivian C. Wong Northwestern.
Regression Discontinuity Design Saralyn Miller Southern Methodist University ____________________ Paper presented at the annual meeting of the Southwest.
Slides to accompany Weathington, Cunningham & Pittenger (2010), Chapter 4: An Overview of Empirical Methods 1.
Research Design and Validity Threats
Who are the participants? Creating a Quality Sample 47:269: Research Methods I Dr. Leonard March 22, 2010.
RESEARCH DESIGN : 1. Kinds of support for making CAUSAL interpretations of observed relationships quality of theory research design used measurement procedures.
Types of Group Designs ____________ group design. The experiment compares groups that receive or _______________ the IV (control group) e.g., behavior.
Chapter 11 Quasi-Experimental Designs ♣ ♣ Introduction   Nonequivalent Comparison Group Design   Time-Series Design   Regression Discontinuity Design.
Probability Sampling uses random selection
Chapter 9 Experimental Research Gay, Mills, and Airasian
McGraw-Hill © 2006 The McGraw-Hill Companies, Inc. All rights reserved. Experimental Research Chapter Thirteen.
Experimental Research
EVALUATING YOUR RESEARCH DESIGN EDRS 5305 EDUCATIONAL RESEARCH & STATISTICS.
Chapter 8 Experimental Research
Experimental Design The Gold Standard?.
I want to test a wound treatment or educational program but I have no funding or resources, How do I do it? Implementing & evaluating wound research conducted.
Difference Two Groups 1. Content Experimental Research Methods: Prospective Randomization, Manipulation Control Research designs Validity Construct Internal.
Experimental and Quasi-Experimental Designs
Research Design for Quantitative Studies
Chapter 3 The Research Design. Research Design A research design is a plan of action for executing a research project, specifying The theory to be tested.
Day 6: Non-Experimental & Experimental Design
EVAL 6970: Cost Analysis for Evaluation Dr. Chris L. S. Coryn Nick Saxton Fall 2014.
Learning Objectives Copyright © 2004 John Wiley & Sons,Inc Primary Data Collection: Experimentation CHAPTER Seven.
Learning Objectives Copyright © 2002 South-Western/Thomson Learning Primary Data Collection: Experimentation CHAPTER eight.
Chapter Eight. Figure 8.1 Relationship of Experimentation to the Previous Chapters and the Marketing Research Process Focus of This Chapter Relationship.
Experimentation Chapter 14 Cooper and Schindler. What is Experimentation? Causal method Allow the researcher to alter systematically the variables of.
URBDP 591 A Lecture 8: Experimental and Quasi-Experimental Design Objectives Basic Design Elements Experimental Designs Comparing Experimental Design Example.
Chapter Four Experimental & Quasi-experimental Designs.
1 Experimental Research Cause + Effect Manipulation Control.
CAUSAL INFERENCE Presented by: Dan Dowhower Alysia Cohen H 615 Friday, October 4, 2013.
Experimental Designs. Experiments are conducted to identify how independent variables influence some change in a dependent variable.
Research methods and statistics.  Internal validity is concerned about the causal-effect relationship in a study ◦ Can observed changes be attributed.
WWC Standards for Regression Discontinuity Study Designs June 2010 Presentation to the IES Research Conference John Deke ● Jill Constantine.
EXPERIMENTAL EPIDEMIOLOGY
Chapter 6 Research Validity. Research Validity: Truthfulness of inferences made from a research study.
Research Design ED 592A Fall Research Concepts 1. Quantitative vs. Qualitative & Mixed Methods 2. Sampling 3. Instrumentation 4. Validity and Reliability.
Chapter 8 – Lecture 6. Hypothesis Question Initial Idea (0ften Vague) Initial ObservationsSearch Existing Lit. Statement of the problem Operational definition.
Experimental & Quasi-Experimental Designs Dr. Guerette.
Research Design. Time of Data Collection Longitudinal Longitudinal –Panel study –Trend study –Cohort study Cross-sectional Cross-sectional.
Quasi-Experimental Designs Slides Prepared by Alison L. O’Malley Passer Chapter 11.
The Experiment Chapter 7. Doing Experiments In Everyday Life Experiments in psychology use the same logic that guides experiments in biology or engineering.
Chapter Eight: Quantitative Methods
Experimental and Ex Post Facto Designs
Chapter 11 Experimental Designs PowerPoint presentation developed by: Sarah E. Bledsoe & E. Roberto Orellana.
David M. Murray, Ph.D. Associate Director for Prevention Director, Office of Disease Prevention Multilevel Intervention Research Methodology September.
Research Methodology Lecture No :19 (Experimental Design-Cont)
William M. Trochim James P. Donnelly Kanika Arora 8 Introduction to Design.
8 Experimental Research Design.
Experimental and Quasi-Experimental Research
Causation & Experimental Design
Lezione di approfondimento su RDD (in inglese)
Experiments Why would a double-blind experiment be used?
Experimental Research Designs
Chapter 8 Experimental Design The nature of an experimental design
Chapter Eight: Quantitative Methods
Making Causal Inferences and Ruling out Rival Explanations
Introduction to Design
Chapter 6 Research Validity.
Variables and Measurement (2.1)
Experiments and Quasi-Experiments
Quasi-Experimental Design
Experiments and Quasi-Experiments
External Validity.
Chapter 18: Experimental and Quasi-Experimental Research
Presentation transcript:

Regression Discontinuity Design Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides

RD Designs A pretest-posttest, program-comparison group strategy Review: Advantages of Pre-tests? Detect differences between groups Detect potential vulnerability to internal validity threats Helps with statistical analysis Advantages of Comparison groups? Helps control sources of error Helps support the counterfactual inference

Underuse of RD? Why? It’s new. Key criteria must be met for use. Perhaps it’s just misunderstood.

Overview of RD O A CXO 2 O A CO 2 “pre” is ANY continuous variable that correlates with the outcome of interest Assignment based on cutoff score Regression line should have vertical displacement at the cutoff score if there is an effect

No Treatment Effect

Positive Effect

Examples  Campell & Stanley’s Ivy League Education Example  Trochim’s Hospital Administration Example Hospital Quality of Care

More about assignment Assignment variables: Must be continuous (or ordinal) Can be a pretest on a dependent variable Can be by order of entry into study Cannot be caused by treatment May or may not be related to the outcome If implementing an RD design in your area of research, what variables would you choose for assignment?

Choosing the Cutoff Score Now referring to the assignment variable(s) you identified, how would you arrive at a cutoff score? Substantive grounds: professional judgment Need or Merit Clinical diagnosis Practical grounds: Available data sets Available resources

Choosing the Cutoff Score Mean of the distribution of assignment scores Politically defined thresh-holds Composite scores of assignment variables Important: Assignment must be controlled! (It is as important as proper random assignment.)

Additional Considerations Functional form relating the assignment and outcome variables A defined population in which it is possible for all units in the study to receive Tx regardless of the choice of a certain cutoff point Intent-to-Treat? : Tx diffusion and cross-over participants

Variations 1. Compare 2 treatment groups 2. Compare 3 conditions 3. Different dose treatment groups 4. Multiple cutoff points 5. …and many more creative ways to think of

Theory of RD – How does this work? RDs as Treatment Effects in REs RE pretest means of Tx and Control groups nearly identical at would be the cutoff score in an RD design through random assignment Cutoff-based assignments creates groups with different pretest means and non-overlapping pretest distributions RD compares regression lines, not means Both RDs and REs control for selection bias Unknown variables do not determine assignment Pretests have no error IF used as the selection variable Regression lines are not affected by posttest errors

Adherence to the Cutoff Overrides of the cutoff Crossovers Attrition “Fuzzy” regression discontinuity

Threats to Validity Statistical Conclusion Validity Nonlinearity Interactions Internal Validity – must occur at the cutoff point History Maturation Mortality Selection-instrumentation

Interaction

Group Exercise: RD Design

Analytical Assumptions No exceptions to the cutoff Adhere to true function of the pre-post relationship Uniform delivery of pretest and program

Combining RD with Randomized Experiments 7 combo examples 3 advantages: Increased power Allows estimation of both groups at the overlap interval Adds clarity to the cutoff point

RD – Quasi-experiment? shortfalls are not yet clear Requires more “demanding statistical analysis” Less statistical power see table 7.2 in SCC (pg. 243)

Analysis Problems The Curvilinear Problem

Steps to Analysis Transform the pretest Examine the relationship visually Specify high order terms and interactions Estimate the initial model Refine

Comparing RD Designs with Experimental Designs IN theory, both designs should produce similar results when all exemplary conditions of each method type exist Question remains do they produce similar results and standard errors in practice (real world settings)? Under exemplary conditions, experiments are 2.75 times more efficient than RDDs If otherwise, the degree of this efficiency will vary Central Question: How to compare these two design options in field settings? Cook, Shadish & Wong 2008

Statistical Power for GRT and RDD the RDD has approximately 36 per cent the efficiency of the GRT. This implies that the RDD will require approximately 2.75 times more groups than a GRT with the same power. The same result was found by Schochet [16] for hierarchical models in education and by Cappelleri and Trochim [31] for trials targeting individuals rather than groups. Pennell et al., 2010

Within Study Comparisons: Proposed methodology from LaLonde Causal estimates derived from an experiment compared with estimates from a non-experiment Same Tx Group Different Control Group Modifications needed to use for RDDs

7 Criteria to Improve Interpretation of Within-Study Comparisons 1. Must demonstrate variation in types of methods being contrasted 2. Both assignment mechanisms cannot be correlated with other factors related to outcome variables 3. The RE must “deserve” its status of the causal “Gold Standard” 4. The non-experiment design must also be good

7 Criteria to Improve Interpretation of Within-Study Comparisons 5. Both study types should estimate the same causal quantity 6. Explicit criteria must be raised on how the two design estimates relate to each other 7. Blind that data analyst!

Further Discussion? Nagging Questions? …or Inspirations?