S-005 Intervention research: True experiments and quasi- experiments.

Slides:

Advertisements

Similar presentations

Value Added in CPS. What is value added? A measure of the contribution of schooling to student performance Uses statistical techniques to isolate the.

Advertisements

Control in Experimental Designs. Control u Key element of experimental and quasi- experimental designs –Subjects (one group or several) on which no variable.

Experimental Design True Experimental Designs n Random assignment n Two comparison groups n Controls threats to internal validity n Strongest evidence.

#ieGovern Impact Evaluation Workshop Istanbul, Turkey January 27-30, 2015 Measuring Impact 1 Non-experimental methods 2 Experiments Vincenzo Di Maro Development.

1 Arlene Ash QMC - Third Tuesday September 21, 2010 (as amended, Sept 23) Analyzing Observational Data: Focus on Propensity Scores.

Group Discussion Describe the fundamental flaw that prevents a nonequivalent group design from being a true experiment? (That is, why can’t these designs.

Experimental Research Designs

Quasi-Experimental Design

Using Rubrics for Evaluating Student Learning. Purpose To review the development of rubrics for the purpose of assessment To share an example of how a.

Correlation AND EXPERIMENTAL DESIGN

Experimental Designs.

Research Design and Validity Threats

Wed Oct 29, 2003 Exams after class Mean(SD): 71.6(11.6)% Range: 44-95% Appointment Research day 1 week.

Experimental Design Week 9 Lecture 1.

Experimental Research Strategy

1 Exploring Quasi-Experiments Lab 5: May 9, 2008 Guthrie, J.T., Wigfield, A., & VonSecker, C. (2000). Effects of integrated instruction on motivation and.

Chapter 51 Experiments, Good and Bad. Chapter 52 Experimentation u An experiment is the process of subjecting experimental units to treatments and observing.

Using Rubrics for Evaluating Student Learning Office of Assessment and Accreditation Indiana State University.

Non-Experimental designs: Developmental designs & Small-N designs

Questions What is the best way to avoid order effects while doing within subjects design? We talked about people becoming more depressed during a treatment.

Experimental Designs.

Group Discussion Describe the similarities and differences between experiments , non-experiments , and quasi-experiments. Actions for Describe the similarities.

Experimental Research

S-005 Types of research in education. Types of research A wide variety of approaches: –Theoretical studies –Summaries of studies Reviews of the literature.

Evaluation of Math-Science Partnership Projects (or how to find out if you’re really getting your money’s worth)

Experimental Design The Gold Standard?.

Quasi-experimental Design CRJS 4466EA. Introduction Quasi-experiment Describes non-randomly assigned participants and controls subject to impact assessment.

Introduction Connecting Questions to Design

I want to test a wound treatment or educational program in my clinical setting with patient groups that are convenient or that already exist, How do I.

Dr. Bonnie J. Faddis & Dr. Margaret Beam RMC Research Fidelity of Implementation and Program Impact.

The Power of Comparison Groups Rita O’Sullivan Evaluation, Assessment, & Policy Connections School of Education, University of North Carolina at Chapel.

Global Workshop on Development Impact Evaluation in Finance and Private Sector Rio de Janeiro, June 6-10, 2011 Mattea Stein Quasi Experimental Methods.

URBDP 591 A Lecture 8: Experimental and Quasi-Experimental Design Objectives Basic Design Elements Experimental Designs Comparing Experimental Design Example.

Estimating Causal Effects from Large Data Sets Using Propensity Scores Hal V. Barron, MD TICR 5/06.

Rigorous Quasi-Experimental Evaluations: Design Considerations Sung-Woo Cho, Ph.D. June 11, 2015 Success from the Start: Round 4 Convening US Department.

AFRICA IMPACT EVALUATION INITIATIVE, AFTRL Africa Program for Education Impact Evaluation David Evans Impact Evaluation Cluster, AFTRL Slides by Paul J.

CHAPTER 9: Producing Data: Experiments. Chapter 9 Concepts 2  Observation vs. Experiment  Subjects, Factors, Treatments  How to Experiment Badly 

Non-experimental methods Markus Goldstein The World Bank DECRG & AFTPM.

Quasi-Experiments. Research Objective Show direct cause & effect Study relationships among variables for existing groups Explain outcomes after the fact.

What is randomization and how does it solve the causality problem? 2.3.

Chapter 10 Common Experimental Research Designs

Chapter 10 Finding Relationships Among Variables: Non-Experimental Research.

Nonexperimental and Quasi- Experimental Designs Distinction is the degree of control over internal validity.

1 Module 3 Designs. 2 Family Health Project: Exercise Review Discuss the Family Health Case and these questions. Consider how gender issues influence.

Experimental & Quasi-Experimental Designs Dr. Guerette.

S-005 Types of research in education. Types of research A wide variety of approaches: –Theoretical studies –Summaries of studies Reviews of the literature.

Three ‘R’s for Evaluating the Memphis Striving Readers Project: Relationships, Real-World Challenges, and RCT Design Jill Feldman, RBS Director of Evaluation.

©2005, Pearson Education/Prentice Hall CHAPTER 6 Nonexperimental Strategies.

Producing Data: Experiments BPS - 5th Ed. Chapter 9 1.

PURPOSES OF CONTINUOUS ASSESSMENT Assessment is an integral part of teaching. Observation is your key assessment tool in the primary and junior grades.

S-005 Intervention research: True experiments and quasi- experiments.

Randomized Control Trials: What, Why, How, When, and Where Mark L. Davison MESI Conference March 11, 2016 Department of Educational Psychology.

Patricia Gonzalez, OSEP June 14, The purpose of annual performance reporting is to demonstrate that IDEA funds are being used to improve or benefit.

Chapter 11 Experimental Designs PowerPoint presentation developed by: Sarah E. Bledsoe & E. Roberto Orellana.

Chapter 12 Quantitative Questions and Procedures.

CHAPTER 13: THE NUTS AND BOLTS OF QUASI- EXPERIMENTS.

Issues in Evaluating Educational Research

EXPERIMENTAL RESEARCH

Quasi Experimental Methods I

Experiments Why would a double-blind experiment be used?

Quasi Experimental Methods I

Ron Sterr Kim Sims Heather Cruz aka “The Carpool”

Matching Methods & Propensity Scores

Matching Methods & Propensity Scores

Impact Evaluation Methods

Quasi-Experimental Design

Matching Methods & Propensity Scores

GED 500 – Lesson 6 Single Subject Designs

Impact Evaluation Methods: Difference in difference & Matching

Reminder for next week CUELT Conference.

Presentation transcript:

S-005 Intervention research: True experiments and quasi- experiments

Intervention research Goal is to assess the effects of some intervention or new idea or approach Examples: –Children’s vocabulary growth under a new curriculum –Effects of new technologies on student achievement –New approaches to professional development for teachers –New drug treatments or other interventions –Other examples? Key distinction between –True experiments We compare the new program (or treatment) to a traditional treatment (or alternative treatment or control group) Random assignment is the key ingredient for a study to qualify as a true experiment –Quasi-experiments Comparison of new program with an alternative (“new” vs. “traditional”, for example) But random assignment not used Comparisons may be biased by other factors Need for great care and additional information for these studies to be convincing Quasi-experimental designs are common in education, but they may have some important limitations. (Some describe them as “mere observations” and not a strong basis for drawing conclusions. Others are not quite so harsh.)

True Experiments (for evaluating an intervention) New treatment Control (or traditional or alternative) New Comparison

True Experiments (for evaluating an intervention) New treatment Control (or traditional or alternative) New Comparison Key issue: How do people get assigned to (or chosen for) the two options? Start with a group of eligible people Assign them randomly

True Experiments (for evaluating an intervention) New treatment Control (or traditional or alternative) New Comparison Key issue: How do people get assigned to (or chosen for) the two options? Start with a group of eligible people Assign them randomly Random assignment is the key ingredient required for a true experiment.

True experiments (for assessing an intervention) New Comparison The process of random assignment Makes it very likely that the groups will be similar. The only difference will be which treatment they receive. The groups are very likely to be similar in age, gender ratio, prior experience, etc. Makes it very likely that the groups will be similar. The only difference will be which treatment they receive. The groups are very likely to be similar in age, gender ratio, prior experience, etc. Some cautions: Compliance? Implementation? Sometimes the special conditions under which we carry out a true experiment make it hard to generalize. Others? Some cautions: Compliance? Implementation? Sometimes the special conditions under which we carry out a true experiment make it hard to generalize. Others?

Quasi-experiments Groups are compared, but no random assignment New Comparison Do we really have a good comparison? Can we really be sure about the new program? Often we need lots of additional information to help us draw stronger conclusions?

Quasi-experiments New programTraditional program Post test average8580 What might we conclude?

Quasi-experiments Two scenarios New programTraditional program Post test average8580 Pre-test6560 New programTraditional program Post test average8580 Pre-test60

Quasi-experiments An even better scenario New programTraditional program Post test average8580 Pre-test6560 % girls50%52% Parent education (HS graduates) 70%74% ELL20%18%

Adjustment strategies in quasi-experiments New programTraditional program Post test average8580 Pre-test6560 % girls50%52% Parent education (HS graduates) 70%74% ELL20%18% Regression adjustments Multiple regression to adjust for other variables Propensity score matching See where the two groups differ and where they overlap Then find good matches for those in the new program (Private schools in Mexico example) Regression adjustments Multiple regression to adjust for other variables Propensity score matching See where the two groups differ and where they overlap Then find good matches for those in the new program (Private schools in Mexico example) What to do when there is non-random assignment?