Experiments II: Validity and Design Considerations

Slides:



Advertisements
Similar presentations
Andrea M. Landis, PhD, RN UW LEAH
Advertisements

Agenda Group Hypotheses Validity of Inferences from Research Inferences and Errors Types of Validity Threats to Validity.
Chapter 8: Construct and External Validity in Experimental Research
Ch 8: Experimental Design Ch 9: Conducting Experiments
Experimental Design I. Definition of Experimental Design
Copyright © Allyn & Bacon (2007) Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8 This multimedia.
 Lab #1  Intro to Experiments and Experimental Design 2.
Validity of Quantitative Research Conclusions. Internal Validity External Validity Issues of Cause and Effect Issues of Generalizability Validity of Quantitative.
Correlation AND EXPERIMENTAL DESIGN
Validity Validity of measurement Validity of research
MSc Applied Psychology PYM403 Research Methods Validity and Reliability in Research.
Lecture 5 Data Coding and Experimental Research Methods.
Experiments Pierre-Auguste Renoir: Barges on the Seine, 1869.
Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.
EVALUATING YOUR RESEARCH DESIGN EDRS 5305 EDUCATIONAL RESEARCH & STATISTICS.
Experimental Research Take some action and observe its effects Take some action and observe its effects Extension of natural science to social science.
Copyright © 2008 by Pearson Education, Inc. Upper Saddle River, New Jersey All rights reserved. John W. Creswell Educational Research: Planning,
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 11 Experimental Designs.
Experimental Research Validity and Confounds. What is it? Systematic inquiry that is characterized by: Systematic inquiry that is characterized by: An.
 Internal Validity  Construct Validity  External Validity * In the context of a research study, i.e., not measurement validity.
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 11 Enhancing Rigor in Quantitative Research.
Experimental Designs. Experiments are conducted to identify how independent variables influence some change in a dependent variable.
Experimental Research
Reading and Evaluating Research Method. Essential question to ask about the Method: “Is the operationalization of the hypothesis valid? Sections: Section.
Chapter 6 Research Validity. Research Validity: Truthfulness of inferences made from a research study.
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
Experimental & Quasi-Experimental Designs Dr. Guerette.
SOCW 671: #6 Research Designs Review for 1 st Quiz.
 The basic components of experiments are: 1) taking action 2) observing the consequence of that action  Experimental model is most closely linked to.
EXPERIMENTS Lecture 5. Administrative STATA Course Mailing List Info:   No subject  In body.
CHAPTER 8 EXPERIMENTS.
Experiments.  Labs (update and questions)  STATA Introduction  Intro to Experiments and Experimental Design 2.
Designing psychological investigations
Research designs Research designs Quantitative Research Designs.
Issues in Evaluating Educational Research
Experimental Research
Experiments Why would a double-blind experiment be used?
Chapter 8 Experimental Design The nature of an experimental design
Hypothesis Testing, Validity, and Threats to Validity
New Media Research Methods
Ron Sterr Kim Sims Heather Cruz aka “The Carpool”
Statistical Analyses & Threats to Validity
Research Methods 3. Experimental Research.
Experiments: Part 3.
Chapter Eight: Quantitative Methods
2 independent Groups Graziano & Raulin (1997).
Introduction to Design
Chapter 6 Research Validity.
Experiments and Quasi-Experiments
Chapter 9 Experimental Research: An Overview
Social Research Methods Experimental Research
How do we misinterpret? Common threats to validity
Experiments and Quasi-Experiments
Experiments: Validity, Reliability and Other Design Considerations
Introduction to Experimental Design
External Validity.
Experiments: Part 2.
Experiments: Part 3.
Experiments: Part 3.
Experimental Research
A Multi-disciplinary Perspective on Decision-making and Creativity:
Group Experimental Design
Study on Method of Mass Communication Research 传播研究方法 (7&8) Dr
Experimental Design I. Definition of Experimental Design
Chapter 11 EDPR 7521 Dr. Kakali Bhattacharya
Experimental Research
Types of Designs: R: Random Assignment of subjects to groups
Research Methods & Statistics
Reminder for next week CUELT Conference.
Misc Internal Validity Scenarios External Validity Construct Validity
Presentation transcript:

Experiments II: Validity and Design Considerations Lecture 6 Experiments II: Validity and Design Considerations

Example: Trust-Building Study Question: Do increased risk-taking behaviors over time increase interpersonal trust?

Trust-Building Study Independent Variable Experiment Condition (2 conditions): Fixed partner on every trial, cannot control amount to entrust to partner Fixed partner on every trial, can control amount to entrust to partner Assignment Random assignment of participants to one of the 2 conditions. Same experiment conducted in Japan and US, and comparisons made between the two studies. Dependent Variable Cooperation rate (i.e., whether they returned the coins to the partner or not)

Measurement and Design Validity Measurement Concerns Construct Validity Design Concerns Internal Validity External Validity Ecological Validity Construct validity: measuring our intended concept? Internal validity: External: Ecological:

Construct Validity How do we know that our independent variable is reflecting the intended causal construct and nothing else? “Face” validity deals with subjective judgement of appropriate operationalization “Content” validity is a more direct check against relevant content domain for the given construct. Examples from operationalization exercise?

Internal Validity Internal Validity deals with questions about whether changes in the dependent variable were caused by the treatment.

? ? Cause Effect ? ?

Threats to Internal Validity History additional I.V. that occurs between pre-test and post-test Maturation Subjects simply get older and change during experiment Testing Subjects “get used” to being tested Regression to the Mean Issue with studies of extremes on some variable

Contamination and Internal Validity Demand Characteristics Anything in the experiment that could guide subjects to expected outcome Experimenter Expectancy Researcher behavior that guides subjects to expected outcome (self-fulfilling prophecy)

General Demand Characteristics Evaluation Apprehension Solutions Double-blind experiments Experiments in natural setting (i.e., subjects do not know they are in an experiment) Cover stories Hidden measurements Subjects know that they are being evaluated and this changes their behavior

Reducing the role of the experimenter: solving expectancy effects Naïve experimenter Those conducting study are not aware of theory or hypotheses in the experiment Blind Researcher is unaware of the experiment condition that he/she is administering Standardization Experimenter follows a script, and only the script “Canned” Experimenter Audio/Video/Print material gives instructions

And More! Selection Bias Mortality Diffusion, Sharing of Treatments Issue with non-random selection of subjects Mortality Departure of subjects in the experiment Diffusion, Sharing of Treatments Control group unexpectedly obtains treatment Other ‘social’ threats? Compensatory rivalry, resentful demoralization, etc.

Three threats to external validity (generalizability) in experiments External Validity– How far does the given experiment generalize to similar groups, individuals, etc? Setting Population History Setting:Physical and social context of the experiment Population:Is there something specific about the sample that interacts with the treatment? Histor: Is there something about the time that interacts with the treatment?

Ecological Validity Approximation of ‘real-life’ situations

The Validity Tradeoff: Truth and Myth Internal Validity External & Ecological Validity Balance is important between the types of validity, but internal validity is usually (if not always) the more important factor.

Pro’s and Con’s of Experiments Gives researcher tight control over independent factors Allows researcher to test key relationships with as few confounding factors as possible Allows for direct causal testing Con’s Usually a smaller N than surveys Sometimes give up large amounts of external validity in favor of construct validity and direct causal analysis Require a large amount of planning, training, and time– sometimes to test relationship between only 2 factors!

Additional considerations before using experiments Cost and Effort Is the effort worth it to test the concepts you are interested in? Manipulation and Control Will you actually be able to manipulate the key concept(s)? Importance of Generalizability Are you testing theory, or trying to establish a real-world test?