Day 6: Non-Experimental & Experimental Design

Slides:



Advertisements
Similar presentations
Educational Research: Causal-Comparative Studies
Advertisements

Experimental and Quasi-Experimental Research
Defining Characteristics
Inadequate Designs and Design Criteria
GROUP-LEVEL DESIGNS Chapter 9.
Experimental Research Designs
Correlation AND EXPERIMENTAL DESIGN
Research Design and Validity Threats
RESEARCH DESIGN : 1. Kinds of support for making CAUSAL interpretations of observed relationships quality of theory research design used measurement procedures.
Group Comparison Research
© 2005 The McGraw-Hill Companies, Inc., All Rights Reserved. Chapter 4 Choosing a Research Design.
9 Quantitative Research Designs.
Chapter 9 Experimental Research Gay, Mills, and Airasian
Experimental Research
Experimental Research
Nasih Jaber Ali Scientific and disciplined inquiry is an orderly process, involving: problem Recognition and identification of a topic to.
EVALUATING YOUR RESEARCH DESIGN EDRS 5305 EDUCATIONAL RESEARCH & STATISTICS.
Chapter 8 Experimental Research
Experimental Design The Gold Standard?.
Research Design Methodology Part 1. Objectives  Qualitative  Quantitative  Experimental designs  Experimental  Quasi-experimental  Non-experimental.
Experimental and Quasi-Experimental Designs
Applying Science Towards Understanding Behavior in Organizations Chapters 2 & 3.
McGraw-Hill/Irwin Copyright © 2011 by The McGraw-Hill Companies, Inc. All rights reserved. Choosing a Research Design.
I want to test a wound treatment or educational program in my clinical setting with patient groups that are convenient or that already exist, How do I.
Chapter 3 The Research Design. Research Design A research design is a plan of action for executing a research project, specifying The theory to be tested.
Quantitative Research Designs
Understanding Statistics
Learning Objectives Copyright © 2004 John Wiley & Sons,Inc Primary Data Collection: Experimentation CHAPTER Seven.
McMillan Educational Research: Fundamentals for the Consumer, 6e © 2012 Pearson Education, Inc. All rights reserved. Educational Research: Fundamentals.
Evaluating a Research Report
Chapter Seven Causal Research Design: Experimentation.
Chapter 8 Causal-Comparative Research Gay, Mills, and Airasian
Copyright ©2008 by Pearson Education, Inc. Pearson Prentice Hall Upper Saddle River, NJ Foundations of Nursing Research, 5e By Rose Marie Nieswiadomy.
Techniques of research control: -Extraneous variables (confounding) are: The variables which could have an unwanted effect on the dependent variable under.
Chapter 9 Causal Research Designs and Test Markets
INTRO TO EXPERIMENTAL RESEARCH, continued Lawrence R. Gordon Psychology Research Methods I.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Group Quantitative Designs First, let us consider how one chooses a design. There is no easy formula for choice of design. The choice of a design should.
Correlational Research Chapter Fifteen Bring Schraw et al.
1 Experimental Research Cause + Effect Manipulation Control.
1 Copyright © 2011 by Saunders, an imprint of Elsevier Inc. Chapter 8 Clarifying Quantitative Research Designs.
Quantitative Research SPED 500 Dr. Sandra Beyda Designs that maximize objectivity by using numbers, statistics, structure, and experimenter control Modes.
Experimental Research
Experimental Research & Understanding Statistics.
Research Design. “The best way to escape a problem is to solve it.” -- Brendan Francis.
Chapter 10 Experimental Research Gay, Mills, and Airasian 10th Edition
Chapter 10 Finding Relationships Among Variables: Non-Experimental Research.
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
Research Design. Time of Data Collection Longitudinal Longitudinal –Panel study –Trend study –Cohort study Cross-sectional Cross-sectional.
SOCW 671: #6 Research Designs Review for 1 st Quiz.
Non-Experimental Design Where are the beakers??. What kind of research is considered the “gold standard” by the Institute of Education Sciences? A.Descriptive.
EXPERIMENTAL DESIGNS. Categories Lab experiments –Experiments done in artificial or contrived environment Field experiments –Experiments done in natural.
 Describe the defining characteristics of quantitative research studies.  List and describe the basic steps in conducting quantitative research studies.
Correlational Research Relationships are everywhere, but are they strong ones…?
Non-Experimental Design Where are the beakers??. Cautions A relationship between two variables does NOT mean one causes the other (Think about the correlation.
Criminal Justice and Criminology Research Methods, Second Edition Kraska / Neuman © 2012 by Pearson Higher Education, Inc Upper Saddle River, New Jersey.
School of Public Administration & Policy Dr. Kaifeng Yang 研究设计 : 实验研究的基本问题.
Research designs Research designs Quantitative Research Designs.
Educational Research Experimental Research Chapter 9 (8 th Edition) Chapter 13 (7 th Edition) Gay and Airasian.
William M. Trochim James P. Donnelly Kanika Arora 8 Introduction to Design.
Chapter 9 Scrutinizing Quantitative Research Design.
EXPERIMENTAL RESEARCH
Ron Sterr Kim Sims Heather Cruz aka “The Carpool”
2 independent Groups Graziano & Raulin (1997).
Introduction to Design
Quantitative Research
The Nonexperimental and Quasi-Experimental Strategies
Group Experimental Design
Reminder for next week CUELT Conference.
Presentation transcript:

Day 6: Non-Experimental & Experimental Design Where are the beakers??

What kind of research is considered the “gold standard” by the Institute of Education Sciences? Descriptive Causal-Comparative Correlational Experimental Why?

Why does most educational research use non-experimental designs?

What is the purpose of non-experimental designs?

Causal-Comparative Example Green & Jaquess (1987) Interested in the effect of high school students’ part-time employment on their academic achievement. Sample: 477 high school juniors who were unemployed or employed > 10 hours/wk.

Causal-Comparative Design A study in which the researcher attempts to determine the cause, or reason, for pre-existing differences in groups of individuals At least two different groups are compared on a dependent variable or measure of performance (called the “effect”) because the independent variable (called the “cause”) has already occurred or cannot be manipulated

Causal-Comparative Design Ex-post facto Causes studied after they have exerted their effect on another variable.

Causal-Comparative Design Drawbacks Difficult to establish causality based on collected data. Unmeasured variables (confounding variables) are always a source of potential alternative causal explanations.

Some Thought Questions…

Correlational Design Determines whether and to what degree a relationship exists between two or more quantifiable variables.

Example of Correlation

Correlational Design The degree of the relationship is expressed as a coefficient of correlation Examples Relationship between math achievement and math attitude Relationship between degree of a school’s racial diversity and student use of stereotypical language Your topics?

Correlation coefficient… -1.00 0.00 +1.00 strong positive strong negative no relationship

Advantages of Correlational Design Analysis of relationships among a large number of variables in a single study Information about the degree of the relationship between the variables being studied

Cautions A relationship between two variables does not mean one causes the other (Think about the reading achievement and body weight correlations) Possibility of low reliability of the instruments makes it difficult to identify relationships

Cautions Lack of variability in scores (e.g. everyone scoring very, very low; everyone scoring very, very high; etc.) makes it difficult to identify relationships Large sample sizes and/or using many variables can identify significant relationships for statistical reasons and not because the relationships really exist (Avoid shotgun approach)

Cautions Need to identify your sample to know what is actually being compared. If using predictor variables, time interval between collecting the predictor and criterion variable data is important.

Correlational Designs Guidelines for interpreting the size of correlation coefficients Much larger correlations are needed for predictions with individuals than with groups Crude group predictions can be made with correlations as low as .40 to .60 Predictions for individuals require correlations above .75

Correlational Designs Guidelines for interpreting the size of correlation coefficients Exploratory studies Correlations of .25 to .40 indicate the need for further research Much higher correlations are needed to confirm or test hypotheses

Correlational Designs Criteria for evaluating correlational studies Causation should not be inferred from correlational studies Practical significance should not be confused with statistical significance The size of the correlation should be sufficient for the use of the results (individuals vs groups)

Think… If you were going to take your action research topic, and create a causal-comparative study, what would it look like? --OR-- If you were going to take your action research project, and create a correlational study, what would it look like?

Experimental Design The Gold Standard?

To Review Why is most educational research comprised of non-experimental research designs?

To Review What is the purpose of non-experimental research?

To Review How does the independent variable function in non-experimental research?

To Review Can non-experimental research claim causality?

An example Read the example given in class and in pairs respond to the questions

Experimental Research Purpose To make causal inferences about the relationship between the independent and dependent variables Characteristics Direct manipulation of the independent variable Control of extraneous variables

Experimental Designs Examples Single Group Post-test Single Group Pre-test Post-test Non-Equivalent Groups Post-test Quasi-Experimental Design Randomized Post-test only Randomized Pre-test Post-test Factorial Examples

Experimental Validity Internal validity The extent to which the independent variable, and not other extraneous variables , produced the observed effect on the dependent variable External validity The extent to which the results are generalizable

Internal Validity Threats that reduce the level of confidence in any causal conclusions Key Question: Is this a plausible threat to the internal validity of the study?

Threats to Internal Validity History Extraneous events have an effect on the subjects’ performance on the dependent variable Ex - The crash of the stock market, 9-11, the invasion of Iraq, etc. Selection Groups that are initially not equal due to differences in the subjects in those groups Ex - Positive and negative attitudes, high and low achievers, etc.

Threats to Internal Validity Maturation Changes experienced within the subject over time Pretesting The effect of having taken a pretest Instrumentation Poor technical quality (i.e. validity, reliability) or changes in instrumentation

Threats to Internal Validity Subject attrition Differential loss of subjects from groups Statistical regression The natural movement of extreme scores toward the mean Diffusion of treatment The treatment is given to the control group Experimenter effects Different characteristics or expectations of those implementing the treatments across groups

Threats to Internal Validity Subject effects The effects of being aware that one is involved in a study Four types Hawthorne effect John Henry effect Resentful demoralization Novelty effect

Internal Validity Key Point: Ultimately, validity is a matter of judgment. Ask if it is reasonable that possible threats are likely to affect the results.

External Validity The extent to which results can be generalized from a sample to a particular population. Question – Why would really good internal validity often result in poor external validity?

External Validity Factors affecting external validity Subjects Representativeness of the sample in comparison to the population Personal characteristics of the subjects Situations - characteristics of the setting Specific environment Special situation Particular school

External Validity Importance of explanation of sampling procedures

Experimental Designs Examples Single Group Post-test Single Group Pre-test Post-test – Libby, Deb Non-Equivalent Groups Post-test – Mary, Cheryl Quasi-Experimental Design – Pete, Laura Randomized Post-test only – Amanda, Nicole, Tam Randomized Pre-test Post-test – Karen, Jen, Justin Examples

Your Task Based on the topic of your proposal, design an experimental study using the design you were assigned. Write a research question and hypothesis. Sketch out the methods. Identify strengths and weaknesses of each design.

Experimental Designs Notation R indicates random selection or random assignment O indicates an observation Test Observation score Scale score X indicates a treatment A, B, C, ... indicates a group

Pre-Experimental Designs No pre-experimental design controls internal validity threats well Single group pretest only A X O Internal validity threats History, maturation, attrition, experimenter effects, subject effects, and instrumentation are viable threats Useful only when the research is sure of the status of the knowledge, skill, or attitude being changed and there are no extraneous variables affecting the results

Pre-Experimental Designs Single group pretest post-test A O X O Internal validity threats Maturation and pretesting are threats History and instrumentation are potential threats Useful when subject effects will not influence the results, history effects can be minimized, and multiple pretests and post-tests are used

Pre-Experimental Designs Non-equivalent groups post-test only A X O B O Internal validity threats Definite Threat: Selection Potential Threats: History, maturation, and instrumentation Useful when groups are comparable and subjects can be assumed to be about the same at the beginning of the study

Quasi-Experimental Designs Types Non-equivalent pretest/post-test, experimental control groups A O X O B O O Non-equivalent pretest/post-test, multiple treatment groups A O X1 O B O X2 O Useful when subjects are in pre-existing groups (e.g. classes, schools, teams, etc.)

Quasi-Experimental Designs Threats to internal validity Selection is the major concern Likely to control for most other threats, provided the groups are not significantly different from one another See Table 9.2 for specific threats related to each design

True Experimental Designs Important terminology Random assignment Subjects placed into groups by random Ensures equivalency of the groups Random selection of subjects Subjects chosen from population by random Ensures generalizability to the population from which the subjects were selected (i.e. external validity)

True Experimental Designs Types Randomized post-test only experimental control groups R A X O R B O Randomized post-test only multiple treatment groups R A X1 O R B X2 O

True Experimental Designs Types (continued) Randomized pretest/post-test multiple treatment groups R A O X1 O R B O X2 O Randomized pretest/post-test experimental control groups R A O X O R B O O

True Experimental Designs Threats to internal validity Controls for selection, maturation, and statistical regression Likely to control for most other threats See Table 9.2 for specific threats related to each design

Evaluating Experimental Designs Criteria for evaluating experimental research The primary purpose is to test causal hypotheses There should be direct manipulation of the independent variable There should be clear identification of the specific research design

Evaluating Experimental Designs Criteria for evaluating experimental research The design should provide maximum control of extraneous variables Treatments are substantively different from one another The number of subjects is dependent on or equal to the number of treatment replications