Research Methods for Counselors COUN 597 University of Saint Joseph Class # 5 Copyright © 2015 by R. Halstead. All rights reserved.

Class Objectives  Trochim  Chapter 8 - Experimental Design  Chapter 9 - Quasi-Experimental Design  Salkind Chapter 7 - Hypotheses  Salking Chapter 18 - Data Collection

 The Classic Experiment  What is a true experimental design?  The hallmarks of a true experimental design are:  1. Independent and Dependent Variables  2. Pretesting and Posttesting  3. Experiment and Control Groups  Let’s take a look.

The Classic Experiment Treatment Group A Measure Depression (D.V.) Prior to Intervention Treatment Group A Measure Depression (D.V.) After Intervention Control Group A Measure Depression (D.V.) No Treatment Control Control Group A Measure Depression (D.V.) No Treatment Control Treatment Over Time No Treatment Over Time Treatment is our I.V.

 What are we looking for?  In a simple experimental design we looking for one of two elements  Null Outcome or Case - when the treatment is shown to have no effect (means do not differ).  Main Effects – when the treatment is shown to have an effect (means differ).  Of course we need to be certain that the outcome was not confounded by any outside variables.

Threats: Factors to Consider when Conducting an Experiment  The Hawthorne Effect - The effect that participation in a study has on an individual or group as it relates to the D.V.  Degree of Blindness on the part of participants and researchers.  Methods for selecting participants  Probability Sampling

Threats: Factors to Consider when Conducting an Experiment  Randomization of assigning participants to experimental and control groups.  Matching assignments of participants - establishing a quota matrix to insure the gross observable differences in the sample are distributed across the experimental and control groups.

Multiple-Group Threats to Internal Validity When ever one deals with additional groups in a design, one increases the degree of threat to the study’s resulting findings understood as “truth.”

The Central Issue  When you move from single to multiple group research the big concern is whether the groups are comparable.  Usually this has to do with how you assign units (for example, persons) to the groups (or select them into groups).  We call this issue selection or selection bias. Trochim, 2001

The Multiple Group Case AdministerprogramMeasureoutcomesMeasurebaseline Alternativeexplanations AlternativeexplanationsXOOOO Do not administer program MeasureoutcomesMeasurebaseline Trochim, 2001

Example  Compensatory education in math for 1st graders  Pre-post program-comparison group design  Measures (O) are standardized achievement tests (at start of grade 1 and start of grade 2; forms A & B)

Selection Threats  Any factor other than the program that leads to posttest differences between groups.  For example, because of group differences, kids in one group watch Sesame Street more frequently and pick up more math concepts.XOOOO Trochim, 2001

Selection-History Threat  Any other event that occurs between pretest and posttest that the groups experience differently.  For example, kids in one group pick up more math concepts because they watch more Sesame Street.XOOOO Trochim, 2001

Selection-Maturation Threat  Differential rates of normal growth between pretest and posttest for the groups.  They are learning at different rates, even without program.XOOOO Trochim, 2001

Selection-Testing Threat  Differential effect on the posttest resulting from taking the pretest.  The test may have “primed” the kids differently in each group or they may have learned differentially from the test, not the program.XOOOO Trochim, 2001

Selection-Instrumentation Threat  Any differential change in the test used for each group from pretest and posttest  For example, change due to different forms of test being given to each group and not due to programXOOOO Trochim, 2001

Selection-Mortality Threat  Differential nonrandom dropout between pretest and posttest.  For example, kids drop out of the study at different rates for each group.XOOOO Trochim, 2001

Selection-Regression Threat  Different rates of regression to the mean because groups differ in score extremities.  For example, program kids are disproportionately lower math scorers and consequently have greater regression to the mean.XOOOO Trochim, 2001

Variations on Experimental Design  Quasi-Experimental Designs (Campbell & Stanley, 1963).  One-Shot Case Study - Measure a D.V. after the administration of some experimental stimulus  One group pretest/posttest design  Static-group comparison - expose one group to some stimulus but not the other group and then measure to see if there are differences.

Types of Designs Random assignment? Control group or multiple measures? Yes No YesNo Randomized or true experiment? Quasi-experimentNonexperiment Trochim, 2001

Factorial Designs  Sometimes a counselor might want to look at more than two groups at time.  In such situations the counselor will employ a factorial design.

A Simple Example Group 2 average Group 4 average Group 1 average Group 3 average Time in Instruction Setting 1 hour/week4 hours/week In-class Pull-out Usually, averages are in the cells. Trochim, 2001

A Simple Example Group 2 average Group 4 average Group 1 average Group 3 average Time in Instruction Setting 1 hour/week4 hours/week In-class Pull-out Usually, averages are in the cells. Trochim, 2001 Levels/ Subdivisions Factors

What are we looking for when we employ a factorial design?  With a factorial design we are looking at several elements  Null Case – means do not differ  Main Effects – means differ across levels of a factor  Interaction Effects – means differ across factors and levels  See Trochim text.

Internal Validity  The threat to internal validity occurs whenever there is something outside to affect the change noted in the D.V.  Examples:  History, Maturation, Testing, Instrumentation  Statistical Regression, Selection Bias  Experimental mortality, Imitation of Treatment  Compensation, Compensatory Rivalry  Demoralization

External Validity  One must always ask if the findings that result from the experiment are generalizable to situations that exist outside of the conditions that were set during the experiment.

The Nature of Good Design  Grounded in Theory – Reflects some body of earlier literature.  Situational in Nature – Reflects the settings of the investigation  Feasibility – You have the resources to carry out the work  Redundant – Allows for further and follow-up study  Efficient – Parsimonious systems are employed in elegant research design

Hypotheses  A hypothesis is a statement that, given what is known thus far, it is thought to be true. It is often referred to as an educated guess.  Much of designing research projects is about figuring out ways of testing that educated guess (the hypothesis) to see if it holds under certain conditions set up in the experiment.

The Null Hypothesis  The Null Hypothesis states that there are no differences between two groups relative to some variable of interest  One uses the Null Hypothesis a priori (before the fact) because if you are looking at something before the fact, there is no real reason to believe that there would be differences.  This is a starting point - you must assume that any difference is a result of chance.

The Research Hypothesis  The Research Hypothesis states that there is a difference between Group A and Group B.  There are two forms of Research Hypotheses  Nondirectional - Predicts a difference between two groups but does not predicts in which direction (e.g. higher or lower) that difference takes.  Directional - Predicts a difference between two groups and predict a direction (e.g. Group A will be greater than Group B) that difference takes.

Null and Research Hypotheses  Null Hypothesis H o :  1 =  2  No Difference in populations means  Nondirectional Hypothesis H 1 : X 1 = X 2  Difference in sample means  Directional HypothesisH 1 : X 1 > X 2 H 1 : X 1 < X 2  Directional difference in sample means

One Tailed and Two Tailed Tests: A Matter of Direction  How come we starting to talk about tails? What is the connection?  Well, remember that normal curve I have drawn for you 903 times so far (and this is just the fifth class)?  You may recall that the normal curve the and standard deviation allows us, given the laws of probability to know or predict what percent of cases we will find as we move out from the mean toward the (count them) two tails!

One Tailed and Two Tailed Tests: A Matter of Direction  Tails come into play a bit further down the road when we begin testing hypotheses to see if there are significant differences between two means.  For now, however, just know that when you are working with a Directional Hypothesis you will be using a one tailed test and when you are working with a Nondirectional Hypothesis you will using a two tailed test.

Good Hypotheses  1. Statements not questions  2. Posit a relationship between variables  3. Are literature based  4. Brief and to the point  5. Must be testable

Data Collection  Type of data for the study – What has been done in the past (read relevant research)  From where or from whom will this data come?  Pilot the study to test procedures and establish early indications of results  Protect your data – two types of people: those that have lost data and those that will lose data  Stay close to your data – be a do it yourself researcher  No one plans to fail but many fail to plan – Data collection rarely goes smoothly but it is best to have a plan and a plan “B” when best plans don’t work out

Data Collection  Be persistent in the data collection process. Although collecting your data is important to you, it is usually just one more unimportant thing for your chosen participants to do. Expect that it will be a chore to get all your data back.  Hold onto data after you have analyzed it. Other researchers may be interested in using it [of course the ACA Code of Ethics (2014) would require you to be sure that the researcher is competent].