1. Experiments Types of Experiments Laboratory Experiments Experiments of short term duration and usually conducted in a lab under very controlled conditions. Commonly used designs Classical Experimental Design Solomon’s Four Group Design One-shot pre-test/post-test One shot post-test Field Trials or Field Experiments Experimental methods used in the “field” or in the “real world” Can use any of the designs listed above

2. Laboratory Experiments Purpose For a Relationship to be “causal” then The independent variable (X) has to come before the dependent variable (Y) in time (X before Y) X and Y have to be related There is no “Z” variable that explains the relationship between X and Y Laboratory Experiments help us establish a causal relationship because we can Control the time order of X and Y Control for any “Z” variables (other influences)

3. Laboratory Experiments Example Question – Is there a relationship between a new fertilizer and crop yield? For a Relationship to be “causal” then Introduction of the fertilizer has to come before increase in crop yield. (X must come before Y) Fields where the fertilizer is used must yield a greater crop yield than those fields using no fertilizer. (X is related to Y) Relationship between fertilizer and crop yield is not explained by more rain in some fields than others. (No “Z” variable) Plant corn in a controlled laboratory environment. Apply fertilizer to some sections and not to others. Put equal amounts of rainwater on all sections. Measure corn yield in different sections and compare across sections.

4. Experiments and Evaluation Research Steps Step 1 Identify independent and dependent variables. In evaluation research, the independent variable would be the intervention (i.e., the workshop, program, etc.) In evaluation research, the dependent variable would measure the objective/impact you want to achieve. Step 2 Develop indicators to measure these variables Independent variable – Construct questions to measure participation in the program (i.e., ask participants if they participated in program and/or how many hours they participated). Dependent variable – Construct questions to ask participants about their behavior/attitudes before/after participation.

5. Experiments and Evaluation Research Steps Step 3 If possible, recruit a control group (similar group that does not participate in program) Step 4 Administer a survey to experimental and control groups BEFORE the Experimental group participates in the program. Step 5 Implement the program.

6. Experiments and Evaluation Research Steps Step 6 Administer a survey to experimental and control groups AFTER the experimental group participates in the program. Step 7 Input data and using statistical analysis compare across groups Step 8 Alternate Plan If plan works

7. Classical Design Experimental Group Pre-test Measure the dependent variable Introduce intervention Post-test Measure the dependent variable again Use a Control Group Pre-test Measure dependent variable Post-test Measure dependent variable again Compare post-test for control group and experimental group

8. Classical Experimental Design Experimental Procedure Pre-test Intervention Post-test Experimental GroupYXY Control GroupYY

9. Classical Experimental Design and Evaluation Research Step 1 - Divide participants into experimental and control groups. Step 2 - State objectives/impacts and develop intervention (i.e., program, workshop, etc.) to meet those objectives/impacts. (Please Note – Objective should have something to do with increasing/decreasing the level of your Y variable.) Step 3 - Measure Y for both groups BEFORE intervention Step 4 - Expose experimental group to intervention Step 5 - Measure Y for both groups after experimental group has been exposed to intervention Step 6 - Compare Y for both the experimental and the control group – before and after intervention. Then compare Y before and after intervention for the experimental group.

10. Classical Experimental Design CAUTION A MAJOR limitation of the Classical Experimental Design is that you have only established the impact of X on Y in a laboratory setting. You don’t know if X will have the same impact on Y in the “real world.”

11. ANOTHER Modification of Experimental Design Solomon Four Group Design has an important advantage over the Classical Experimental Design. Advantages It can help you determine if measuring the pre-test can actually have an impact on the post-test. For instance - IF you take the SAT a second time, will already having taken the test impact your score the second time? HOWEVER, there are important disadvantages, namely Takes more Time Takes more Money

12. A Quick Visual Comparison of the Two Designs

13. Experimental designs have HIGH internal validity….in other words, they tend to yield accurate results under the specified conditions. HOWEVER, to ensure high internal validity, you should remember the following:  History  Anything that happens between the pre and post-test can potentially influence the post-test (i.e., the room gets warm!)  Maturation  Participants get tired or may actually get older and even die between the time of the pre and post-tests.  Testing  Taking the pre-test can actually influence how participants do on the post-test – even in the laboratory.  Instrumentation  In a lab, you must ask accurate questions if you are going to get accurate answers.

14. Experimental designs have HIGH internal validity….in other words, they tend to yield accurate results under the specified conditions. HOWEVER, to ensure high internal validity, you should remember the following:  Causal Time Order  You have to control the introduction of X and Y in the right order, they won’t just “happen” in the right order.  Diffusion or Limitation of Treatment  If the experimental and control groups interact during this time, they can influence the results of the other.  Compensation  It is important that you do not “feel sorry” for the control group, because they don’t receive the intervention, and thus inadvertently treat them better/different than the experimental group.  Compensatory Rivalry  If the control group discovers they are NOT the experimental group and thus feel deprived, they may “try harder” to do better than the experimental group.

15. Experimental designs have HIGH internal validity….in other words, they tend to yield accurate results under the specified conditions. HOWEVER, to ensure high internal validity, you should remember the following:  Statistical Regression to the Means  Extreme answers on the pre-test will tend to yield less extreme answers on the post-test regardless of what happens between. For instance, individuals who score a perfect score (1600) on the SAT are likely to score slightly less if they take it again – even if they study in the meantime.  Demoralization  The control group may become demoralized when they realize they aren’t receiving the “treatment,” and give up.  Experimental Bias  The experimentalist may want to obtain certain results and consciously or unconsciously act in ways that will increase the probability of receiving the desired results.

16. Field Trials OR Field experiments are Similar to Laboratory Experiments EXCEPT:  They are conducted in the “real” world – not in laboratories.  They are generally used when, for practical reasons, laboratory experiments cannot be used. For instance:  When the program or intervention lasts longer than a few hours and/or cannot occur within a laboratory.  When participants cannot/won’t come into a laboratory.  They generally take considerably longer than laboratory experiments, and in fact, it may take months or years to conduct laboratory experiments. Field Trials or Experiments

17. Example of a FAMOUS Field Experiment  The Perry Preschool Field Experiment  In the 1960s, low income children in the US went to Head Start.  Studies were done to evaluate the program. First researchers compared the IQ scores of the Head Start group to a control group. See the following slide for the results.  Parents and researchers felt IQ was not a good and valid indicators of the impact of Head Start. A consortium of researchers selected small subgroup of Head Start students and compared them to a control group over 19 years. See slide 19 for results.  Once they evaluated the program, they calculated the savings, to the government, of Head Start. The results are on slide 20. Field Trials or Experiments

18. IQ The Head Start Program - A Famous Field Experiment

19. Schooling Success High School Graduation or GED College or Vocation Functional Competence Classified as Mentally Retarded Years in Special Education School Responsibility Detained or Arrested Teen Pregnancy Socioeconomic Success Employed Received Welfare Slide 19

21. Power Point Created by: Dr. Carol Albrecht Carol.albrecht@ usu.edu 979-777-2421 For More Information Contact: Assessment Specialist USU Extension