1. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Chapter 10 The Simple Experiment

2. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina JolleyOverview l Basic Logic and Terminology l Errors in determining whether results are statistically significant l Statistics and the design of the simple experiment l Nonstatistical considerations in the design of the simple experiment l Analyzing data from the simple experiment

3. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Basic Logic and Terminology l Hypotheses l Manipulating the independent variable (IV) and measuring the dependent variable (DV) l Experimental and control groups l The importance of independence l The importance of assignment l Statistically significant vs. null results

4. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Hypotheses l Experimental hypothesis: The treatment has an effect (IV-->DV) l Null hypothesis: The treatment does not have an effect

5. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Manipulating the Independent Variable l Experimental and control groups: Similar, but treated differently

6. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Independence: Why Control and Experimental “Groups” Shouldn’t Really Be Groups –Why you should not choose to use two pre-existing groups –Why you shouldn’t let your “groups” become groups

7. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley The Value of Assignment (Manipulating the Treatment) –Random assignment makes the treatment the only systematic difference between groups –Without random assignment you do not have an experiment

8. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Collecting the Dependent Variable

9. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley The Statistical Significance Decision: l The decision is whether to declare that a difference is not due to chance. l Statistically significant results* l Nonsignificant (null) results*

10. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Statistically Significant Results: Declaring that the Treatment Has an Effect l Statistically significant effects are not necessarily large l Statistically significant results may not be in the direction you expect

11. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Null Results: Why We Can’t Draw Conclusions from Nonsignificant Results l Nonsignificant results are not significant l Null results do not prove the null hypothesis: “I didn’t find it” doesn’t mean it doesn’t exist

12. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Errors in Determining Whether Results Are Statistically Significant l Type 1 Errors: “Crying wolf”* l Type 2 Errors: Failing to announce the wolf*

13. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Type 1 Errors: “Crying Wolf” l Reducing the risk of a Type 1 error l Accepting the risk of a Type 1 error

14. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Type 2 Errors: Failing to Announce the Wolf The need to prevent type 2 errors: Why you want the power to find significant differences

15. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Statistics and the Design of the Simple Experiment l Power* l Other statistical issues*

16. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Power and the Design of the Simple Experiment l Reduce the effect of random error l Standardize procedures and use reliable measures l Use a homogeneous group of participants l Code data carefully l Let random error balance out l Create larger effects: Bigger effects are easier to see

17. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley How Statistical Issues Other Than Power Affect the Design of the Simple Experiment l Inability to prove the null hypothesis limits hypotheses l Requirement of independent random assignment limits what you can study and how you assign participants l Independence assumption may cause you to test participants individually

18. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Nonstatistical Considerations in the Design of the Simple Experiment l External validity versus power l Construct validity versus power l Ethics versus power

19. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Analyzing Data from the Simple Experiment: Basic Logic l Estimating what you want to know: your means are sample means l Calculating sample means: Getting your estimates l Comparing sample means: How to compare two imperfect estimates l Why we must do more than subtract the means from each other

20. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley How Random Error Affects Data from the Simple Experiment l Random error makes scores within a group differ l Random error can make group means differ

21. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley When Is a Difference Too Big to Be Due to Random Error? l Bigger differences are less likely to be due to chance alone l “Too big to be due to chance” depends on 1. How big chance is l Differences within groups tell you how big chance is 2. The extent to which chance balances out l With larger samples, random error tends to balance out

22. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Analyzing the Results of the Simple Experiment: The t Test Using the t table l Decide on significance level before doing experiment l Look in table under appropriate significance level and degrees of freedom (N - 2) l Absolute value of t must exceed tabled value

23. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Assumptions of the t Test l Two Critical Assumptions –Observations are independent –Data are interval or ratio l Two Less Critical Assumptions –Underlying distributions are normally distributed –Both groups’ distributions have the same variance

24. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Questions Raised by Results l Questions Raised by Nonsignificant Results –Questions about power, such as “Did the experiment study enough participants?” l Questions Raised by Significant Results –External validity l Do results generalize to other levels of the IV? l Do other variables moderate the IV’s effect? –Construct validity (Was control group good enough?)

25. PowerPoint presentation to accompany Research Design Explained 6th edition ; ©2007 Mark Mitchell & Janina Jolley Concluding Remarks l Simple experiment is a simple way to establish internal validity l Independent random assignment is its cornerstone l Logic behind the simple experiment can be used to create experiments that use more than two groups. Such experiments may –Have more construct validity than simple experiments –Have more external validity than simple experiments –Answer more questions than simple experiments