Experimental Research Methods in Language Learning

Slides:



Advertisements
Similar presentations
MANOVA (and DISCRIMINANT ANALYSIS) Alan Garnham, Spring 2005
Advertisements

MANOVA: Multivariate Analysis of Variance
Analysis of variance (ANOVA)-the General Linear Model (GLM)
Analysis of variance (ANOVA)-the General Linear Model (GLM)
C82MST Statistical Methods 2 - Lecture 7 1 Overview of Lecture Advantages and disadvantages of within subjects designs One-way within subjects ANOVA Two-way.
Inferential Stats for Two-Group Designs. Inferential Statistics Used to infer conclusions about the population based on data collected from sample Do.
ANCOVA Psy 420 Andrew Ainsworth. What is ANCOVA?
Independent Samples and Paired Samples t-tests PSY440 June 24, 2008.
Statistics for Managers Using Microsoft® Excel 5th Edition
Lecture 9: One Way ANOVA Between Subjects
Two Groups Too Many? Try Analysis of Variance (ANOVA)
Analysis of Variance & Multivariate Analysis of Variance
Today Concepts underlying inferential statistics
DOCTORAL SEMINAR, SPRING SEMESTER 2007 Experimental Design & Analysis Further Within Designs; Mixed Designs; Response Latencies April 3, 2007.
Chapter 14 Inferential Data Analysis
Repeated Measures ANOVA Used when the research design contains one factor on which participants are measured more than twice (dependent, or within- groups.
Two-Way Analysis of Variance STAT E-150 Statistical Methods.
Chapter 12 Inferential Statistics Gay, Mills, and Airasian
Analysis of Variance. ANOVA Probably the most popular analysis in psychology Why? Ease of implementation Allows for analysis of several groups at once.
ANCOVA Lecture 9 Andrew Ainsworth. What is ANCOVA?
LEARNING PROGRAMME Hypothesis testing Intermediate Training in Quantitative Analysis Bangkok November 2007.
Repeated Measures ANOVA
SPSS Series 1: ANOVA and Factorial ANOVA
By Hui Bian Office for Faculty Excellence 1. K-group between-subjects MANOVA with SPSS Factorial between-subjects MANOVA with SPSS How to interpret SPSS.
© 2002 Prentice-Hall, Inc.Chap 9-1 Statistics for Managers Using Microsoft Excel 3 rd Edition Chapter 9 Analysis of Variance.
Chapter 11 HYPOTHESIS TESTING USING THE ONE-WAY ANALYSIS OF VARIANCE.
Statistics for Education Research Lecture 8 Tests on Three or More Means with Repeated Measures: One-Way ANOVA with Repeated Measures Instructor: Dr. Tung-hsien.
t(ea) for Two: Test between the Means of Different Groups When you want to know if there is a ‘difference’ between the two groups in the mean Use “t-test”.
B AD 6243: Applied Univariate Statistics Repeated Measures ANOVA Professor Laku Chidambaram Price College of Business University of Oklahoma.
Experimental Research Methods in Language Learning Chapter 15 Non-parametric Versions of T-tests and ANOVAs.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
ANOVA (Analysis of Variance) by Aziza Munir
Psychology 301 Chapters & Differences Between Two Means Introduction to Analysis of Variance Multiple Comparisons.
Between-Groups ANOVA Chapter 12. >When to use an F distribution Working with more than two samples >ANOVA Used with two or more nominal independent variables.
Statistics for Managers Using Microsoft Excel, 4e © 2004 Prentice-Hall, Inc. Chap 10-1 Chapter 10 Analysis of Variance Statistics for Managers Using Microsoft.
Inference and Inferential Statistics Methods of Educational Research EDU 660.
Statistics for the Social Sciences Psychology 340 Fall 2013 Tuesday, October 15, 2013 Analysis of Variance (ANOVA)
Analysis of Variance 1 Dr. Mohammed Alahmed Ph.D. in BioStatistics (011)
Educational Research Chapter 13 Inferential Statistics Gay, Mills, and Airasian 10 th Edition.
ANOVA: Analysis of Variance.
Analysis of variance John W. Worley AudioGroup, WCL Department of Electrical and Computer Engineering University of Patras, Greece
Repeated Measures ANOVA patterns over time. Case Study.
Analysis of Variance (One Factor). ANOVA Analysis of Variance Tests whether differences exist among population means categorized by only one factor or.
Experimental Research Methods in Language Learning Chapter 10 Inferential Statistics.
1 ANALYSIS OF VARIANCE (ANOVA) Heibatollah Baghi, and Mastee Badii.
Smoking Data The investigation was based on examining the effectiveness of smoking cessation programs among heavy smokers who are also recovering alcoholics.
ANOVA P OST ANOVA TEST 541 PHL By… Asma Al-Oneazi Supervised by… Dr. Amal Fatani King Saud University Pharmacy College Pharmacology Department.
ONE-WAY BETWEEN-GROUPS ANOVA Psyc 301-SPSS Spring 2014.
ANOVAs.  Analysis of Variance (ANOVA)  Difference in two or more average scores in different groups  Simplest is one-way ANOVA (one variable as predictor);
T tests comparing two means t tests comparing two means.
Business Statistics: A First Course (3rd Edition)
Handout Nine: Repeated Measures –Design, Analysis, & Assumptions.
Handout Six: Sample Size, Effect Size, Power, and Assumptions of ANOVA EPSE 592 Experimental Designs and Analysis in Educational Research Instructor: Dr.
Experimental Research Methods in Language Learning Chapter 13 Paired-Samples and Independent- Samples T-tests.
Formula for Linear Regression y = bx + a Y variable plotted on vertical axis. X variable plotted on horizontal axis. Slope or the change in y for every.
HYPOTHESIS TESTING FOR DIFFERENCES BETWEEN MEANS AND BETWEEN PROPORTIONS.
MANOVA Lecture 12 Nuance stuff Psy 524 Andrew Ainsworth.
Independent Samples ANOVA. Outline of Today’s Discussion 1.Independent Samples ANOVA: A Conceptual Introduction 2.The Equal Variance Assumption 3.Cumulative.
Differences Among Groups
Aron, Aron, & Coups, Statistics for the Behavioral and Social Sciences: A Brief Course (3e), © 2005 Prentice Hall Chapter 10 Introduction to the Analysis.
MM570Sec02 Zrotowski Unit 8, Unit 8, Chapter 9 Chapter 9 1 ANOVA 1 ANOVA.
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
Statistics for Education Research Lecture 4 Tests on Two Means: Types and Paired-Sample T-tests Instructor: Dr. Tung-hsien He
Quantitative data analysis for social science using PASW/SPSS (Part 2) Conducted by Dr Shamala.
Multivariate vs Univariate ANOVA: Assumptions. Outline of Today’s Discussion 1.Within Subject ANOVAs in SPSS 2.Within Subject ANOVAs: Sphericity Post.
CHAPTER 15: THE NUTS AND BOLTS OF USING STATISTICS.
12 Inferential Analysis.
Analysis of Variance (ANOVA)
12 Inferential Analysis.
Chapter 10 Introduction to the Analysis of Variance
Presentation transcript:

Experimental Research Methods in Language Learning Chapter 14 Analyses of Variance (ANOVAs)

Leading Questions If there are three experimental groups and you would like to determine which group is more effective in terms of learning improvement, what would you do? What do you know about ANOVA and ANCOVA? If you know something of these statistical tests, do you think they are difficult to learn to use? Why or why not?

ANOVA Family Analysis of variance (ANOVA) allows researchers to examine mean differences in experimental research. This chapter presents three types of ANOVAs. One-way ANOVA ANCOVA Repeated-measures ANOVA

The One-way Analysis of Variance (ANOVA) A one-way (or one-factor) (independent-measures) analysis of variance (ANOVA) is a parametric test used to determine whether there is a statistical significance between scores obtained by two or more groups in an experimental study. A one-way ANOVA can perform the same analysis as the independent-samples t-test.

The One-way ANOVA A one-way ANOVA has several advantages over a t-test. Its major advantage is that it can compare more than two groups (e.g., three, four, five, and so on) in one single analysis. ANOVA can minimize the possibility of a Type I Error (i.e., rejecting the null hypothesis when it should not be rejected) more conservatively than a t-test. A one-way ANOVA considers errors arising from within-group differences when it analyses group differences.

Post hoc Analysis Since a one-way ANOVA performs comparisons among three or more groups all at once, when there is a statistical significance, we will not know which groups differ. In order to identify where a group difference exists, we need to perform a statistical test known as a post hoc analysis. Examples of post hoc tests: Bonferrori, Scheffe, and Tukey. Each of these post hoc tests functions similarly to an independent-sample t-test.

Examples of Studies Akakura 2012; Ahmadian 2012; Ahmadian & Tavakoli 2011; Morgan-Short & Bowden 2006; Sheen 2010; van Gelderen et al. 2010.

Statistical Assumptions of the One-way ANOVA The statistical assumptions for the one-way ANOVA are the same as those for the independent-samples t-tests: Type of Scale Random Sampling Normal Distribution

Example of ANOVA Results Table 14.1.1 reports on descriptive statistics (e.g., means, SD, and standard error mean) between the three groups.

Example of ANOVA Results Table 14.1.2 reports the Levene’s Test for Equality of Variance. Remember that it must not be statistically significant in order to make sure that both groups were relatively equal.

Example of ANOVA Results Table 14.1.3 shows the ANOVA findings. We examine the F, df and Sig columns in this output. Sig will tell us whether there was a statistically significant difference between the three groups. We found that such a difference did exist (F(2, 59) = 7.97, p = 0.001).

Example of ANOVA Results Table 14.1.4 presents the Bonferrori post hoc test outcome.

Example of ANOVA Results It was found that there were statistically significant differences between the experimental methods 1 and 2 (p = 0.029), and the experimental method 1 and the control group (p = 0.001). However, the experimental method 2 and the control group did not differ statistically (p = 0.55).

Example of ANOVA Results Cohen’s d effect size for these two pairs can be computed: http://www.cognitiveflexibility.org/effectsize/ It was found that a Cohen’s d for the difference between the experimental groups 1 and 2 was 0.796 (≈ 0.80), whereas a Cohen’s d for the difference between the experimental group 1 and the control group was 1.35.

Analysis of Covariance (ANCOVA) Analysis of covariance (ANCOVA) is designed to control a pre-existing difference between comparison groups before an experimental treatment. Examples of studies: Ammar (2008); Ammar & Spada (2006); Brantmeier (2005); Goo (2012); Lee & Kalyuga (2011); Lyddon (2011); Van Beuningen, De Jong, & Kuiken (2011).

Statistical Assumptions of the ANCOVA The statistical assumptions of the ANCOVA are the same as those of the one-way ANOVA. These assumptions must be met before ANCOVA is implemented.

Example of ANCOVA Results Table 14.2.1 reports on the descriptive statistics (including means, SD, and std error mean) between the two groups.

Example of ANCOVA Results Table 14.2.2 presents the Levene’s Test for Equality of Variance. This test was non-significant (p = 0.26).

Example of ANCOVA Results Table 14.2.3 presents the results of the Test of Between-subject Effect.

Example of ANCOVA Results We examine the F, df and Sig columns in this output. Sig will tell us whether there was a statistically significant difference between the two groups. According to this table, it was found that the pre-existing test-anxiety trait was still the main factor affecting the posttest scores (F(1, 48) = 14.96, p < 0.05, partial eta squared (ηp2) = 0.24). There was also a statistically significant difference in the group effect (F(1, 48) = 4.70, p < 0.05, partial eta squared (ηp2) = 0.09).

Comparison with a Univariate ANOVA Let’s compare a univariate ANOVA on the posttest without using the test-anxiety as a covariate.

Comparison with a Univariate ANOVA We find the following main effect of the experimental treatment: F(1,48) = 10.48, p < 0.05, ηp2 = 0.19. The effect size of the experimental treatment in the ANCOVA was reduced to half the size of that produced in a univariate ANOVA. ANCOVA therefore allows us to be more realistic in evaluating the effect of an experimental treatment.

Repeated-measures Analysis of Variance (ANOVA) The repeated-measures ANOVA can serve the same function as the paired-samples t-test. However, the repeated-measures ANOVA can measure changes at more than two times points. The repeated-measures ANOVA has broad applications for experimental data analysis. It is useful for a pretest-, posttest-, and delayed-posttest design.

Repeated-measures Analysis of Variance (ANOVA) The repeated-measures ANOVA takes sources of variation into account when evaluating whether there have been statistically significant changes across times points : variation associated with the average scores (i.e., mean scores) for each group each time; variation associated with the within-subject variance between Times 1, 2 and 3; and the interaction between the first and the second sources; Urdan 2005 ).

Repeated-measures Analysis of Variance (ANOVA) Examples: Baralt & Gurzynski-Weiss 2011; Benati 2005; Folse 2006; Iwashita, McNamara, & Elder 2001; Kissling 2013; Moskovsky, Alrabai, Paolini, & Ratcheva 2012; Shintani 2011; Shintani et al. 2014; Strapp, Helmick, Tonkovich, & Bleakney 2011; Takimoto 2008; Tian & Macaro 2012). The repeated-measures ANOVA is one of the most commonly used statistical techniques in experimental research in language learning.

Statistical Assumptions of the Repeated-measures ANOVA The statistical assumptions of the repeated-measures ANOVA are similar to those discussed previously in this chapter. There is one additional assumption that we need to consider. This assumption is known as the sphericity assumption. Sphericity refers to whether the variances of the differences between all possible pairs of comparison groups are equal. This assumption is assessed by using the Mauchly’ sphericity test.

Example of Repeated-Measures ANOVA Results

Example of Repeated-Measures ANOVA Results Table 14.3.2 is the Box’ Test of Equality of Covariance Matrices. This test must not be significant at 0.001 (p > 0.001). This statistic tells us that we have homogeneity of variance.

Example of Repeated-Measures ANOVA Results Table 14.3.3 presents the multivariate tests.

Example of Repeated-Measures ANOVA Results The multivariate tests determine whether there are significant group differences for a linear combination of the dependent variables (i.e., the reading tests). This table contains 4 tests (i.e., Pillai’s Trace, Wilks’ Lambda, Hotelling’s Trace, and Roy’s Largest Root). However, we only need to consider one test. Choose the Pillai’s Trace. This test must be statistically significant at 0.05.

Example of Repeated-Measures ANOVA Results For both ‘time’ and ‘time and factor’, the Pillai’s Trace was significant (p = 0.00). This statistical significance means that we can now move on to examine the univariate/between-subjects effects that follow. If the Pillai’s Trace is not significant, we stop our analysis here because the rest of the outcome will not be meaningful because there are no group differences.

Example of Repeated-Measures ANOVA Results Table 14.3.4 presents the analysis for Mauchly’s sphericity test. This test tells us whether our data violate the sphericity assumption. We need this test to be non-significant for this assumption to be met. In Table 14.3.4, we can see that this test was significant (p < 0.05). Examine an alternative assessment to the sphericity assumption. Choose the Hyunh-Feldt test, which was significant at 0.896.

Example of Repeated-Measures ANOVA Results

Example of Repeated-Measures ANOVA Results Table 14.3.5 shows the results for the test within-subject effect. Focus on Huynh-Feldt.

Example of Repeated-Measures ANOVA Results Table 14.3.6 presents the Levene’ Test of Equality of Error Variances. For all the three reading tests, the homogeneity assumption was not violated (p > 0.05).

Example of Repeated-Measures ANOVA Results Table 14.3.7 reports the Tests of Between-Subjects Effects, which will tell us whether there is a statistical group difference. According to this table, we found that the three groups did not significantly differ from one another (F(2, 57) = 2.719, p = 0.08, ηp2 = 0.09).

Discussion What are the principles underlying an ANOVA? How do they differ from an independent-samples t-test? What kind of experimental situations do we need to use an ANCOVA? What are experimental research designs (discussed in Chapter 4) that a repeated-measures ANOVA is suitable for? What is a post hoc test? When do we need to use a post hoc test?