Calculations of Reliability We are interested in calculating the ICC –First step: Conduct a single-factor, within-subjects (repeated measures) ANOVA –This is an inferential test for systematic error –All subsequent equations are derived from the ANOVA table

Trial A1 Trial A2  Trial B1 Trial B2 

Repeated Measures ANOVA Steps for calculation: 1.Arrange the raw data (X) into tabular form, placing the data for subjects in rows (R), and repeated measures in columns (C).

Repeated Measures ANOVA Steps for calculation: 2.Square each value = (Trial A1) 2 3.Calculate the row totals (ΣR) using the original scores 4.Calculate the column totals (ΣC) using the original scores 5.Calculate the grand total (ΣX T ) or (ΣΣR) – same thing.

Repeated Measures ANOVA Steps for calculation: 6.Sum the row totals = ΣΣR – This is also the “total sum” of all original scores. – Also = ΣX T 7.Square each row total = (ΣR) 2 8.Sum the squares of the row totals = Σ(ΣR) 2

Repeated Measures ANOVA Steps for calculation: 9.Compute the mean values for each column.

Repeated Measures ANOVA Steps for calculation: 10.Sum the squares of columns = Σ(Trial A1) 2 11.Sum the sum of squared columns = Σ(Σ(Trial A1) 2 ) – This is also referred to as ΣX Σ(ΣR) 2 was calculated in step N = the number of subjects. 14.k = the number of trials.

Repeated Measures ANOVA Steps for calculation: 15.Compute the sum of squares between columns (SS C ), which is the variability due to the repeated-measures treatment effect. – In this case, SS C is “systematic variability.”

Repeated Measures ANOVA Steps for calculation: 16.Compute the sum of squares between rows (SS R ), which is the variability due to differences among subjects.

Repeated Measures ANOVA Steps for calculation: 17.Calculate the total sum of squares (SS T ), which is the variability due to subjects (rows), treatment (columns), and unexplained residual variability (error).

Repeated Measures ANOVA Steps for calculation: 18.Calculate the total sum of squares due to error (SS E ), which is the unexplained variability due to error. This will be used in the denominator for the F ratio.

Repeated Measures ANOVA Steps for calculation: 19.Calculate the degrees of freedom for each source of variance (df C, df R, df E, and df T ).

Repeated Measures ANOVA Steps for calculation: 20.Construct an ANOVA table: df R df C df E df T SS R SS C SS E SS T Between Subjects = rows Trials = columns Within Subjects = Trials + Error

Repeated Measures ANOVA Steps for calculation: 21.Calculate the mean square for each source of variance (MS C, MS R, and MS E ).

Repeated Measures ANOVA Steps for calculation: 22.Calculate the F ratio for the treatment effect (columns, F C ).

Repeated Measures ANOVA Determining the Significance of F: –Use the F Distribution Critical Values table. df C = df B – columns across the top df E = df E – rows down the side –If your calculated F ratio is greater than the critical F ratio, then reject the null hypothesis. There is a significant difference from Trial A1 to Trial A2 There is a significant systematic error –If your calculated F ratio is less than the critical F ratio, then accept the null hypothesis. There is no difference from Trial A1 to Trial A2 There is no systematic error

Using ANOVA Table for ICC 2 sources of variability for ICC model 3,1 –Subjects (MS S ) Between-subjects variability (for calculating the ICC) –Error (MS E ) Random error (for calculating the ICC) Equation reported by Weir (2005)Same equation, but modified for our terminology (MS S = MS R ).

MS E MS R or MS S

Using ANOVA Table for ICC Calculating the ICC 3,1 :

Interpreting the ICC If ICC = 0.95 –95% of the observed score variance is due to true score variance –5% of the observed score variance is due to error 2 factors for examining the magnitude of the ICC –Which version of the ICC was used? –Magnitude of the ICC depends on the between- subjects variability in the data Because of the relationship between the ICC magnitude and between-subjects variability, standard error of measurement values (SEM) should be included with the ICC

Implications of a Low ICC Low reliability Real differences –Argument to include SEM values Type I vs. Type II error –Type I error is rejecting H 0 when there was no effect (i.e., H 0 = 0) –Type II error is failing to reject the H 0 when there is an effect (i.e., H 0 ≠ 0) A low ICC means that more subjects will be necessary to overcome the increased percentage of the observed score variance due to error.

Standard Error of Measurement ICC  relative measure of reliability –No units SEM  absolute index of reliability –Same units as the measurement of interest The SEM is the standard error in estimating observed scores from true scores.

Calculating the SEM Calculating the SEM 3,1 :

SEM We can report SEM values in addition to the ICC values and the results of the ANOVA We can calculate the minimum difference (MD) that can be considered “real” between scores

Minimum Difference The SEM can be used to determine the minimum difference (MD) to be considered “real” and can be calculated as follows:

Example Problem Now use your skills (by hand) to calculate a repeated measures ANOVA, ICC 3,1, SEM 3,1, and MD 3,1 for Trials B1 and B2. –Report your results. –Compare your results to Trials A1 and A2. What is the primary difference?

Trial A1 Trial A2  Trial B1 Trial B2 

Using the Reliability Worksheet Online Go to the course website and download the Reliability.xls worksheet. –Calculate the ANOVA, ICC, SEM, and MD values for both Trials A1 and A2 and Trials B1 and B2 and compare your results.