1. Types of Reliability

2. Reliability of Consistency of What?
Observers or raters
Tests over time
Different versions of the same test
A test at one point in time

3. Inter-Rater or Inter-Observer Reliability
Object or
phenomenon

4. Inter-Rater or Inter-Observer Reliability
Object or
phenomenon
Observer 1

5. Inter-Rater or Inter-Observer Reliability
Object or
phenomenon
Observer 1
Observer 2

6. Inter-Rater or Inter-Observer Reliability
Object or
phenomenon ? =
Observer 1
Observer 2

7. Inter-Rater or Inter-Observer Reliability
Are different observers consistent?
Can establish this outside of your study in a pilot study.
Can look at percent of agreement (especially with category ratings).
Can use correlation (with continuous ratings).

8. Test-Retest Reliability
Time 1
Time 2

9. Test-Retest Reliability=
Test
Time 1
Time 2

10. Test-Retest Reliability
Stability over time
Test =
Test
Time 1
Time 2

11. Test-Retest Reliability
Measure instrument at two times for multiple persons.
Compute correlation between the two measures.
Assumes there is no change in the underlying trait between time 1 and time 2.

12. Parallel-Forms Reliability
Time 1
Time 2

13. Parallel-Forms Reliability
Form A =
Form B
Time 1
Time 2

14. Parallel-Forms Reliability
Stability across forms
Form A =
Form B
Time 1
Time 2

15. Parallel-Forms Reliability
Administer both forms to the same people.
Get correlation between the two forms.
Usually done in educational contexts where you need alternative forms because of the frequency of retesting and where you can sample from lots of equivalent questions.

16. Internal Consistency Reliability
Average inter-item correlation

17. Internal Consistency Reliability
Average Inter-Item Correlation
Test

18. Internal Consistency Reliability
Item 1
Average Inter-Item correlation
Item 2
Item 3
Test
Item 4
Item 5
Item 6

19. Internal Consistency Reliability
Item 1
Average inter-item correlation
Item 2
I1 I2 I3 I4 I5 I6
1.00 I1 I2 I3 I4 I5 I6
Item 3
Test
Item 4
Item 5
Item 6

20. Internal Consistency Reliability
Item 1
Average inter-item correlation
Item 2
I1 I2 I3 I4 I5 I6
1.00 I1 I2 I3 I4 I5 I6
Item 3
Test
Item 4
Item 5
Item 6
.90

21. Internal Consistency Reliability
Average item-total correlation

22. Internal Consistency Reliability
Average item-total correlation
Test

23. Internal Consistency Reliability
Average item-total correlation
Item 1
Item 2
Item 3
Test
Item 4
Item 5
Item 6

24. Internal Consistency Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6
Item 2 I1 I2 I3 I4 I5 I6
Total
1.00
Item 3
Test
Item 4
Item 5
Item 6

25. Internal Consistency Reliability
Average item-total correlation
Item 1
I1 I2 I3 I4 I5 I6
Item 2 I1 I2 I3 I4 I5 I6
Total
1.00
Item 3
Test
Item 4
Item 5
Item 6
.85

26. Internal Consistency Reliability
Split-half correlations

27. Internal Consistency Reliability
Split-half correlations
Test

28. Internal Consistency Reliability
Split-half correlations
Item 1
Item 2
Item 3
Test
Item 4
Item 5
Item 6

29. Internal Consistency Reliability
Split-half correlations
Item 1
Item 1
Item 3
Item 4
Item 2
Item 3
Test
Item 4
Item 5
Item 6

30. Internal Consistency Reliability
Split-half correlations
Item 1
Item 1
Item 3
Item 4
Item 2
Item 3
Test
Item 4
Item 5
Item 2
Item 5
Item 6
Item 6

31. Internal Consistency Reliability
Split-half correlations
Item 1
Item 1
Item 3
Item 4
Item 2
Item 3
Test
.87
Item 4
Item 5
Item 2
Item 5
Item 6
Item 6

32. Internal Consistency Reliability
Cronbach’s alpha ()

33. Internal Consistency Reliability
Cronbach’s alpha ()
Test

34. Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
Item 2
Item 3
Test
Item 4
Item 5
Item 6

35. Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
Item 2
Item 3
Test
Item 4
Item 5
Item 6

36. Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
Item 2
Item 3
Test
SH1 .87
SH2 .85
SH3 .91
SH4 .83
SH5 .86
...
SHn .85
Item 4
Item 5
Item 6

37. Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
Item 2
Item 3
Test
SH1 .87
SH2 .85
SH3 .91
SH4 .83
SH5 .86
...
SHn .85
Item 4
Item 5
Item 6
 = .85

38. Like the average of all possible split-half correlations
Internal Consistency Reliability
Cronbach’s alpha ()
Item 1
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
.87
item 1
item 3
item 4
item 2
item 5
item 6
Item 2
Item 3
Test
SH1 .87
SH2 .85
SH3 .91
SH4 .83
SH5 .86
...
SHn .85
Item 4
Like the average of all possible split-half correlations
Item 5
Item 6
 = .85

39. Internal Consistency Reliability - Summary
Average inter-item correlation
Average item-total correlation
Split-half reliability
Cronbach’s alpha ()