Methods for Estimating Reliability Dr. Shahram Yazdani

Types of Reliability Inter-Rater or Inter-Observer Reliability: Used to assess the degree to which different raters or observers give consistent estimates of the same phenomenon Test-Retest Reliability: Used to assess the consistency of a measure from one time to another Parallel-Forms Reliability: Used to assess the consistency of the results of two tests constructed same way from the same content domain Internal Consistency Reliability: Used to assess the consistency of results across items within a test Dr. Shahram Yazdani

Interrater or Interobserver Reliability object or phenomenon observer 1 observer 2 = ? Dr. Shahram Yazdani

Inter-rater Reliability Statistics used Nominal/categorical data Kappa statistic Ordinal data Kendall’s tau to see if pairs of ranks for each of several individuals are related Two judges rate 20 elementary school children on an index of hyperactivity and rank order them Interval or ratio data Pearson r using data obtained from the hyperactivity index Dr. Shahram Yazdani

Test-Retest Reliability = Stability over Time test time 1 time 2 Dr. Shahram Yazdani

Test-Retest Reliability Statistics used Pearson r or Spearman rho Important caveat Correlation decreases over time because error variance INCREASES (and may change in nature) Closer in time the two scores were obtained, the more the factors which contribute to error variance are the same Dr. Shahram Yazdani

Parallel-Forms Reliability form B form A Stability Across Forms = time 1 time 2 Dr. Shahram Yazdani

Parallel Forms Reliability Statistic used Pearson r or Spearman rho Important caveat Even when randomly chosen, the two forms may not be truly parallel Dr. Shahram Yazdani

Internal consistency Internal consistency Average inter-item correlation Average item total correlation Split-half reliability Dr. Shahram Yazdani

Average Inter-item Correlation Definition: calculate correlation of each item (Pearson r) with all other items. Dr. Shahram Yazdani

Internal Consistency Reliability item 1 Average Inter-Item Correlation item 2 I1 I2 I3 I4 I5 I6 I1 I2 I3 I4 I5 I6 1.00 .89 1.00 .91 .92 1.00 .88 .93 .95 1.00 .84 .86 .92 .85 1.00 .88 .91 .95 .87 .85 1.00 item 3 test item 4 item 5 .90 item 6 Dr. Shahram Yazdani

Average Item Total Correlation Definition: calculate correlation of each item scores with total score. Dr. Shahram Yazdani

Internal Consistency Reliability Average Item-Total Correlation item 1 I1 I2 I3 I4 I5 I6 item 2 I1 I2 I3 I4 I5 I6 Total 1.00 .89 1.00 .91 .92 1.00 .88 .93 .95 1.00 .84 .86 .92 .85 1.00 .88 .91 .95 .87 .85 1.00 .84 .88 .86 .87 .83 .82 1.00 item 3 test item 4 item 5 item 6 .85 Dr. Shahram Yazdani

Split-half Reliability Definition: Randomly divide the test into two forms; calculate scores for Form A, B; calculate Pearson r as index of reliability Dr. Shahram Yazdani

Internal Consistency Reliability Split-Half Correlations item 1 item 1 item 3 item 4 item 2 .87 item 3 test item 4 item 5 item 2 item 5 item 6 item 6 Dr. Shahram Yazdani

Cronbach’s alpha & Kuder-Richardson-20 Measures the extent to which items on a test are homogeneous; mean of all possible split-half combinations Kuder-Richardson-20 (KR-20): for dichotomous data Cronbach’s alpha: for non-dichotomous data Dr. Shahram Yazdani

Internal Consistency Reliability Cronbach’s alpha () item 1 .87 item 1 item 3 item 4 item 2 item 5 item 6 .85 item 1 item 3 item 4 item 2 item 5 item 6 .91 item 1 item 3 item 4 item 2 item 5 item 6 item 2 item 3 test SH1 .87 SH2 .85 SH3 .91 SH4 .83 SH5 .86 ... SHn .85 item 4 item 5 item 6  = .85 Dr. Shahram Yazdani

Reducing Measurement Error pilot test your instruments -- get feedback from respondents train your interviewers or observers make observation/measurement as unobtrusive as possible double-check your data triangulate across several measures that might have different biases Dr. Shahram Yazdani

Validity vs Reliability Dr. Shahram Yazdani

Thank you ! Any Question ? Dr. Shahram Yazdani