1. On Some Statistical Aspects of Agreement Among Measurements BIKAS K SINHA [ISI, Kolkata] Tampere August 28, 2009

2. Part II : Statistical Assessment of Agreement Understanding Agreement among Raters involving Continuous Measurements…. Theory & Applications…..

3. Continuous Measurements Evaluation of agreement when the data are measured on a continuous scale…… Pearson correlation coefficient, regression analysis, paired t-tests, least squares analysis for slope and intercept, within-subject coefficient of variation, and intra-class correlation coefficient…..

4. General Overview…. 1. Comparison of Gold Standard or Reference Method and one (or more) New or Test Method(s) If the two agree fairly well, we can use them interchangeably or the New One which is possibly cheaper or more convenient in place of the Gold Standard ! 2. Calibration : Establish mathematical relationship between the two sets of measurements.

5. Overview…contd. 3. Conversion : Compare two approx. methods, measuring same underlying quantity. Goal : Interpret results of one in terms of the other Temperature recorded in two instruments…...one in ^oF and the other in ^oC. Talk focuses on # 1 : Comparison of GS & TM GS : Gold Standard & TM : Test Method

6. Two Approaches….. Bland & Altman :Limits Of Agreement [LOA] Approach [with over 6000 citations in the Institute for Scientific Information Database] Lawrence Lin : Use of Concordance Correlation Coefficient Lin & Collaborators…..serious in-depth study with pharmaceutical applications

7. LOA Approach….. Subjects Rater 1 Rater 2 1 x_1 y_1 2 x_2 y_2 …………………………….. n x_n y_n Model : x_ j = S_ j + Beta_1 + e_ 1j y_ j = S_ j + Beta_2 + e_2 j S_ j : True Unobservable Measurement for the j-th subject…randomly distributed as

8. LOA Approach….Model…. N(Mu, sigma^2_s) Beta_1 & Beta_2 : Fixed Raters’ Bias Terms e_1 j : iid N(0, sigma^2_e1) e_2 j : iid N(0, sigma^2_e2) S_ j, e_1j, e_2 j ….all independent This is Grubbs’ Model sigma^2_s : Between-subject variance sigma^2_e = measurement error variance

9. LOA Approach…..Model…. E(X) = Mu + Beta_1, V(X) = sigma^2_s + sigma^2_e1 = sigma^2_x E(Y) = Mu + Beta_2 V(Y) = sigma^2_s + sigma^2_e2 = sigma^2_y Cov(X, Y) = sigma^2_s Rho = sigma^2_s / sigma_x. sigma_y Rho_x = sigma^2_s / sigma^2_x = Reliability Coeff. for Rater 1

10. LOA Approach…..Model… Rho_y = sigma^2_s / sigma^2_y = Reliability Coeff. for Rater 2 Rho^2 = Rho^2_x. Rho^2_y Notion of Perfect Agreement : All paired observations (x_ j, y_ j) lie on the 45^o line through Origin Equivalent Conditions : Same means, same variances and Rho = 1 Leads to Testing Issues……

11. LOA Approach….Data Analysis m = E(X – Y) = (m 1 –m 2 )  2 = Var(X-Y) = (  1 2 +  2 2 - 2  1  2 ) Estimates are based on paired data LOA has 2 components : (i) 95% LOA, defined by m^ +/- 1.96  ^ (ii) Plot of mean (x+y)/2 vs D = x – y, with LOA superimposed....Bland- Altman Plot...SAS JMP produces Plot

12. LOA Approach…. If a large proportion of the paired differences [D’s] are sufficiently close to zero, the two methods have satisfactory agreement. Step I : Estimate the set m +/- 1.96  Step II : Declare ‘sufficient’ agreement if the differences within these limits are not clinically important as determined by the investigator specified threshold value delta_o depending on the question of clinical judgement.

13. Lin Approach…. Lin et al in a series of papers made in-depth study of agreement using such notions as concordance correlation coefficient, total deviation index, coverage probability etc We will now elaborate on these concepts.

14. Continuous Measurements Two raters – n units for measurement Data : [{x i, y i }; 1 ≤ i ≤ n] Scatter Plot : Visual Checking Product Moment Corr. Coeff.: High +ve : What does it mean ? Squared Deviation : D 2 = (X-Y) 2 MSD:E[D 2 ]=(m 1 –m 2 ) 2 + (  1 2 +  2 2 - 2  1  2 )

15. Carotid Stenosis Screening Study Emory Univ.1994-1996 Gold Standard : Invasive Intra-arterial Angiogram [IA] Method Non-invasive Magnetic Resonance Angiography [MRA] Method Two Measurements under MRA: 2D & 3D Time of Flight Three Technicians : Each on Left & Right Arteries for 55 Patients by IA & MRA [2d & 3D] :3x3x2 =18 Obs. / patient

16. Data Structure…. Between Technicians : No Difference Left vs Right : Difference 2D vs 3D : Difference Q. Agreement between IA & 2D ? 3D ? Barnhart & Williamson (2001, 2002) : Biometrics papers …..no indication of any strong agreement

17. Scatter Plot : IA-2D-3D

18. Right vs Left Arteries [IA]

19. Descriptive Statistics : Carotid Stenosis Screening Study Sample Means Methods 1A, MRA-2D & MRA-3D by Sides Method N Left Artery Right Artery ---------------------------------------------------------- 1A 55 4.99 4.71 MRA-2D 55 5.36 5.73 MRA-3D 55 5.80 5.52

20. Descriptive Statistics (contd.) Sample Variance – Covariance Matrix 1A MRA-2D MRA-3D L R L R L R 1A-L 11.86 1.40 8.18 1.18 6.80 1.08 1A-R 10.61 2.67 7.53 1.78 7.17 2D-L 10.98 2.70 8.69 1.74 2d-R 8.95 2.19 7.69 3D-L 11.02 2.65 3D-R 10.24

21. Data Analysis : Lin Approach Recall MSD = E[(X-Y) 2 ] : Normed? No ! Lin (1989):Converted MSD to Corr.Coeff Concordance Corr. Coeff. [CCC] CCC = 1 – [MSD / MSD Ind. ] = 2  1  2 /[(m 1 –m 2 ) 2 + (  1 2 +  2 2 ) Properties :Perfect Agreement [CCC = 1] Perfect Disagreement [CCC = -1] No Agreement [CCC = 0]

22. CCC… CCC = 2  1  2 /[(m 1 –m 2 ) 2 + (  1 2 +  2 2 )] = .  a  = Accuracy Coefficient  a = Precision Coeff. [<=1]  a = [2 / {  + 1/  +  2 }] where  =  1 /  2 and  2 = (m 1 –m 2 ) 2 /  1  2 CCC = 1 iff  = 1 &  a = 1  a = 1 iff [ m 1 = m 2 ] & [  1 =  2 ] hold simultaneously !!

23. Study of CCC…. Identity of Marginals:Max.Precision High value of  : High Accuracy Needed BOTH for Agreement Simultaneous Inference on H 0 :    0, [m 1 = m 2 ] & [  1 =  2 ] LRT & Other Tests based on CCC Pornpis/Montip/Bimal Sinha (2006)‏ Thermo Pukkila Volume…..

24. Total Deviation Index Lin (1991) & Lin et al (JASA, 2002) Assume BVN distribution of (X,Y)‏  = P[ |Y – X| < k] = P[ D 2 < k 2 ]; D = Y - X =  2 [k 2, 1, m D 2 /  D 2 ]..non-central  2 TDI = Value of k for given  Inference based on TDI Choice of  : 90 % or more

25. Coverage Probability Lin et al (JASA, 2002) : BVN distribution CP(d) = P[ |Y – X| < d] =  [(d - m D ) /  D ] -  [(- d - m D ) /  D ] Emphasis is on given d and high CP. CP^ : Plug-in Estimator using sample means, variances & corr. coeff. Var[CP^] : LSA V^[CP^] : Plug-in Estimator

26. Graphical Display …

27. Back to Data Analysis… Carotid Stenosis Screening Study Emory Univ.1994-1996 GS : Method IA Competitors : 2D & 3D Methods Left & Right Arteries : Different Range of readings : 0 – 100 % Choice of d : 2%

28. Doctoral Thesis… Robieson, W. Z. (1999) : On the Weighted Kappa and Concordance Correlation Coefficient. Ph.D. Thesis, University of Illinois at Chicago, USA Lou, Congrong (2006) : Assessment of Agree- ment : Multi-Rater Case. Ph D Thesis, University of Illinois at Chicago, USA

29. Data Analysis…. Lou(2006) derived expressions for CP(d)^, V^(CP^(d)), COV^(…,…)‏ where CP iJ = P[|X i – X J |< d]

30. Data Analysis : CP 12, CP 13 & CP 23 Estimated Coverage Probability [CP] & Estimated Var. & Cov. for Screening Study Side Pairwise CP^ V^(CP^) COV^ Left CP 12 (L)^=0.56 0.0019 0.0009 Left CP 13 (L)^=0.47 0.0015 Right CP 12 (R)^=0.60 0.0021 0.0010 Right CP 13 (R)^=0.54 0.0019 Left CP 23 (L)^ =0.64 0.0021 Right CP 23 (R)^=0.69 0.0021

31. 95% Lower Confidence Limits Left Side CP 12 (L)^=0.56  95% Lower CL = 0.48 CP 13 (L)^=0.47  95% Lower CL = 0.40 Right Side CP 12 (R)^=0.60  95% Lower CL = 0.51 CP 13 (R)^=0.54  95% Lower CL = 0.46 Conclusion : Poor Agreement in all cases

32. Data Analysis (contd.)‏ Testing Hyp. Statistic p - value H 0L : CP 12 (L)= CP 13 (L) Z-score 0.0366 [against both-sided alternatives ] H 0R : CP 12 (R)= CP 13 (R) Z-score 0.1393 Conclusions : For “Left Side”, CP for [1A vs 2D] & for [1A vs 3D] are likely to be different while for “Right Side” these are likely to be equal.

33. Testing Multiple Hypotheses For “K” alternatives [1, 2, …, K] to the Gold Standard [0], interest lies in H 0L : CP 01 (L)= CP 02 (L) = … = CP 0K (L)‏ H 0R : CP 01 (R)= CP 02 (R) = … = CP 0K (R)‏ This is accomplished by performing Large Sample Chi-Square Test [Rao (1973)] Set for “Left Side”  L = ( CP 01 (L)^ CP 02 (L)^ ….CP 0 K (L)^)‏

34. Chi-Square Test… Chi-Sq.Test Statistic  L W^ -1  L - [  L W^ -1 1] 2 / [1 W^ -1 1] where W tt = Var ( CP 0t (L)^); t = 1, 2,.. W st = Cov ( CP 0s (L)^, CP 0t (L)^); s # t Asymptotic Chi-Sq. with K-1 df Slly…for “Right Side” Hypothesis.

35. Simultaneous Lower Confidence Limits Pr[ CP 01  L 1,CP 02  L 2, …,CP 0K  L k ]  95% Set Z t = [CP 0t ^ – CP 0t ] /  Var^(CP 0t ^)‏ Assume : Z t ‘s Jointly follow Multivariate Normal Dist. Work out estimated Correlation Matrix as usual. Solve for “z” such that Pr[Z 1  z, Z 2  z, Z 3  z,…, Z K  z]  95% Then L t = CP 0t ^ – z.  Var^(CP 0t ^)‏ t = 1, 2,.., K Stat Package : Available with Lou (2006).

36. Other Approaches….. Union-Intersection Principle…. H_o : [│m│ > d_m] U [     │  │  U  U  Excellent Review Paper by Choudhary & Nagaraja : Journal of Stat Planning & Inference.....

37. Conclusion Useful Statistical Concepts Sound Technical Tools Diverse Application Areas Scope for Further Research on Combining Evidences from Multi- Location Experiments...Meta Analysis ! Thanks !