1. Comparing Proportions & Analysing Categorical Data Scott Harris October 2009

2. 2 Learning outcomes By the end of this session you should be able to choose between, perform (using SPSS and CIA) and interpret the results from the following methods of analysing categorical data: –A test for association or independence (Chi-square or Fisher’s exact test). –A test for assessing if a sample proportion differs from a specified proportion (Chi-square). –A test for a change in categorical response (McNemar’s test). –A test of agreement of categories between 2 raters (Kappa test). You should also be aware of the concept of odds, odds ratios and how to calculate them from a 2x2 table.

3. 3 Contents Introduction –Refresher - types of data. –Data requirements. –The example dataset: CISR data. Association between 2 variables –Test information. –‘How to’ in SPSS and CIA. One sample versus a specified proportion –Test information. –‘How to’ in SPSS and CIA.

4. 4 Contents Change in response –Test information. –‘How to’ in SPSS and CIA. Quick crosstabs –Summary Data tables in SPSS: ‘How to’ Agreement between 2 raters –Test information. –‘How to’ in SPSS and CIA.

5. 5 Refresher: Types of data Quantitative – a measured quantity. –Continuous – Measurements from a continuous scale: Height, weight, age. –Discrete – Count data: Children in a family, number of days in hospital. Qualitative – Assessing a quality. –Ordinal – An order to the data: Likert scale (much worse, worse, the same, better, much better), age group (18-25, 26- 30…). –Categorical / Nominal – Simple categories: Blood group (O, A, B, AB). A special case is binary data (two levels): Status (Alive, dead), Infection (yes, no).

6. 6 Data requirements The Statistical tests that will be covered in this talk compare a sample with a categorical outcome against either: –a published or hypothesised proportion, –another group / another category or multiple categories, –a repeated categorical outcome from the same individual, –another measurement of the same outcome from another source or –a gold standard ‘true’ outcome. A different type of test / method is used in each of the situations above.

7. 7 Example dataset: Information CISR (Clinical Interview Schedule: Revised) data: –Measure of depression – the higher the score the worse the depression. –A CISR value of 12 or greater is used to indicate a clinical case of depression. –3 groups of patients (each receiving a different form of treatment: GP, CMHN and CMHN problem solving). –Data collected at two time points (baseline and then a follow- up visit 6 months later). –An additional reading at 6 months was taken by another researcher.

8. 8 Example CISR dataset: Raw data

9. 9 Example CISR dataset: Labelled data

10. Association / Independence (Difference in proportions) Chi-square test or Fisher’s exact test

11. 11 The most common statistic used when dealing with categorical data. Alongside the t test this is the most often seen statistical technique. The Chi-square ( ) or Pearson Chi-squared statistic compares the observed proportion of a categorical response with an expected value. The null hypothesis (as always) is that there is no difference or no association between the variables (depending on the context). As the difference between these observed and expected values increases then the evidence supporting a difference or an association builds up. Chi-square statistic

12. 12 The following equation is used to calculate the chi-square statistic: Observed = The actual count in each cell. Expected =The number expected to be in each cell of the table if the test proportion was true or the rows and columns were unrelated / independent (i.e. assuming no difference in response). This is distributed with (n 1 -1) x (n 2 -1) degrees of freedom. Theory: Chi-square statistic Number of levels for row variable Number of levels for column variable

13. 13 Chi-square statistic: Example Table of Observed values for Clinical status at 6 months, split by Gender Clinical caseNon caseTotal Male1522 37 Female2448 72 Total3970109 We want to see if Gender is associated with Clinical status at 6 months

14. 14 Theory: Chi-square statistic Table of Observed values (Expected values) Clinical caseNon caseTotal Male15 (13.24)22 (23.76) 37 Female24 (25.76)48 (46.24) 72 Total3970109 The expected values are produced by multiplying together the 2 marginal totals and then dividing by the grand total. For Male clinical cases: 37 x 39 = 1443,1443 / 109 = 13.24 (2dp)

15. 15 Theory: Chi-square statistic For Male clinical cases: (15 - 13.24) 2 / 13.24 = 3.0976 / 13.24 = 0.23 (2dp) For Male non cases: (22 – 23.76) 2 / 23.76 = 3.0976 / 23.76 = 0.13 (2dp) For Female clinical cases: (24 – 25.76) 2 / 25.76 = 3.0976 / 25.76 = 0.12 (2dp) For Female non cases: (48 – 46.24) 2 / 46.24 = 3.0976 / 46.24 = 0.07 (2dp) Chi-square statistic = 0.23 + 0.13 + 0.12 + 0.07 = 0.552 (3dp)

16. 16 Chi-square alternative: Fisher’s exact test The chi-square test is only appropriate when the sample size is large enough that there are no ‘rare’ combinations of categories in the cross-tabulation. The definition of ‘rare’ is that the expected counts for all of the cells in the table need to be at least 5. If at least one of the cells in the table has an expected count <5 then the Pearson Chi-square statistic should not be reported and an alternative test called Fisher’s exact test should be used instead. Fisher’s exact test is an exact permutation test for categorical variables. It is convention to only use Fisher’s exact test when Pearson’s Chi-square is not appropriate.

17. 17 Chi-square statistic: SPSS Analyze  Descriptive statistics  Crosstabs… * Chi-square test for Sex and M6Cat. CROSSTABS /TABLES=SEX BY M6Cat /FORMAT= AVALUE TABLES /STATISTIC=CHISQ /CELLS= COUNT ROW /COUNT ROUND CELL.

18. 18 Chi-square statistic: Output Pearson Chi-square p value = 0.457 Fisher’s exact test p value = 0.529 (Notice how only the 2-sided tests are considered.) 2x2 cross tabulation with suitable percentages (as we are comparing the two genders here we use row percentages).

19. 19 Fisher’s exact test: Table > 2 x 2 Everything should be set up as before, with the following additional option:

20. 20 Chi-square statistic: 3 x 2 Output (Exact) Pearson Chi-square p value = 0.380 Fisher’s exact test p value = 0.379 (Again notice how only the 2-sided tests are considered.) 3x2 cross tabulation with suitable percentages (as we are comparing the treatment groups here we use row percentages).

21. 21 Info: Chi-square in SPSS 1)From the menus select ‘Analyze’  ‘Descriptive Statistics’  ‘Crosstabs…’. 2)Put one of your categorical variables into the ‘Row(s):’ box and the other into the ‘Column(s):’ box. 3)Click the ‘Cells…’ button and then select the box for any percentages that you require. Then click the ‘Continue’ button. 4)Click the ‘Statistics…’ button and tick the option for ‘Chi- square’. Then click the ‘Continue’ button. 5)If your table will be bigger than a 2x2 then click the ‘Exact…’ button and tick the option for ‘Exact’. Then click the ‘Continue’ button. 6)Finally click ‘OK’ to produce the cross tabulation with the Chi- square statistic or ‘Paste’ to add the syntax for this into your syntax file.

22. 22 Theory: Options for a 2 x 2 table Non case Clinical caseTotal Female 48 (a) 24 (b) 72 (a+b) Male 22 (c) 15 (d) 37 (c+d) 1. Difference in proportions (absolute difference): 2. Relative risk (multiplicative difference): 3. Another common alternative is the Odds ratio: Both the Chi-square test and Fisher’s exact tests are tests for association between the two independent variables. They do not quantify the effect size. For a 2 x 2 table there are a number of options that you can use to quantify effects, each of which has pros and cons.

23. 23 Odds are a tricky topic for some to understand, but easy for others. They are most commonly encountered in gambling situations. Theory: Odds and odds ratios Outcome of interest: Clinical case Odds for Females = b/a Odds for Males = d/c The odds for females being a clinical case are 24 to 48 or 1 to 2. For females, you would expect 1 clinical case for every 2 non cases. Odds ratios are simply the ratio of the 2 odds (1 divided by the other). Non case Clinical caseTotal Female 48 (a) 24 (b) 72 (a+b) Male 22 (c) 15 (d) 37 (c+d)

24. 24 Theory: Odds and odds ratios Outcome of interest: Clinical case Odds for Females = b/a = 24/48 = 0.5 Odds for Males = d/c = 15/22 = 0.68 (2dp) Odds ratios are produced in logistic regression This is often reported as 2 to 1 against in gambling situations. = 0.68 / 0.5 = 1.36 (2dp) The odds of being a clinical case are 1.36 times larger for Males than Females. Here we have taken females as the reference category (we have divided by the female result) thereby getting the relative ‘increase’ in odds for being male.

25. 25 Reminder: 95% confidence intervals in CIA Always check that you are producing 95% CI’s:  Options menu in CIA

26. 26 Methods  Proportions and their differences  Unpaired samples Difference in proportions: CIA Example

27. 27 It is Easiest to have the group with the largest ‘feature’ proportion as Sample 1. This will produce a positive difference. Difference in proportions: CIA Example

28. 28 Observed proportions and difference in proportions 95% confidence interval. (This CI includes 0, therefore agreeing with the earlier p value from SPSS) Difference in proportions: CIA Example

29. 29 Chi-square statistic: Presentation Table 1: Frequency table of clinical status by gender. Figures are number (percentage). GenderClinical caseNon case Male15 (40.5%)22 (59.5%) Female24 (33.3%)48 (66.7%) There was found to be no significant difference (Pearson Chi square: p = 0.457) in the proportions of clinical cases, in male and female patients (Difference 7.2%, 95% CI: -11.1% to 25.9%).

30. One sample vs. a specific proportion Chi-square test or Exact test

31. 31 Chi-square statistic for one variable In the same way that the chi-square test can be used when you have larger than a 2 x 2 table, it can also be used when you have a n x 1 table. In this situation you are testing whether the proportion of some event (or events) that you have seen are different to a value that you specify. In this case the expected values are calculated from the specified proportion(s) but in all other regards the test is computed in the same way.

32. 32 One variable Chi-square: Example Analyze  Nonparametric tests  Chi-square… * Chi-square test for B0cat vs. 0.1 / 0.9. NPAR TEST /CHISQUARE=B0Cat /EXPECTED=0.1 0.9 /MISSING ANALYSIS /METHOD=EXACT TIMER(5). The specified proportions are entered here, the ordering is important and is dependent on how the variable was set up.

33. 33 Info: One variable Chi-square in SPSS 1)From the menus select ‘Analyze’  ‘Nonparametric tests’  ‘Chi-square…’. 2)Put your categorical variable into the ‘Test variable list:’ box. 3)Either specify expected proportions one at a time in the ‘Values’ box (in the order of the levels in the categorical variable), each time clicking the ‘Add’ button or leave the test to compare against equal proportions. 4)Click the ‘Exact…’ button and tick the option for ‘Exact’. Then click the ‘Continue’ button. 5)Finally click ‘OK’ to produce the one variable Chi-square statistic or ‘Paste’ to add the syntax for this into your syntax file.

34. 34 One variable Chi-square: Output Pearson Chi-square p value = 0.213 Exact test p value = 0.263 The number of observed frequencies in each category as well as the expected number if the specified proportions of 0.1 and 0.9 were true.

35. 35 Single sample Chi-square: CIA Example Methods  Proportions and their differences  Single sample

36. 36 Single sample Chi-square: CIA Example Decide on which proportion you would like to produce the confidence interval for by setting that as the ‘feature’.

37. 37 Single sample Chi-square: CIA Example Observed proportion 95% confidence interval (This CI includes 0.9, therefore agreeing with the earlier p value from SPSS)

38. Test for change in paired proportions McNemar test

39. 39 The McNemar test When you have paired or repetitious binary categories then the Chi-square test is no longer appropriate and you should make use of an alternative test known as the McNemar test. An example of this type of data are the binary clinical case variables in the example dataset. Here we have the same information on an individual at both baseline and 6 months. These readings are paired categorical results. The McNemar test looks at whether there has been a significant shift in state in the two paired results. The focus of this test is whether there is a large shift in one direction rather than the other, as well as how much change there has actually been in the paired results.

40. 40 Theory: McNemar test Looking at the example cross tabulation on the right then: –The empty cells of the table indicate where there is no change / difference in outcomes 1 and 2. –The solid red cell indicates those who didn’t have the outcome at time 1, but did have it at time 2. –Vice versa the red striped cell indicates those who had the outcome at time 1, but not at time 2. If both of the coloured cells are sufficiently small in proportion then there has been little change in response. Likewise if the proportion in each of the shaded cells is similar, then overall there has been little change in response direction. Outcome at time 2 No outcome at time 2 Outcome at time 1 No outcome at time 1 McNemar’s test takes both of these into account

41. 41 McNemar test: Example Analyze  Descriptive statistics  Crosstabs… * McNemar test for B0Cat and M6Cat. CROSSTABS /TABLES=B0Cat BY M6Cat /FORMAT= AVALUE TABLES /STATISTIC=MCNEMAR /CELLS= COUNT TOTAL /COUNT ROUND CELL.

42. 42 Info: McNemar in SPSS 1)From the menus select ‘Analyze’  ‘Descriptive Statistics’  ‘Crosstabs…’. 2)Put one of your paired categorical variables into the ‘Row(s):’ box and the other into the ‘Column(s):’ box. 3)Click the ‘Cells…’ button and then select the box for ‘Total’ percentages. Then click the ‘Continue’ button. 4)Click the ‘Statistics…’ button and tick the option for ‘McNemar’. Then click the ‘Continue’ button. 5)Finally click ‘OK’ to produce the cross tabulation with the McNemar statistic or ‘Paste’ to add the syntax for this into your syntax file.

43. 43 From the above table we can see that 58.7% of the total sample were clinical cases at baseline and became non cases by 6 months, whereas only 0.9% initially were non cases and became cases. McNemar statistic: Example McNemar test p value: p <0.001 2x2 cross tabulation with overall percentages.

44. 44 McNemar test: CIA Example Methods  Proportions and their differences  Paired samples

45. 45 McNemar test: CIA Example There is an alternative Table view that may be easier for entering data:

46. 46 McNemar test: CIA Example It is easiest to put the largest change proportion into the bottom left corner of this table. In this way you will get positive differences:

47. 47 McNemar test: CIA Example Observed difference in proportions 95% confidence interval (This CI excludes 0, therefore agreeing with the earlier p value from SPSS)

48. 48 McNemar test: Presentation Table 2: Frequency table of change in clinical status. Figures are number (total percentage). 6 month status Baseline statusClinical caseNon case Clinical case 38 (34.9%)64 (58.7%) Non case 1 (0.9%)6 (5.5%) There was found to be a highly significant change in clinical status of depression (McNemar: p < 0.001), from baseline to 6 months (Difference 57.8%, 95% CI: 47.0% to 66.5%) in favour of a lessoning of symptoms (reduction of clinical cases).

49. Quick Crosstabs Dealing with summary data in SPSS

50. 50 Summary Data: SPSS If you only have access to the cross-tabulated data (or you only have to do one quick analysis) then rather than having to enter one row of data for each individual in the dataset you can enter it as a summary data table. A summary data table will only contain one row for each cross combination in the table (one row per table cell). The number of observations in that table will not affect the amount of data entry. Summary data tables can be used for any, all categorical technique.

51. 51 Summary Data: SPSS One variable in SPSS should contain the category of the row from the table. Another variable should contain the category of the column from the table. A third variable (Count) should contain the number of observations in that combination of categories. Once the data are entered you will need to let SPSS know that the data have been entered to represent more than 1 observation per row. To do this you will need to make use of the ‘Weight cases’ option. You can now analyse the data in the normal way.

52. 52 Summary Data: Example One of the crosstabs from earlier: Table of Observed values for Clinical status at 6 months, split by Gender Clinical caseNon caseTotal Male1522 37 Female2448 72 Total3970109

53. 53 Summary Data: SPSS * Weighting the summary data. WEIGHT BY Count. Warning displayed in bottom left corner of data editor, when weight cases is on. Data  Weight cases…

54. 54 Chi-square statistic: Summary Data Analyze  Descriptive statistics  Crosstabs… * Chi-square test for Sex and M6Cat. CROSSTABS /TABLES=Gender BY Clin_status /FORMAT= AVALUE TABLES /STATISTIC=CHISQ /CELLS= COUNT ROW /COUNT ROUND CELL.

55. 55 Info: Summary data in SPSS 1)One variable in SPSS should contain the category of the row from the table. 2)Another variable should contain the category of the column from the table. 3)A third variable should contain the number of observations in that combination of categories 4)From the menus select ‘Data’  ‘Weight cases…’. 5)Select ‘Weight cases by’. 6)Put the third variable that contains the number of observations for each row into the ‘Frequency Variable:’ box. 7)Finally click ‘OK’ to weight the dataset or ‘Paste’ to add the syntax for this into your syntax file. 8)You can now conduct your analysis in the usual way.

56. 56 Radiologist B Radiologist A NormalBenign Suspected cancer Cancer Total Normal 21 12 0 033 Benign 4 17 1 022 Suspected cancer 3 9 15 229 Cancer 0 0 0 11 Total 283816385 Summary Data: Example 2 The following table is a classification by two radiologists of 85 xeromammograms (Boyd et al., 1982):

57. 57 Summary Data: SPSS

58. 58 Summary Data: SPSS * Weighting the summary data. WEIGHT BY Count. Data  Weight cases… Warning displayed in bottom left corner of data editor, when weight cases is on.

59. Agreement between two raters or devices Kappa statistic

60. 60 Kappa Statistic It is possible to assess the level of agreement between 2 raters or 2 methods (when the responses are categorical) by making use of the Kappa statistic. The Kappa statistic looks at the observed proportion of agreeing responses between the 2 raters and compares this to what may have been expected just by chance alone. It is considered a better statistic than just looking simply at the percent agreement as it takes into account agreement occurring simply by chance. Kappa can be used for any size of data table and for either nominal or ordinal categorical data. For ordinal data there is an alternative known as Weighted Kappa but this cannot be computed easily in SPSS. The diagonal of the cross tabulation that indicates agreement between the raters is known as the ‘diagonal’ and the other cells that indicate disagreement are know as the ‘off diagonals’.

61. 61 Kappa Statistic: Interpretation Kappa (2dp)Strength of agreement ≤ 0.20Poor 0.21 to 0.40Fair 0.41 to 0.60Moderate 0.61 to 0.80Good 0.81 to 1.00Very good In a similar approach as to that used for correlation the emphasis for a Kappa statistic is not placed on the significance of the p value but the actual value of the Kappa statistic. The table to the right is the most common way of assessing the strength of the agreement. It simply involves looking up the Kappa statistic in the table and reporting the appropriate level of agreement. (D G Altman, Practical Statistics for Medical Research, 1999)

62. 62 Analyze  Descriptive statistics  Crosstabs… * Kappa for M6Cat and M6Cat2. CROSSTABS /TABLES=M6Cat BY M6Cat2 /FORMAT= AVALUE TABLES /STATISTIC=KAPPA /CELLS= COUNT TOTAL /COUNT ROUND CELL. Kappa statistic: SPSS

63. 63 Info: Kappa in SPSS 1)From the menus select ‘Analyze’  ‘Descriptive Statistics’  ‘Crosstabs…’. 2)Put one of your paired categorical variables into the ‘Row(s):’ box and the other into the ‘Column(s):’ box. 3)Click the ‘Cells…’ button and then select the box for ‘Total’ percentages. Then click the ‘Continue’ button. 4)Click the ‘Statistics…’ button and tick the option for ‘Kappa’. Then click the ‘Continue’ button. 5)Finally click ‘OK’ to produce the cross tabulation with the Kappa statistic or ‘Paste’ to add the syntax for this into your syntax file.

64. 64 From the above table we can see that there is 91.7% actual agreement between the 2 measures (57.8% + 33.9%). Kappa statistic: Example Kappa test p value: p <0.001 2x2 cross tabulation with overall percentages. Kappa statistic, indicating very good agreement.

65. 65 Kappa statistic: CIA Example Methods  Diagnostic studies  Kappa

66. 66 Kappa statistic: CIA Example It doesn’t matter which way around you enter the rows and columns of this table, as long as the diagonal (1,1 and 2,2 in this case) indicate agreement:

67. 67 Kappa statistic: CIA Example Observed Kappa value 95% confidence interval (This CI excludes 0, therefore agreeing with the earlier p value from SPSS)

68. 68 Summary You should now be able to choose between, perform (using SPSS and CIA) and interpret the results from the following methods of analysing categorical data: –A test for association or independence (Chi-square or Fisher’s exact test). –A test for assessing if a sample proportion differs from a specified proportion (Chi-square). –A test for a change in categorical response (McNemar’s test). –A test of agreement of categories between 2 raters (Kappa test). –You should also be aware of the concept of odds, odds ratios and how to calculate them from a 2x2 table.

69. 69 References Practical statistics for medical research, D Altman: Chapters 10 & 14. Medical statistics, B Kirkwood, J Stern: Chapters 16 & 17. An introduction to medical statistics, M Bland: Chapter 13. Statistics for the Terrified: Analysing 2x2 classification tables. http://statpages.org/ctab2x2.html Swets JA, Pickett RM. Evaluation of diagnostic systems. New York: Academic Press,1982. Langlotz CP. Fundamental measures of diagnostic examination performance: usefulness for clinical decision making and research. Radiology 2003; 228:3-9. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143:29-36. Hanley JA, McNeil BJ. A Method of Comparing the Areas under Receiver Operating Characteristic Curves Derived from the Same Cases. Radiology 1983; 148:839- 843.