1. Logistic Regression Binary response variable Y (1 – Success, 0 – Failure) Continuous, Categorical independent Variables –Similar to Multiple Regression –Can use Dummy variables to handle categorical independent variables –Various selection procedures for determining the “best” model – forward, backward, stepwise

2. Discrete Multivariate Analysis Analysis of Multivariate Categorical Data

3. References 1.Fienberg, S. (1980), Analysis of Cross-Classified Data, MIT Press, Cambridge, Mass. 2.Fingelton, B. (1984), Models for Category Counts, Cambridge University Press. 3.Alan Agresti (1990) Categorical Data Analysis, Wiley, New York.

4. Example 1 In this study we examine n = 1237 individuals measuring X, Systolic Blood Pressure and Y, Serum Cholesterol

5. Example 2 The following data was taken from a study of parole success involving 5587 parolees in Ohio between 1965 and 1972 (a ten percent sample of all parolees during this period).

6. The study involved a dichotomous response Y –Success (no major parole violation) or –Failure (returned to prison either as technical violators or with a new conviction) based on a one-year follow-up. The predictors of parole success included are: 1.type of committed offence (Person offense or Other offense), 2.Age (25 or Older or Under 25), 3.Prior Record (No prior sentence or Prior Sentence), and 4.Drug or Alcohol Dependency (No drug or Alcohol dependency or Drug and/or Alcohol dependency).

7. The data were randomly split into two parts. The counts for each part are displayed in the table, with those for the second part in parentheses. The second part of the data was set aside for a validation study of the model to be fitted in the first part.

8. Table

9. Analysis of a Two-way Frequency Table:

10. Frequency Distribution (Serum Cholesterol and Systolic Blood Pressure)

11. Joint and Marginal Distributions (Serum Cholesterol and Systolic Blood Pressure) The Marginal distributions allow you to look at the effect of one variable, ignoring the other. The joint distribution allows you to look at the two variables simultaneously.

12. Conditional Distributions ( Systolic Blood Pressure given Serum Cholesterol ) The conditional distribution allows you to look at the effect of one variable, when the other variable is held fixed or known.

13. Conditional Distributions (Serum Cholesterol given Systolic Blood Pressure)

14. GRAPH: Conditional distributions of Systolic Blood Pressure given Serum Cholesterol

15. Notation: Let x ij denote the frequency (no. of cases) where X (row variable) is i and Y (row variable) is j.

16. Different Models The Multinomial Model: Here the total number of cases N is fixed and x ij follows a multinomial distribution with parameters  ij

17. The Product Multinomial Model: Here the row (or column) totals R i are fixed and for a given row i, x ij follows a multinomial distribution with parameters  j|i

18. The Poisson Model: In this case we observe over a fixed period of time and all counts in the table (including Row, Column and overall totals) follow a Poisson distribution. Let  ij denote the mean of x ij.

19. Independence

20. Multinomial Model if independent and The estimated expected frequency in cell (i,j) in the case of independence is:

21. The same can be shown for the other two models – the Product Multinomial model and the Poisson model namely The estimated expected frequency in cell (i,j) in the case of independence is: Standardized residuals are defined for each cell:

22. The Chi-Square Statistic The Chi-Square test for independence Reject H 0 : independence if

23. Table Expected frequencies, Observed frequencies, Standardized Residuals  2 = 20.85 (p = 0.0133)

24. Example In the example N = 57,407 cases in which individuals were victimized twice by crimes were studied. The crime of the first victimization (X) and the crime of the second victimization (Y) were noted. The data were tabulated on the following slide

25. Table 1: Frequencies

26. Table 2: Standardized residuals

27. Table 3: Conditional distribution of second victimization given the first victimization (%)

28. Log Linear Model

29. Recall, if the two variables, rows (X) and columns (Y) are independent then and

30. In general let then where (1) Equation (1) is called the log-linear model for the frequencies x ij.

31. Note: X and Y are independent if In this case the log-linear model becomes

32. Three-way Frequency Tables

33. Example Data from the Framingham Longitudinal Study of Coronary Heart Disease (Cornfield [1962]) Variables 1.Systolic Blood Pressure (X) –< 127, 127-146, 147-166, 167+ 2.Serum Cholesterol –<200, 200-219, 220-259, 260+ 3.Heart Disease –Present, Absent The data is tabulated on the next slide

34. Three-way Frequency Table

35. Log-Linear model for three-way tables Let  ijk denote the expected frequency in cell (i,j,k) of the table then in general where

36. Hierarchical Log-linear models for categorical Data For three way tables The hierarchical principle: If an interaction is in the model, also keep lower order interactions and main effects associated with that interaction

37. 1.Model: (All Main effects model) ln  ijk = u + u 1(i) + u 2(j) + u 3(k) i.e. u 12(i,j) = u 13(i,k) = u 23(j,k) = u 123(i,j,k) = 0. Notation: [1][2][3] Description: Mutual independence between all three variables.

38. 2.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 12(i,j) i.e. u 13(i,k) = u 23(j,k) = u 123(i,j,k) = 0. Notation: [12][3] Description: Independence of Variable 3 with variables 1 and 2.

39. 3.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 13(i,k) i.e. u 12(i,j) = u 23(j,k) = u 123(i,j,k) = 0. Notation: [13][2] Description: Independence of Variable 2 with variables 1 and 3.

40. 4.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 23(j,k) i.e. u 12(i,j) = u 13(i,k) = u 123(i,j,k) = 0. Notation: [23][1] Description: Independence of Variable 3 with variables 1 and 2.

41. 5.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 12(i,j) + u 13(i,k) i.e. u 23(j,k) = u 123(i,j,k) = 0. Notation: [12][13] Description: Conditional independence between variables 2 and 3 given variable 1.

42. 6.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 12(i,j) + u 23(j,k) i.e. u 13(i,k) = u 123(i,j,k) = 0. Notation: [12][23] Description: Conditional independence between variables 1 and 3 given variable 2.

43. 7.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 13(i,k) + u 23(j,k) i.e. u 12(i,j) = u 123(i,j,k) = 0. Notation: [13][23] Description: Conditional independence between variables 1 and 2 given variable 3.

44. 8.Model: ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 12(i,j) + u 13(i,k) + u 23(j,k) i.e. u 123(i,j,k) = 0. Notation: [12][13][23] Description: Pairwise relations among all three variables, with each two variable interaction unaffected by the value of the third variable.

45. 9.Model: (the saturated model) ln  ijk = u + u 1(i) + u 2(j) + u 3(k) + u 12(i,j) + u 13(i,k) + u 23(j,k) + u 123(i,j,k) Notation: [123] Description: No simplifying dependence structure.

46. Hierarchical Log-linear models for 3 way table ModelDescription [1][2][3] Mutual independence between all three variables. [1][23] Independence of Variable 1 with variables 2 and 3. [2][13] Independence of Variable 2 with variables 1 and 3. [3][12] Independence of Variable 3 with variables 1 and 2. [12][13] Conditional independence between variables 2 and 3 given variable 1. [12][23] Conditional independence between variables 1 and 3 given variable 2. [13][23] Conditional independence between variables 1 and 2 given variable 3. [12][13] [23] Pairwise relations among all three variables, with each two variable interaction unaffected by the value of the third variable. [123] The saturated model

47. Maximum Likelihood Estimation Log-Linear Model

48. For any Model it is possible to determine the maximum Likelihood Estimators of the parameters Example Two-way table – independence – multinomial model or

49. Log-likelihood where With the model of independence

50. and with also

51. Let Now

52. Since

53. Now or

54. Hence and Similarly Finally

55. Hence Now and

56. Hence Note or

57. Comments Maximum Likelihood estimates can be computed for any hierarchical log linear model (i.e. more than 2 variables) In certain situations the equations need to be solved numerically For the saturated model (all interactions and main effects), the estimate of  ijk… is x ijk….

58. Goodness of Fit Statistics These statistics can be used to check if a log-linear model will fit the observed frequency table

59. Goodness of Fit Statistics The Chi-squared statistic The Likelihood Ratio statistic: d.f. = # cells - # parameters fitted We reject the model if  2 or G 2 is greater than

60. Example: Variables 1.Systolic Blood Pressure (B) Serum Cholesterol (C) Coronary Heart Disease (H)

61. MODEL DF LIKELIHOOD- PROB. PEARSON PROB. RATIO CHISQ CHISQ ----- -- ----------- ------- ------- ------- B,C,H. 24 83.15 0.0000 102.00 0.0000 B,CH. 21 51.23 0.0002 56.89 0.0000 C,BH. 21 59.59 0.0000 60.43 0.0000 H,BC. 15 58.73 0.0000 64.78 0.0000 BC,BH. 12 35.16 0.0004 33.76 0.0007 BH,CH. 18 27.67 0.0673 26.58 0.0872 n.s. CH,BC. 12 26.80 0.0082 33.18 0.0009 BC,BH,CH. 9 8.08 0.5265 6.56 0.6824 n.s. Goodness of fit testing of Models Possible Models: 1. [BH][CH] – B and C independent given H. 2. [BC][BH][CH] – all two factor interaction model

62. Model 1: [BH][CH] Log-linear parameters Heart disease -Blood Pressure Interaction

63. Multiplicative effect Log-Linear Model

64. Heart Disease - Cholesterol Interaction

65. Multiplicative effect

66. Model 2: [BC][BH][CH] Log-linear parameters Blood pressure-Cholesterol interaction:

67. Multiplicative effect

68. Heart disease -Blood Pressure Interaction

69. Multiplicative effect

70. Heart Disease - Cholesterol Interaction

71. Multiplicative effect

72. Log Linear Model

73. Two-way table where Note: X and Y are independent if In this case the log-linear model becomes

74. Three-way Frequency Tables

75. Log-Linear model for three-way tables Let  ijk denote the expected frequency in cell (i,j,k) of the table then in general where

76. Hierarchical Log-linear models for categorical Data For three way tables The hierarchical principle: If an interaction is in the model, also keep lower order interactions and main effects associated with that interaction

77. Hierarchical Log-linear models for 3 way table ModelDescription [1][2][3] Mutual independence between all three variables. [1][23] Independence of Variable 1 with variables 2 and 3. [2][13] Independence of Variable 2 with variables 1 and 3. [3][12] Independence of Variable 3 with variables 1 and 2. [12][13] Conditional independence between variables 2 and 3 given variable 1. [12][23] Conditional independence between variables 1 and 3 given variable 2. [13][23] Conditional independence between variables 1 and 2 given variable 3. [12][13] [23] Pairwise relations among all three variables, with each two variable interaction unaffected by the value of the third variable. [123]

78. Maximum Likelihood Estimation Log-Linear Model

79. For any Model it is possible to determine the maximum Likelihood Estimators of the parameters Example Two-way table – independence – multinomial model or

80. Log-likelihood where With the model of independence

81. and with also

82. Let Now

83. Since

84. Now or

85. Hence and Similarly Finally

86. Hence Now and

87. Hence Note or

88. Comments Maximum Likelihood estimates can be computed for any hierarchical log linear model (i.e. more than 2 variables) In certain situations the equations need to be solved numerically For the saturated model (all interactions and main effects)

89. Goodness of Fit Statistics These statistics can be used to check if a log-linear model will fit the observed frequency table

90. Goodness of Fit Statistics The Chi-squared statistic The Likelihood Ratio statistic: d.f. = # cells - # parameters fitted We reject the model if  2 or G 2 is greater than

91. Example: Variables 1.Systolic Blood Pressure (B) Serum Cholesterol (C) Coronary Heart Disease (H)

92. MODEL DF LIKELIHOOD- PROB. PEARSON PROB. RATIO CHISQ CHISQ ----- -- ----------- ------- ------- ------- B,C,H. 24 83.15 0.0000 102.00 0.0000 B,CH. 21 51.23 0.0002 56.89 0.0000 C,BH. 21 59.59 0.0000 60.43 0.0000 H,BC. 15 58.73 0.0000 64.78 0.0000 BC,BH. 12 35.16 0.0004 33.76 0.0007 BH,CH. 18 27.67 0.0673 26.58 0.0872 n.s. CH,BC. 12 26.80 0.0082 33.18 0.0009 BC,BH,CH. 9 8.08 0.5265 6.56 0.6824 n.s. Goodness of fit testing of Models Possible Models: 1. [BH][CH] – B and C independent given H. 2. [BC][BH][CH] – all two factor interaction model

93. Model 1: [BH][CH] Log-linear parameters Heart disease -Blood Pressure Interaction

94. Multiplicative effect Log-Linear Model

95. Heart Disease - Cholesterol Interaction

96. Multiplicative effect

97. Model 2: [BC][BH][CH] Log-linear parameters Blood pressure-Cholesterol interaction:

98. Multiplicative effect

99. Heart disease -Blood Pressure Interaction

100. Multiplicative effect

101. Heart Disease - Cholesterol Interaction

102. Multiplicative effect

103. Another Example In this study it was determined for N = 4353 males 1.Occupation category 2.Educational Level 3.Academic Aptidude

104. 1.Occupation categories a.Self-employed Business b.Teacher\Education c.Self-employed Professional d.Salaried Employed 2.Education levels a.Low b.Low/Med c.Med d.High/Med e.High

105. 3.Academic Aptitude a.Low b.Low/Med c.High/Med d.High

108. It is common to handle a Multiway table by testing for independence in all two way tables. This is similar to looking at all the bivariate correlations In this example we learn that: 1.Education is related to Aptitude 2.Education is related to Occupational category 3.Education is related to Aptitude Can we do better than this?

109. Fitting various log-linear models Simplest model that fits is: [Apt,Ed][Occ,Ed] This model implies conditional independence between Aptitude and Occupation given Education.

110. Log-linear Parameters Aptitude – Education Interaction

111. Aptitude – Education Interaction (Multiplicative)

112. Occupation – Education Interaction

113. Occupation – Education Interaction (Multiplicative)