1. Demonstration of different analytic methods to account for clustering in an observational study of emergency department use in primary care Michelle Howard, James Goertzen, Janusz Kaczorowski, Lehana Thabane, Brian Hutchison, Kelly Morris, Mitch Levine, Alexandra Papaioannou Departments of Family Medicine, Clinical Epidemiology & Biostatistics, and Medicine, McMaster University, Centre for Evaluation of Medicines, St. Joseph’s Healthcare, Hamilton Northern Ontario School of Medicine

2. Background Standard statistical tests assume independence of the outcome between observations Many health research study designs may violate this assumption –Interventions delivered at the level of school, family, physician practice –Outcomes measured at individual participant level Requires special considerations for sample size calculation and statistical analysis –For RCTs AND observational studies

3. Type I error in analysis Reduced variance Inflated power Reduced p value Type II error in sample size calculation ignoring clustering in sample size calculation may reduce power What does clustering do?

4. Objective To demonstrate options for the statistical analysis of correlated binary outcome data, using a study comparing emergency department (ED) use among patients nested within family physicians who had different practice attributes

5. Family Practice Models FHNFHGFFS PaymentCapitatedFFS After-hours clinics Yes No Tele-triageYes No Negated for outside GP use YesNo FHN=Family Health Network FHG=Family Health Group FFS=fee-for-service

6. Overall study objective Compare FHN, to FHG and FFS patients for use of ED for most recent urgent health problem in past 6 months 36 FPs (FHN=8, FHG=16, FFS=12) Nearly all family practice in Thunder Bay Cross-sectional survey to random sample of patients Approx 250/practice To detect 3% difference in ED use (10% vs 7%) in FHN vs. others ICC of 0.001 used to inflate sample size Study Methods

7. Statistical Methods Used Standard methods –Chi-square –Binary logistic regression Adjusted methods –Cluster-adjusted chi square –Generalized estimating equation (GEE) –Random intercepts model Cluster-level –T-test (unweighted & weighted)/analysis of variance

8. Demographic characteristics and ED use FHN (n=1772)FHG+FFS (n=4015) % female59.960.4 Mean age (years), SD42.2, 22.744.4, 21.7 Some or completed post- secondary education 48.350.2 Annual household income <$30,000 331 (20.3)795 (22.7) Self-reported health status Excellent Very good Good Fair Poor 426 (24.3) 636 (36.3) 502 (28.6) 148 (8.4) 41 (2.3) 755 (19.0) 1497 (37.7) 1179 (29.7) 422 (10.6) 118 (3.0) Visited emergency dept in past 6 months 11.414.9 response rate 62.3% (5884/9373)

9. Between-group comparison of ED use (FHN vs others) Type of estimateEstimate (95% CI)P value Individual-level Analyses (ICC=0.004) Chi-squareChi-square statistic14.2, df=1<0.001 Logistic regression* Odds ratio0.71 (0.59-0.86)<0.001 Cluster-adjusted Chi-square Chi-square statistic7.73, df=10.005 GEE*Odds ratio0.72 (0.62-0.84)<0.001 Random intercepts*† Odds ratio0.70 (0.58-0.86)<0.001 * Adjusted for age, income, self-reported health† OR for physician heterogeneity=1.003, 95% CI=0.97 to 1.04

10. Type of estimateEstimate (95% CI)P value Cluster-level Analyses T-test (unweighted) Mean difference in proportion 3.8% (1.1%-6.5%)0.02 T-test (weighted*)Mean difference in proportion 3.7% (1.3%-6.2%)0.004 Analysis of covariance (unweighted) F value5.23, df=10.03 Analysis of covariance (weighted*) F value5.93, df=10.02 * weighted by inverse of variance of cluster-specific event rate- W ij =m ij /[1+(m ij -1)  ]

11. Discussion Several statistical approaches for cluster adjustment gave same conclusions ICC for ED use in family practice is small –Other studies in primary care 0.01 to 0.02 Clustering issues can be avoided if doing analysis at cluster (practice) level GEE requires approximately 40 clusters to be reliable No significant physician heterogeneity –Not always the case with very provider-dependent outcomes

12. Conclusions Several approaches to cluster adjustment are possible –GEE is appealing in observational studies where controlling for covariates is important Adjustment is important given the unknown extent of ICC in primary care outcomes